2024-09-13T14:09:59.655 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T14:09:59.681 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T14:09:59.795 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564 branch: wip-aclamk-cbt-combined description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} email: akupczyk@redhat.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - echo wait for mgr daemons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo wait for minority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for majority of mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for all mons to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - sleep 60 - echo wait for half of osds to upgrade - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done first-half-tasks: - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' first_in_suite: false job_id: '7903564' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run basic import/export cli tests for rbd ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - mons down - mon down - MON_DOWN - out of quorum - PG_AVAILABILITY sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - sleep 60 - echo wait for upgrade to complete - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 6545 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: reef clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: reef clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 682/13280 suite: upgrade suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2Z13lEdz71lz0vEFVZ1IijtNyRmwfJOEhVZguQHOT7cL5dcAQt9wzqNlch3a/vrmSbYYt1FS6zwOs6YlooXZM= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/ZfmgIjCsCJNSCw83Iqyn8dhpRNDB/4EvtXLd5m5DCC3hM+hB8G3GOycCPMjQayY+X+YEbYnUCRG2SvmHOBgs= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: branch: reef exclude_packages: - ceph-volume - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_10:00:18 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T14:09:59.795 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T14:09:59.796 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T14:09:59.796 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T14:09:59.797 INFO:teuthology.task.internal:Checking packages... 2024-09-13T14:09:59.818 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T14:09:59.818 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:09:59.818 INFO:teuthology.packaging:ref: None 2024-09-13T14:09:59.818 INFO:teuthology.packaging:tag: None 2024-09-13T14:09:59.818 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T14:09:59.818 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:09:59.818 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T14:09:59.976 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T14:09:59.977 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T14:09:59.986 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T14:09:59.986 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T14:10:00.010 INFO:teuthology.task.internal:Saving configuration 2024-09-13T14:10:00.033 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T14:10:00.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T14:10:00.073 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 14:04:41.462372', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF2Z13lEdz71lz0vEFVZ1IijtNyRmwfJOEhVZguQHOT7cL5dcAQt9wzqNlch3a/vrmSbYYt1FS6zwOs6YlooXZM='} 2024-09-13T14:10:00.105 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 14:04:41.461168', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB/ZfmgIjCsCJNSCw83Iqyn8dhpRNDB/4EvtXLd5m5DCC3hM+hB8G3GOycCPMjQayY+X+YEbYnUCRG2SvmHOBgs='} 2024-09-13T14:10:00.106 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T14:10:00.113 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-13T14:10:00.114 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-13T14:10:00.114 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T14:10:00.192 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5a4e417be0>, signals=[15]) 2024-09-13T14:10:00.192 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T14:10:00.200 INFO:teuthology.task.internal:Opening connections... 2024-09-13T14:10:00.200 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T14:10:00.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:10:00.285 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-09-13T14:10:00.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:10:00.370 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T14:10:00.378 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-09-13T14:10:00.396 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-09-13T14:10:00.397 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-09-13T14:10:00.453 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-09-13T14:10:00.453 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T14:10:00.454 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T14:10:00.455 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-09-13T14:10:00.493 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-09-13T14:10:00.514 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-09-13T14:10:00.514 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-09-13T14:10:00.572 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-09-13T14:10:00.572 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-09-13T14:10:00.572 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T14:10:00.573 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T14:10:00.574 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-09-13T14:10:00.610 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T14:10:00.621 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T14:10:00.631 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T14:10:00.632 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-09-13T14:10:00.634 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-09-13T14:10:00.650 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T14:10:00.663 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T14:10:00.664 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-09-13T14:10:00.692 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-09-13T14:10:00.780 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T14:10:00.848 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T14:10:00.862 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T14:10:00.863 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T14:10:00.863 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T14:10:00.863 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T14:10:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:10:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:10:00.882 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-09-13T14:10:00.882 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-13T14:10:00.940 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-503.el9.x86_64 2024-09-13T14:10:00.940 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-503.el9.x86_64 2024-09-13T14:10:00.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-09-13T14:10:00.941 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-503.el9.x86_64 2024-09-13T14:10:00.942 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.14.0-503.el9.x86_64 2024-09-13T14:10:00.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2024-09-13T14:10:03.269 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-13T14:10:04.974 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.2 MB 00:02 2024-09-13T14:10:05.278 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-13T14:10:09.691 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 20 MB 00:03 2024-09-13T14:10:09.796 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.5 MB 00:00 2024-09-13T14:10:11.405 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 19 kB 00:00 2024-09-13T14:10:12.859 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T14:10:15.200 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-13T14:10:16.750 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 19 kB 00:00 2024-09-13T14:10:17.671 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-13T14:10:18.348 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T14:10:20.329 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T14:10:20.330 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T14:10:20.424 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:10:20.436 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:10:20.436 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T14:10:20.436 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:10:20.436 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:10:20.436 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T14:10:20.436 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-09-13T14:10:20.437 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:10:20.438 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 79 M 2024-09-13T14:10:20.438 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 126 M 2024-09-13T14:10:20.438 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:10:20.557 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 228 kB/s | 23 kB 00:00 2024-09-13T14:10:21.316 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-09-13T14:10:21.900 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 24 MB/s | 35 MB 00:01 2024-09-13T14:10:22.533 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 14 MB/s | 29 MB 00:01 2024-09-13T14:10:22.533 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:22.533 INFO:teuthology.orchestra.run.smithi154.stdout:Total 38 MB/s | 79 MB 00:02 2024-09-13T14:10:23.077 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T14:10:23.216 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:10:23.569 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:10:23.569 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:10:24.521 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:10:24.521 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:10:25.067 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T14:10:25.067 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T14:10:25.169 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:10:25.174 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:10:25.174 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-09-13T14:10:25.174 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:10:25.174 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:10:25.175 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T14:10:25.175 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:10:25.175 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T14:10:25.175 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T14:10:25.176 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T14:10:25.176 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:10:25.176 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:10:25.176 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:10:25.176 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-09-13T14:10:25.176 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:10:25.178 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 79 M 2024-09-13T14:10:25.178 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 126 M 2024-09-13T14:10:25.178 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:10:25.362 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 164 kB/s | 23 kB 00:00 2024-09-13T14:10:25.846 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-09-13T14:10:26.337 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 31 MB/s | 35 MB 00:01 2024-09-13T14:10:26.596 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 23 MB/s | 29 MB 00:01 2024-09-13T14:10:26.596 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:10:26.596 INFO:teuthology.orchestra.run.smithi172.stdout:Total 55 MB/s | 79 MB 00:01 2024-09-13T14:10:26.987 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:10:27.280 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:10:27.625 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:10:27.625 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:10:27.735 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:10:27.826 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:10:28.561 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:10:28.561 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:10:30.627 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:10:30.764 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:10:31.021 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:10:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:10:31.746 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:10:34.550 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:10:34.675 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:10:35.634 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:10:35.777 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:10:39.505 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:10:39.600 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:10:40.634 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:10:44.424 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:18.855 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:18.895 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:19.270 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:19.270 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:11:19.270 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:11:19.270 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:11:20.134 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:20.134 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:11:20.134 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:11:20.134 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T14:11:20.134 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T14:11:20.134 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T14:11:20.135 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T14:11:20.135 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:11:20.135 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:11:20.473 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:11:20.565 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:11:20.566 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:11:20.566 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T14:11:20.566 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T14:11:20.566 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T14:11:20.566 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T14:11:20.566 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T14:11:20.566 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:11:20.626 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:11:20.626 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:11:20.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.06 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 82.el9 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Group : Unspecified 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 8274670 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2024-09-13T14:11:20.659 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : builder@centos.org 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T14:11:20.660 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T14:11:20.661 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2024-09-13T14:11:20.661 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:11:20.661 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T14:11:20.662 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T14:11:20.662 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 5.14.0-505.el9.x86_64 2024-09-13T14:11:20.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T14:11:21.305 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-09-13T14:11:22.564 INFO:teuthology.orchestra.run.smithi154.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T14:11:22.597 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-09-13T14:11:22.600 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-13T14:11:22.617 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.9643Z5Svus 2024-09-13T14:11:22.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9643Z5Svus 2024-09-13T14:11:22.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.9643Z5Svus 2024-09-13T14:11:22.825 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.9643Z5Svus is 6KB 2024-09-13T14:11:22.847 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.9643Z5Svus 2024-09-13T14:11:22.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T14:11:22.933 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T14:11:22.933 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T14:11:22.933 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T14:11:22.933 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T14:11:22.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T14:11:22.963 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:23.003 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:23.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-09-13T14:11:23.346 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:23.346 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T14:11:23.346 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T14:11:23.346 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T14:11:24.165 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T14:11:24.165 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:11:24.165 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:11:24.166 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T14:11:24.166 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T14:11:24.166 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T14:11:24.166 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T14:11:24.166 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:11:24.166 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:11:24.501 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:11:24.591 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:11:24.592 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:11:24.592 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T14:11:24.592 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T14:11:24.592 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T14:11:24.592 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T14:11:24.592 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T14:11:24.592 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T14:11:24.650 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T14:11:24.651 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T14:11:24.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2024-09-13T14:11:24.682 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.06 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 82.el9 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Group : Unspecified 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 8274670 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T14:11:24.683 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : builder@centos.org 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:11:24.684 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T14:11:24.685 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T14:11:24.685 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 5.14.0-505.el9.x86_64 2024-09-13T14:11:24.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T14:11:25.364 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-09-13T14:11:26.660 INFO:teuthology.orchestra.run.smithi172.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T14:11:26.703 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-09-13T14:11:26.705 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-09-13T14:11:26.722 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.YYItxSKzMq 2024-09-13T14:11:26.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YYItxSKzMq 2024-09-13T14:11:26.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.YYItxSKzMq 2024-09-13T14:11:26.927 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.YYItxSKzMq is 6KB 2024-09-13T14:11:26.948 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.YYItxSKzMq 2024-09-13T14:11:26.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T14:11:27.026 INFO:teuthology.orchestra.run.smithi172.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T14:11:27.026 INFO:teuthology.orchestra.run.smithi172.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T14:11:27.026 INFO:teuthology.orchestra.run.smithi172.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T14:11:27.026 INFO:teuthology.orchestra.run.smithi172.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T14:11:27.027 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T14:11:27.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-09-13T14:11:53.101 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T14:11:53.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T14:11:53.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:11:57.181 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T14:11:57.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-13T14:11:57.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:12:27.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-13T14:12:31.677 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-09-13T14:12:36.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T14:12:36.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:12:40.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-13T14:12:40.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:13:08.310 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-13T14:13:08.734 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T14:13:08.735 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T14:13:08.735 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-13T14:13:08.753 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-505.el9.x86_64 2024-09-13T14:13:08.753 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T14:13:08.753 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T14:13:08.754 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T14:13:09.754 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T14:13:09.754 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T14:13:09.830 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-09-13T14:13:09.858 DEBUG:teuthology.parallel:result is None 2024-09-13T14:13:40.694 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T14:13:52.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-13T14:13:52.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:13:53.124 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-09-13T14:13:53.475 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-13T14:13:53.475 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T14:13:53.475 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-09-13T14:13:53.492 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-505.el9.x86_64 2024-09-13T14:13:53.492 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T14:13:53.492 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T14:13:53.492 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T14:13:54.493 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T14:13:54.493 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T14:13:54.562 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-09-13T14:13:54.586 DEBUG:teuthology.parallel:result is None 2024-09-13T14:13:54.586 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T14:13:54.594 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T14:13:54.595 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T14:13:54.599 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T14:13:54.617 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T14:13:54.624 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T14:13:54.633 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T14:13:54.634 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T14:13:54.659 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T14:13:54.686 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T14:13:54.693 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T14:13:54.693 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T14:13:54.722 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T14:13:54.756 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:13:54.763 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:13:54.767 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:13:54.775 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:13:54.776 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T14:13:54.783 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T14:13:54.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T14:13:54.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T14:13:54.846 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T14:13:54.857 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T14:13:54.857 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T14:13:54.888 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T14:13:54.904 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:13:54.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:13:55.037 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:13:55.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:13:55.160 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:13:55.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T14:13:55.226 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:13:55.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T14:13:55.320 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:13:55.411 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T14:13:55.446 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:13:55.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T14:13:55.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-09-13T14:13:55.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-09-13T14:13:55.566 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T14:13:55.609 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T14:13:56.025 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T14:13:56.032 INFO:teuthology.task.internal:Starting timer... 2024-09-13T14:13:56.032 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T14:13:56.042 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T14:13:56.050 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-13T14:13:56.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-09-13T14:13:56.121 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-09-13T14:13:56.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-09-13T14:13:56.174 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-09-13T14:13:56.176 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T14:13:56.176 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-09-13T14:13:56.205 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-09-13T14:13:56.206 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-09-13T14:13:56.242 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-09-13T14:13:56.242 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi154.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-09-13T14:13:56.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T14:13:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:13:56.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T14:13:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:13:56.308 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T14:13:56.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-09-13T14:13:56.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-09-13T14:13:56.386 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T14:13:56.396 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T14:13:56.407 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T14:13:56.408 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi154.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T14:17:45.850 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi154.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-09-13T14:17:45.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T14:17:45.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:17:45.938 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-13T14:17:46.020 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T14:17:46.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-13T14:17:46.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T14:17:46.110 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-09-13T14:17:46.189 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-13T14:17:46.189 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T14:17:46.201 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T14:17:46.201 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:17:46.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:17:46.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T14:17:46.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T14:17:46.230 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:17:46.245 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:17:46.258 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T14:17:46.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-09-13T14:17:46.272 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T14:17:46.280 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-09-13T14:17:46.294 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:17:46.297 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-09-13T14:17:46.308 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-09-13T14:17:46.309 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:17:46.322 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T14:17:46.336 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T14:17:46.366 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:17:46.370 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:17:46.370 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-13T14:17:46.370 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.370 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.370 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.370 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.394 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-09-13T14:17:46.398 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T14:17:46.398 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-09-13T14:17:46.398 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.398 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.398 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.398 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T14:17:46.398 INFO:teuthology.run_tasks:Running task pexec... 2024-09-13T14:17:46.408 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-13T14:17:46.408 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T14:17:46.409 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:17:46.409 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi172.front.sepia.ceph.com 2024-09-13T14:17:46.409 DEBUG:teuthology.orchestra.run.smithi172:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T14:17:46.759 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:17:46.775 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:17:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:Removing: 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:Removing unused dependencies: 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:Remove 2 Packages 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:Freed space: 5.5 M 2024-09-13T14:17:46.776 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:17:46.780 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:17:46.780 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:17:46.787 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:17:46.796 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:17:46.796 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-09-13T14:17:46.796 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:17:46.796 INFO:teuthology.orchestra.run.smithi172.stdout:Removing: 2024-09-13T14:17:46.796 INFO:teuthology.orchestra.run.smithi172.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout:Removing unused dependencies: 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout:Remove 2 Packages 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout:Freed space: 5.5 M 2024-09-13T14:17:46.797 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:17:46.801 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:17:46.801 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:17:46.846 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:17:46.846 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:17:46.854 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:17:46.855 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:17:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:17:46.992 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:17:47.062 INFO:teuthology.orchestra.run.smithi172.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:17:47.069 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T14:17:47.116 INFO:teuthology.orchestra.run.smithi172.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:17:47.131 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:17:47.530 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:17:47.531 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:17:47.605 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T14:17:47.605 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T14:17:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:17:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:47.766 INFO:teuthology.orchestra.run.smithi172.stdout:Removed: 2024-09-13T14:17:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:17:47.766 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:47.766 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:17:47.850 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T14:17:47.850 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:47.850 INFO:teuthology.orchestra.run.smithi154.stdout:Removed: 2024-09-13T14:17:47.850 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:17:47.850 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:47.850 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:17:48.662 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:45 ago on Fri 13 Sep 2024 02:17:03 PM UTC. 2024-09-13T14:17:48.681 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 02:16:54 PM UTC. 2024-09-13T14:17:48.796 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:17:48.797 INFO:teuthology.orchestra.run.smithi172.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2024-09-13T14:17:48.798 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:48.812 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.0 M 2024-09-13T14:17:48.812 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 9.2 M 2024-09-13T14:17:48.812 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:17:48.820 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:17:48.821 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:17:48.821 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T14:17:48.821 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:17:48.821 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:17:48.821 INFO:teuthology.orchestra.run.smithi154.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T14:17:48.821 INFO:teuthology.orchestra.run.smithi154.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T14:17:48.821 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-09-13T14:17:48.822 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:48.827 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.0 M 2024-09-13T14:17:48.827 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 9.2 M 2024-09-13T14:17:48.827 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:17:48.946 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 521 kB/s | 44 kB 00:00 2024-09-13T14:17:48.971 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 957 kB/s | 106 kB 00:00 2024-09-13T14:17:49.005 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-13T14:17:49.030 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-13T14:17:49.063 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.4 MB/s | 906 kB 00:00 2024-09-13T14:17:49.130 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.5 MB/s | 837 kB 00:00 2024-09-13T14:17:49.131 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:17:49.131 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.3 MB/s | 2.0 MB 00:00 2024-09-13T14:17:49.205 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:17:49.219 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:17:49.219 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:17:49.495 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 69 kB/s | 44 kB 00:00 2024-09-13T14:17:49.520 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 159 kB/s | 106 kB 00:00 2024-09-13T14:17:49.522 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:17:49.522 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:17:49.762 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 996 kB/s | 906 kB 00:00 2024-09-13T14:17:49.949 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:17:50.049 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-13T14:17:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-13T14:17:50.121 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-13T14:17:50.171 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 106 kB/s | 72 kB 00:00 2024-09-13T14:17:50.187 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-kmod-0.9-32.el9.x86_64.rpm 126 kB/s | 84 kB 00:00 2024-09-13T14:17:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-13T14:17:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T14:17:50.746 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 857 kB/s | 837 kB 00:00 2024-09-13T14:17:50.747 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:17:50.747 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.0 MB/s | 2.0 MB 00:01 2024-09-13T14:17:50.793 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T14:17:50.827 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:17:50.833 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T14:17:50.842 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:17:50.842 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:17:51.137 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:17:51.138 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:17:51.555 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:17:51.641 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-13T14:17:51.685 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-13T14:17:51.720 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-13T14:17:51.802 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T14:17:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-13T14:17:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-13T14:17:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-13T14:17:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-13T14:17:51.803 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-13T14:17:51.933 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-13T14:17:51.976 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:17:52.159 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:17:52.418 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T14:17:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T14:17:52.477 DEBUG:teuthology.parallel:result is None 2024-09-13T14:17:53.412 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T14:17:53.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-13T14:17:53.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-13T14:17:53.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-13T14:17:53.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-13T14:17:53.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-13T14:17:53.776 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-13T14:17:53.776 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:53.776 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:17:53.776 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T14:17:53.776 INFO:teuthology.orchestra.run.smithi154.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T14:17:53.776 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T14:17:53.776 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:17:53.777 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:17:54.073 DEBUG:teuthology.parallel:result is None 2024-09-13T14:17:54.073 INFO:teuthology.run_tasks:Running task install... 2024-09-13T14:17:54.082 DEBUG:teuthology.task.install:project ceph 2024-09-13T14:17:54.082 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:17:54.082 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:17:54.082 INFO:teuthology.task.install:Using flavor: default 2024-09-13T14:17:54.088 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T14:17:54.088 INFO:teuthology.task.install:extra packages: [] 2024-09-13T14:17:54.088 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:17:54.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:17:54.089 INFO:teuthology.packaging:ref: None 2024-09-13T14:17:54.089 INFO:teuthology.packaging:tag: None 2024-09-13T14:17:54.089 INFO:teuthology.packaging:branch: reef 2024-09-13T14:17:54.089 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:54.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:17:54.091 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T14:17:54.091 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:17:54.091 INFO:teuthology.packaging:ref: None 2024-09-13T14:17:54.091 INFO:teuthology.packaging:tag: None 2024-09-13T14:17:54.091 INFO:teuthology.packaging:branch: reef 2024-09-13T14:17:54.091 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:54.092 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:17:54.338 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/ 2024-09-13T14:17:54.339 INFO:teuthology.task.install.rpm:Package version is 18.2.4-843.g2c01458a 2024-09-13T14:17:54.374 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/ 2024-09-13T14:17:54.374 INFO:teuthology.task.install.rpm:Package version is 18.2.4-843.g2c01458a 2024-09-13T14:17:54.473 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:17:54.473 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:17:54.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:17:54.495 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T14:17:54.495 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:17:54.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T14:17:54.507 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:17:54.507 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:17:54.507 INFO:teuthology.packaging:ref: None 2024-09-13T14:17:54.507 INFO:teuthology.packaging:tag: None 2024-09-13T14:17:54.507 INFO:teuthology.packaging:branch: reef 2024-09-13T14:17:54.507 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:54.507 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:17:54.523 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T14:17:54.523 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:17:54.523 INFO:teuthology.packaging:ref: None 2024-09-13T14:17:54.523 INFO:teuthology.packaging:tag: None 2024-09-13T14:17:54.523 INFO:teuthology.packaging:branch: reef 2024-09-13T14:17:54.523 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:17:54.524 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T14:17:54.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:17:54.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T14:17:54.657 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:17:54.693 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T14:17:54.717 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-09-13T14:17:54.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-09-13T14:17:54.723 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2024-09-13T14:17:54.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2024-09-13T14:17:54.936 INFO:teuthology.orchestra.run.smithi154.stdout:45 files removed 2024-09-13T14:17:54.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2024-09-13T14:17:54.969 INFO:teuthology.orchestra.run.smithi172.stdout:45 files removed 2024-09-13T14:17:54.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2024-09-13T14:17:55.526 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 280 kB/s | 77 kB 00:00 2024-09-13T14:17:55.560 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 285 kB/s | 77 kB 00:00 2024-09-13T14:17:55.819 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 46 kB/s | 11 kB 00:00 2024-09-13T14:17:55.835 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 50 kB/s | 11 kB 00:00 2024-09-13T14:17:56.094 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-13T14:17:56.094 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-13T14:17:56.502 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 22 MB/s | 8.2 MB 00:00 2024-09-13T14:17:58.002 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-13T14:18:01.803 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 20 MB 00:04 2024-09-13T14:18:03.503 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-13T14:18:06.171 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-13T14:18:07.837 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-13T14:18:08.079 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-13T14:18:09.340 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 19 kB 00:00 2024-09-13T14:18:09.775 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-13T14:18:11.108 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-13T14:18:15.181 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T14:18:16.531 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T14:18:17.545 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:18:17.549 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:18:17.549 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:17.549 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:18:17.549 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 7.7 M 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.3 M 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:18.2.4-843.g2c01458a.el9 ceph 5.1 M 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 18 M 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:18.2.4-843.g2c01458a.el9 ceph 25 k 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:18:17.550 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 712 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 477 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.5 M 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:18:17.551 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 45 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 129 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:18.2.4-843.g2c01458a.el9 ceph 162 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:18.2.4-843.g2c01458a.el9 ceph 321 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 301 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 99 k 2024-09-13T14:18:17.552 INFO:teuthology.orchestra.run.smithi172.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T14:18:17.553 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:18:17.553 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:17.553 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:18:17.553 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:17.553 INFO:teuthology.orchestra.run.smithi172.stdout:Install 28 Packages 2024-09-13T14:18:17.553 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2024-09-13T14:18:17.553 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:17.554 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 53 M 2024-09-13T14:18:17.554 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:18:18.415 INFO:teuthology.orchestra.run.smithi172.stdout:(1/30): ceph-base-18.2.4-843.g2c01458a.el9.x86_ 7.5 MB/s | 5.1 MB 00:00 2024-09-13T14:18:18.515 INFO:teuthology.orchestra.run.smithi172.stdout:(2/30): ceph-radosgw-18.2.4-843.g2c01458a.el9.x 10 MB/s | 7.7 MB 00:00 2024-09-13T14:18:18.540 INFO:teuthology.orchestra.run.smithi172.stdout:(3/30): ceph-selinux-18.2.4-843.g2c01458a.el9.x 201 kB/s | 25 kB 00:00 2024-09-13T14:18:18.574 INFO:teuthology.orchestra.run.smithi172.stdout:(4/30): libcephfs2-18.2.4-843.g2c01458a.el9.x86 12 MB/s | 712 kB 00:00 2024-09-13T14:18:18.599 INFO:teuthology.orchestra.run.smithi172.stdout:(5/30): libradosstriper1-18.2.4-843.g2c01458a.e 7.9 MB/s | 477 kB 00:00 2024-09-13T14:18:18.650 INFO:teuthology.orchestra.run.smithi172.stdout:(6/30): python3-ceph-argparse-18.2.4-843.g2c014 901 kB/s | 45 kB 00:00 2024-09-13T14:18:18.725 INFO:teuthology.orchestra.run.smithi172.stdout:(7/30): python3-ceph-common-18.2.4-843.g2c01458 1.7 MB/s | 129 kB 00:00 2024-09-13T14:18:18.909 INFO:teuthology.orchestra.run.smithi172.stdout:(8/30): ceph-common-18.2.4-843.g2c01458a.el9.x8 16 MB/s | 18 MB 00:01 2024-09-13T14:18:18.934 INFO:teuthology.orchestra.run.smithi172.stdout:(9/30): python3-cephfs-18.2.4-843.g2c01458a.el9 778 kB/s | 162 kB 00:00 2024-09-13T14:18:18.956 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 7.7 M 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.3 M 2024-09-13T14:18:18.958 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:18.2.4-843.g2c01458a.el9 ceph 5.1 M 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 18 M 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:18.2.4-843.g2c01458a.el9 ceph 25 k 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 712 k 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T14:18:18.959 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 477 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.5 M 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 45 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:18.2.4-843.g2c01458a.el9 ceph 129 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:18.2.4-843.g2c01458a.el9 ceph 162 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T14:18:18.960 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:18.2.4-843.g2c01458a.el9 ceph 321 k 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 301 k 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:18.2.4-843.g2c01458a.el9 ceph 99 k 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout:Install 28 Packages 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2024-09-13T14:18:18.961 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:18.962 INFO:teuthology.orchestra.run.smithi172.stdout:(10/30): python3-rados-18.2.4-843.g2c01458a.el9 6.3 MB/s | 321 kB 00:00 2024-09-13T14:18:18.962 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 53 M 2024-09-13T14:18:18.962 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:18:19.026 INFO:teuthology.orchestra.run.smithi172.stdout:(11/30): librgw2-18.2.4-843.g2c01458a.el9.x86_6 9.9 MB/s | 4.5 MB 00:00 2024-09-13T14:18:19.051 INFO:teuthology.orchestra.run.smithi172.stdout:(12/30): python3-rbd-18.2.4-843.g2c01458a.el9.x 2.5 MB/s | 301 kB 00:00 2024-09-13T14:18:19.076 INFO:teuthology.orchestra.run.smithi172.stdout:(13/30): python3-rgw-18.2.4-843.g2c01458a.el9.x 851 kB/s | 99 kB 00:00 2024-09-13T14:18:19.185 INFO:teuthology.orchestra.run.smithi172.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-13T14:18:19.210 INFO:teuthology.orchestra.run.smithi172.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 581 kB/s | 107 kB 00:00 2024-09-13T14:18:19.236 INFO:teuthology.orchestra.run.smithi172.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 872 kB/s | 160 kB 00:00 2024-09-13T14:18:19.286 INFO:teuthology.orchestra.run.smithi172.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 849 kB/s | 42 kB 00:00 2024-09-13T14:18:19.311 INFO:teuthology.orchestra.run.smithi172.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-13T14:18:19.361 INFO:teuthology.orchestra.run.smithi172.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-13T14:18:19.403 INFO:teuthology.orchestra.run.smithi172.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 946 kB/s | 111 kB 00:00 2024-09-13T14:18:19.428 INFO:teuthology.orchestra.run.smithi172.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-13T14:18:19.454 INFO:teuthology.orchestra.run.smithi172.stdout:(22/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-13T14:18:19.479 INFO:teuthology.orchestra.run.smithi172.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-09-13T14:18:19.496 INFO:teuthology.orchestra.run.smithi172.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-13T14:18:19.546 INFO:teuthology.orchestra.run.smithi172.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-13T14:18:19.613 INFO:teuthology.orchestra.run.smithi172.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-13T14:18:19.713 INFO:teuthology.orchestra.run.smithi172.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-09-13T14:18:19.889 INFO:teuthology.orchestra.run.smithi172.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-09-13T14:18:19.898 INFO:teuthology.orchestra.run.smithi154.stdout:(1/30): ceph-base-18.2.4-843.g2c01458a.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-09-13T14:18:19.982 INFO:teuthology.orchestra.run.smithi154.stdout:(2/30): ceph-selinux-18.2.4-843.g2c01458a.el9.x 303 kB/s | 25 kB 00:00 2024-09-13T14:18:19.983 INFO:teuthology.orchestra.run.smithi172.stdout:(29/30): librados2-18.2.4-843.g2c01458a.el9.x86 9.0 MB/s | 3.3 MB 00:00 2024-09-13T14:18:20.082 INFO:teuthology.orchestra.run.smithi154.stdout:(3/30): libcephfs2-18.2.4-843.g2c01458a.el9.x86 7.0 MB/s | 712 kB 00:00 2024-09-13T14:18:20.147 INFO:teuthology.orchestra.run.smithi172.stdout:(30/30): librbd1-18.2.4-843.g2c01458a.el9.x86_6 6.9 MB/s | 3.0 MB 00:00 2024-09-13T14:18:20.152 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:18:20.152 INFO:teuthology.orchestra.run.smithi172.stdout:Total 20 MB/s | 53 MB 00:02 2024-09-13T14:18:20.199 INFO:teuthology.orchestra.run.smithi154.stdout:(4/30): ceph-radosgw-18.2.4-843.g2c01458a.el9.x 10 MB/s | 7.7 MB 00:00 2024-09-13T14:18:20.225 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:18:20.233 INFO:teuthology.orchestra.run.smithi154.stdout:(5/30): libradosstriper1-18.2.4-843.g2c01458a.e 3.1 MB/s | 477 kB 00:00 2024-09-13T14:18:20.260 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:18:20.260 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:18:20.300 INFO:teuthology.orchestra.run.smithi154.stdout:(6/30): python3-ceph-argparse-18.2.4-843.g2c014 671 kB/s | 45 kB 00:00 2024-09-13T14:18:20.383 INFO:teuthology.orchestra.run.smithi154.stdout:(7/30): python3-ceph-common-18.2.4-843.g2c01458 1.5 MB/s | 129 kB 00:00 2024-09-13T14:18:20.459 INFO:teuthology.orchestra.run.smithi154.stdout:(8/30): librgw2-18.2.4-843.g2c01458a.el9.x86_64 17 MB/s | 4.5 MB 00:00 2024-09-13T14:18:20.484 INFO:teuthology.orchestra.run.smithi154.stdout:(9/30): python3-cephfs-18.2.4-843.g2c01458a.el9 1.6 MB/s | 162 kB 00:00 2024-09-13T14:18:20.517 INFO:teuthology.orchestra.run.smithi154.stdout:(10/30): python3-rados-18.2.4-843.g2c01458a.el9 5.4 MB/s | 321 kB 00:00 2024-09-13T14:18:20.543 INFO:teuthology.orchestra.run.smithi154.stdout:(11/30): python3-rbd-18.2.4-843.g2c01458a.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-13T14:18:20.576 INFO:teuthology.orchestra.run.smithi154.stdout:(12/30): python3-rgw-18.2.4-843.g2c01458a.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-13T14:18:20.724 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:18:20.724 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:18:20.760 INFO:teuthology.orchestra.run.smithi154.stdout:(13/30): ceph-common-18.2.4-843.g2c01458a.el9.x 14 MB/s | 18 MB 00:01 2024-09-13T14:18:21.135 INFO:teuthology.orchestra.run.smithi154.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 121 kB/s | 45 kB 00:00 2024-09-13T14:18:21.160 INFO:teuthology.orchestra.run.smithi154.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 173 kB/s | 107 kB 00:00 2024-09-13T14:18:21.219 INFO:teuthology.orchestra.run.smithi154.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 249 kB/s | 160 kB 00:00 2024-09-13T14:18:21.328 INFO:teuthology.orchestra.run.smithi154.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-09-13T14:18:21.386 INFO:teuthology.orchestra.run.smithi154.stdout:(18/30): fmt-8.1.1-5.el9.x86_64.rpm 1.9 MB/s | 111 kB 00:00 2024-09-13T14:18:21.428 INFO:teuthology.orchestra.run.smithi154.stdout:(19/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 7.2 MB/s | 308 kB 00:00 2024-09-13T14:18:21.454 INFO:teuthology.orchestra.run.smithi154.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-09-13T14:18:21.504 INFO:teuthology.orchestra.run.smithi154.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 523 kB/s | 26 kB 00:00 2024-09-13T14:18:21.537 INFO:teuthology.orchestra.run.smithi154.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-13T14:18:21.571 INFO:teuthology.orchestra.run.smithi154.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-13T14:18:21.629 INFO:teuthology.orchestra.run.smithi154.stdout:(24/30): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 662 kB 00:00 2024-09-13T14:18:21.713 INFO:teuthology.orchestra.run.smithi154.stdout:(25/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 839 kB 00:00 2024-09-13T14:18:21.738 INFO:teuthology.orchestra.run.smithi154.stdout:(26/30): re2-20211101-20.el9.x86_64.rpm 1.7 MB/s | 191 kB 00:00 2024-09-13T14:18:21.922 INFO:teuthology.orchestra.run.smithi154.stdout:(27/30): librados2-18.2.4-843.g2c01458a.el9.x86 18 MB/s | 3.3 MB 00:00 2024-09-13T14:18:21.964 INFO:teuthology.orchestra.run.smithi154.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-09-13T14:18:22.031 INFO:teuthology.orchestra.run.smithi154.stdout:(29/30): libarrow-9.0.0-11.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-09-13T14:18:22.079 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:18:22.098 INFO:teuthology.orchestra.run.smithi154.stdout:(30/30): librbd1-18.2.4-843.g2c01458a.el9.x86_6 17 MB/s | 3.0 MB 00:00 2024-09-13T14:18:22.103 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:18:22.103 INFO:teuthology.orchestra.run.smithi154.stdout:Total 17 MB/s | 53 MB 00:03 2024-09-13T14:18:22.156 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T14:18:22.173 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:18:22.186 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T14:18:22.208 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:18:22.208 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:18:22.235 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T14:18:22.556 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T14:18:22.676 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:18:22.680 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:18:22.680 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:18:22.773 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:18:22.861 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:18:22.894 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:18:22.957 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:18:22.994 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T14:18:23.037 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 9/32 2024-09-13T14:18:23.103 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:18:23.130 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:18:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:18:23.310 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T14:18:23.354 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T14:18:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T14:18:23.435 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T14:18:23.462 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T14:18:23.515 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T14:18:23.548 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T14:18:23.849 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T14:18:23.885 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:18:23.993 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:18:24.024 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 21/32 2024-09-13T14:18:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:18:24.059 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T14:18:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T14:18:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T14:18:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T14:18:24.465 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T14:18:24.487 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:18:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:18:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 25/32 2024-09-13T14:18:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 26/32 2024-09-13T14:18:24.747 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T14:18:24.802 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:18:24.894 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:18:24.977 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:18:25.012 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:18:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:18:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T14:18:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 9/32 2024-09-13T14:18:25.221 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:18:25.247 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:18:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:18:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T14:18:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T14:18:25.483 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T14:18:25.512 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T14:18:25.539 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T14:18:25.575 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T14:18:25.608 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T14:18:25.910 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T14:18:25.962 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:18:26.036 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 20/32 2024-09-13T14:18:26.067 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 21/32 2024-09-13T14:18:26.103 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T14:18:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T14:18:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:18:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 24/32 2024-09-13T14:18:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 25/32 2024-09-13T14:18:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 26/32 2024-09-13T14:18:26.735 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:18:26.770 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:18:27.342 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:18:27.389 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:18:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:18:27.479 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:18:27.479 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:18:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:18:28.763 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:18:28.805 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:18:29.365 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 27/32 2024-09-13T14:18:29.416 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:18:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 28/32 2024-09-13T14:18:29.516 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T14:18:29.516 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T14:18:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:18:37.582 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:18:37.582 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2024-09-13T14:18:37.582 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2024-09-13T14:18:37.582 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2024-09-13T14:18:37.582 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2024-09-13T14:18:37.582 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2024-09-13T14:18:37.582 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /root 2024-09-13T14:18:37.583 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2024-09-13T14:18:37.583 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:38.321 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:18:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:18:38.346 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:18:38.346 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:18:38.346 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:18:38.346 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:18:38.346 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:38.348 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:18:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:18:38.423 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:18:39.328 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:18:39.328 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-09-13T14:18:39.329 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-09-13T14:18:39.329 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-09-13T14:18:39.329 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-09-13T14:18:39.329 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-09-13T14:18:39.329 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-09-13T14:18:39.329 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-09-13T14:18:39.329 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 1/32 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2/32 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 3/32 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 4/32 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:18:39.557 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 8/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 9/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 12/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 13/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T14:18:39.558 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T14:18:39.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 31/32 2024-09-13T14:18:40.092 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:18:40.117 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 30/32 2024-09-13T14:18:40.117 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:18:40.117 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T14:18:40.118 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:18:40.118 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T14:18:40.118 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:40.119 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:18:40.203 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T14:18:40.204 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:18:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:18:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T14:18:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:40.431 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:18:40.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2024-09-13T14:18:41.342 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 1/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 3/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 4/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 5/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 6/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 7/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x 8/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86 9/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 10/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 11/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 12/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 13/32 2024-09-13T14:18:41.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T14:18:41.344 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T14:18:41.345 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T14:18:41.345 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T14:18:41.345 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T14:18:41.345 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:18.2.4-843.g2c01458a.el9.x86_64 29/32 2024-09-13T14:18:41.345 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T14:18:41.345 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 31/32 2024-09-13T14:18:41.358 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:26 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:18:41.490 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:18:41.491 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:18:41.491 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:41.491 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:18:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:18.2.4-843.g2c01458a.el9 ceph 40 M 2024-09-13T14:18:41.491 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:18:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:18.2.4-843.g2c01458a.el9 ceph 166 k 2024-09-13T14:18:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:18:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:18:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:41.492 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:18:41.492 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:41.492 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-09-13T14:18:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:41.493 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 40 M 2024-09-13T14:18:41.493 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 164 M 2024-09-13T14:18:41.493 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:18:41.673 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-13T14:18:41.706 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-13T14:18:41.773 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): libcephsqlite-18.2.4-843.g2c01458a.el9.x 713 kB/s | 166 kB 00:00 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.322 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.323 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:42.324 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:18:42.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2024-09-13T14:18:42.848 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): ceph-test-18.2.4-843.g2c01458a.el9.x86_6 30 MB/s | 40 MB 00:01 2024-09-13T14:18:42.851 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:18:42.851 INFO:teuthology.orchestra.run.smithi172.stdout:Total 30 MB/s | 40 MB 00:01 2024-09-13T14:18:42.878 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:18:42.898 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:18:42.898 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:18:43.222 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:18:43.222 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:18:43.276 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:27 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:18:43.411 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:18.2.4-843.g2c01458a.el9 ceph 40 M 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:18.2.4-843.g2c01458a.el9 ceph 166 k 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T14:18:43.412 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:43.413 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:18:43.413 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:43.413 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-09-13T14:18:43.413 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:43.413 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 40 M 2024-09-13T14:18:43.413 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 164 M 2024-09-13T14:18:43.413 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:18:43.694 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:18:43.798 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:18:43.839 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:18:43.867 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:18:43.902 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-13T14:18:43.918 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): libcephsqlite-18.2.4-843.g2c01458a.el9.x 732 kB/s | 166 kB 00:00 2024-09-13T14:18:43.952 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-09-13T14:18:44.952 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): ceph-test-18.2.4-843.g2c01458a.el9.x86_6 31 MB/s | 40 MB 00:01 2024-09-13T14:18:44.954 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:18:44.955 INFO:teuthology.orchestra.run.smithi154.stdout:Total 26 MB/s | 40 MB 00:01 2024-09-13T14:18:44.983 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:18:45.002 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:18:45.002 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:18:45.333 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:18:45.333 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:18:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:18:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T14:18:45.951 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T14:18:45.979 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:18:47.584 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:18:47.657 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:18:49.655 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:18:49.655 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 1/4 2024-09-13T14:18:49.655 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2/4 2024-09-13T14:18:49.655 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:18:49.676 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 3/4 2024-09-13T14:18:49.752 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:18:49.981 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:18:49.981 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:49.981 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:18:49.981 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:49.981 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:49.981 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:18:49.981 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:18:49.982 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:49.982 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:18:50.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2024-09-13T14:18:50.877 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:35 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:18:51.014 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:18:51.016 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:51.016 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:18:51.016 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:51.016 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:18:51.016 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:18.2.4-843.g2c01458a.el9 ceph 6.5 k 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:18.2.4-843.g2c01458a.el9 ceph 2.1 M 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:18.2.4-843.g2c01458a.el9 ceph 1.5 M 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 246 k 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.7 M 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 17 M 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:18:51.017 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:18:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:18:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:18:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:18:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:18:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:18:51.018 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:18:51.019 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout:Install 33 Packages 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 29 M 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 104 M 2024-09-13T14:18:51.020 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:18:51.240 INFO:teuthology.orchestra.run.smithi172.stdout:(1/33): ceph-18.2.4-843.g2c01458a.el9.x86_64.rp 57 kB/s | 6.5 kB 00:00 2024-09-13T14:18:51.433 INFO:teuthology.orchestra.run.smithi172.stdout:(2/33): ceph-mds-18.2.4-843.g2c01458a.el9.x86_6 6.8 MB/s | 2.1 MB 00:00 2024-09-13T14:18:51.474 INFO:teuthology.orchestra.run.smithi172.stdout:(3/33): ceph-mgr-18.2.4-843.g2c01458a.el9.x86_6 4.2 MB/s | 1.5 MB 00:00 2024-09-13T14:18:51.575 INFO:teuthology.orchestra.run.smithi172.stdout:(4/33): ceph-mon-18.2.4-843.g2c01458a.el9.x86_6 14 MB/s | 4.7 MB 00:00 2024-09-13T14:18:51.600 INFO:teuthology.orchestra.run.smithi172.stdout:(5/33): ceph-mgr-modules-core-18.2.4-843.g2c014 1.9 MB/s | 246 kB 00:00 2024-09-13T14:18:51.676 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 4/4 2024-09-13T14:18:51.676 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 1/4 2024-09-13T14:18:51.676 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2/4 2024-09-13T14:18:51.677 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T14:18:51.700 INFO:teuthology.orchestra.run.smithi172.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-09-13T14:18:51.743 INFO:teuthology.orchestra.run.smithi172.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-13T14:18:51.784 INFO:teuthology.orchestra.run.smithi172.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-09-13T14:18:51.809 INFO:teuthology.orchestra.run.smithi172.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-13T14:18:51.835 INFO:teuthology.orchestra.run.smithi172.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-13T14:18:51.860 INFO:teuthology.orchestra.run.smithi172.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T14:18:51.991 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:51.992 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:18:52.027 INFO:teuthology.orchestra.run.smithi172.stdout:(12/33): ceph-osd-18.2.4-843.g2c01458a.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-13T14:18:52.252 INFO:teuthology.orchestra.run.smithi172.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 89 kB/s | 35 kB 00:00 2024-09-13T14:18:52.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2024-09-13T14:18:52.411 INFO:teuthology.orchestra.run.smithi172.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 133 kB/s | 51 kB 00:00 2024-09-13T14:18:52.453 INFO:teuthology.orchestra.run.smithi172.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 208 kB/s | 42 kB 00:00 2024-09-13T14:18:52.478 INFO:teuthology.orchestra.run.smithi172.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 268 kB/s | 172 kB 00:00 2024-09-13T14:18:52.503 INFO:teuthology.orchestra.run.smithi172.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-13T14:18:52.545 INFO:teuthology.orchestra.run.smithi172.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-09-13T14:18:52.570 INFO:teuthology.orchestra.run.smithi172.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 716 kB/s | 18 kB 00:00 2024-09-13T14:18:52.596 INFO:teuthology.orchestra.run.smithi172.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 126 kB/s | 11 kB 00:00 2024-09-13T14:18:52.621 INFO:teuthology.orchestra.run.smithi172.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-09-13T14:18:52.646 INFO:teuthology.orchestra.run.smithi172.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-13T14:18:52.671 INFO:teuthology.orchestra.run.smithi172.stdout:(23/33): python3-jaraco-collections-3.0.0-8.el9 230 kB/s | 23 kB 00:00 2024-09-13T14:18:52.697 INFO:teuthology.orchestra.run.smithi172.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-09-13T14:18:52.722 INFO:teuthology.orchestra.run.smithi172.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-09-13T14:18:52.747 INFO:teuthology.orchestra.run.smithi172.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-13T14:18:52.764 INFO:teuthology.orchestra.run.smithi172.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-09-13T14:18:52.789 INFO:teuthology.orchestra.run.smithi172.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-13T14:18:52.815 INFO:teuthology.orchestra.run.smithi172.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-09-13T14:18:52.840 INFO:teuthology.orchestra.run.smithi172.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-13T14:18:52.857 INFO:teuthology.orchestra.run.smithi172.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-13T14:18:52.870 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:36 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:18:52.882 INFO:teuthology.orchestra.run.smithi172.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-13T14:18:52.907 INFO:teuthology.orchestra.run.smithi172.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-13T14:18:52.912 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:18:52.912 INFO:teuthology.orchestra.run.smithi172.stdout:Total 15 MB/s | 29 MB 00:01 2024-09-13T14:18:53.006 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:18.2.4-843.g2c01458a.el9 ceph 6.5 k 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:18.2.4-843.g2c01458a.el9 ceph 2.1 M 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:18.2.4-843.g2c01458a.el9 ceph 1.5 M 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 246 k 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:18.2.4-843.g2c01458a.el9 ceph 4.7 M 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 17 M 2024-09-13T14:18:53.009 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T14:18:53.010 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:53.011 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:18:53.012 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:18:53.012 INFO:teuthology.orchestra.run.smithi154.stdout:Install 33 Packages 2024-09-13T14:18:53.012 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:18:53.012 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 29 M 2024-09-13T14:18:53.013 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 104 M 2024-09-13T14:18:53.013 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:18:53.036 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:18:53.074 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:18:53.074 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:18:53.449 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:18:53.449 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:18:53.569 INFO:teuthology.orchestra.run.smithi154.stdout:(1/33): ceph-18.2.4-843.g2c01458a.el9.x86_64.rp 60 kB/s | 6.5 kB 00:00 2024-09-13T14:18:53.761 INFO:teuthology.orchestra.run.smithi154.stdout:(2/33): ceph-mds-18.2.4-843.g2c01458a.el9.x86_6 7.0 MB/s | 2.1 MB 00:00 2024-09-13T14:18:53.803 INFO:teuthology.orchestra.run.smithi154.stdout:(3/33): ceph-mgr-18.2.4-843.g2c01458a.el9.x86_6 4.3 MB/s | 1.5 MB 00:00 2024-09-13T14:18:53.928 INFO:teuthology.orchestra.run.smithi154.stdout:(4/33): ceph-mon-18.2.4-843.g2c01458a.el9.x86_6 13 MB/s | 4.7 MB 00:00 2024-09-13T14:18:53.954 INFO:teuthology.orchestra.run.smithi154.stdout:(5/33): ceph-mgr-modules-core-18.2.4-843.g2c014 1.6 MB/s | 246 kB 00:00 2024-09-13T14:18:54.096 INFO:teuthology.orchestra.run.smithi154.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-13T14:18:54.146 INFO:teuthology.orchestra.run.smithi154.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-13T14:18:54.321 INFO:teuthology.orchestra.run.smithi154.stdout:(8/33): ceph-osd-18.2.4-843.g2c01458a.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-13T14:18:54.346 INFO:teuthology.orchestra.run.smithi154.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-09-13T14:18:54.380 INFO:teuthology.orchestra.run.smithi154.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-09-13T14:18:54.430 INFO:teuthology.orchestra.run.smithi154.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-13T14:18:54.489 INFO:teuthology.orchestra.run.smithi154.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-09-13T14:18:54.658 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:18:54.710 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T14:18:54.714 INFO:teuthology.orchestra.run.smithi154.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 122 kB/s | 35 kB 00:00 2024-09-13T14:18:54.751 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T14:18:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T14:18:54.856 INFO:teuthology.orchestra.run.smithi154.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 139 kB/s | 51 kB 00:00 2024-09-13T14:18:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T14:18:54.889 INFO:teuthology.orchestra.run.smithi154.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-09-13T14:18:54.906 INFO:teuthology.orchestra.run.smithi154.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 217 kB/s | 42 kB 00:00 2024-09-13T14:18:54.940 INFO:teuthology.orchestra.run.smithi154.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-13T14:18:54.957 INFO:teuthology.orchestra.run.smithi154.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 726 kB/s | 11 kB 00:00 2024-09-13T14:18:54.974 INFO:teuthology.orchestra.run.smithi154.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-13T14:18:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T14:18:54.999 INFO:teuthology.orchestra.run.smithi154.stdout:(20/33): python3-mako-1.1.4-6.el9.noarch.rpm 278 kB/s | 172 kB 00:00 2024-09-13T14:18:55.025 INFO:teuthology.orchestra.run.smithi154.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-13T14:18:55.037 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T14:18:55.058 INFO:teuthology.orchestra.run.smithi154.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-09-13T14:18:55.072 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T14:18:55.083 INFO:teuthology.orchestra.run.smithi154.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 384 kB/s | 20 kB 00:00 2024-09-13T14:18:55.101 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T14:18:55.109 INFO:teuthology.orchestra.run.smithi154.stdout:(24/33): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2024-09-13T14:18:55.134 INFO:teuthology.orchestra.run.smithi154.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 629 kB/s | 46 kB 00:00 2024-09-13T14:18:55.136 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T14:18:55.159 INFO:teuthology.orchestra.run.smithi154.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-13T14:18:55.169 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T14:18:55.184 INFO:teuthology.orchestra.run.smithi154.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-13T14:18:55.201 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T14:18:55.210 INFO:teuthology.orchestra.run.smithi154.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-13T14:18:55.235 INFO:teuthology.orchestra.run.smithi154.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-13T14:18:55.245 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T14:18:55.260 INFO:teuthology.orchestra.run.smithi154.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-13T14:18:55.279 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T14:18:55.285 INFO:teuthology.orchestra.run.smithi154.stdout:(31/33): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T14:18:55.311 INFO:teuthology.orchestra.run.smithi154.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-09-13T14:18:55.331 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T14:18:55.336 INFO:teuthology.orchestra.run.smithi154.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-13T14:18:55.341 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:18:55.341 INFO:teuthology.orchestra.run.smithi154.stdout:Total 13 MB/s | 29 MB 00:02 2024-09-13T14:18:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T14:18:55.464 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:18:55.501 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:18:55.501 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:18:55.569 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T14:18:55.653 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T14:18:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T14:18:55.744 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T14:18:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T14:18:55.865 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T14:18:55.889 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:18:55.890 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:18:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T14:18:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T14:18:56.185 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T14:18:56.273 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T14:18:56.326 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T14:18:56.445 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T14:18:56.596 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 28/33 2024-09-13T14:18:56.623 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:18:56.651 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:18:56.651 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:18:56.651 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:18:56.651 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:18:56.651 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:18:56.651 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:18:57.180 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T14:18:57.212 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T14:18:57.245 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T14:18:57.336 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T14:18:57.406 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T14:18:57.447 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T14:18:57.482 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T14:18:57.521 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T14:18:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T14:18:57.597 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T14:18:57.637 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T14:18:57.672 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T14:18:57.707 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T14:18:57.756 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T14:18:57.826 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T14:18:57.963 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T14:18:58.063 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T14:18:58.065 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:18:58.093 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:18:58.093 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:18:58.093 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:18:58.094 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:18:58.094 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:18:58.094 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:58.113 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T14:18:58.152 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T14:18:58.203 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T14:18:58.258 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T14:18:58.443 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T14:18:58.491 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T14:18:58.533 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:18:58.557 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:18:58.557 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:18:58.557 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:18:58.557 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:18:58.557 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:18:58.557 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:58.560 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T14:18:58.649 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T14:18:58.697 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T14:18:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:18:58.791 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:18:58.791 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:18:58.791 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:18:58.791 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:18:58.792 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:18:58.792 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:18:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T14:18:58.855 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:18:58.973 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 28/33 2024-09-13T14:18:59.001 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:18:59.026 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 29/33 2024-09-13T14:18:59.026 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:18:59.027 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T14:18:59.027 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:18:59.027 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T14:18:59.027 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:00.443 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:19:00.471 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 30/33 2024-09-13T14:19:00.471 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:19:00.471 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T14:19:00.471 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:19:00.471 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T14:19:00.471 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 1/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 3/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 4/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 5/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 6/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T14:19:00.569 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T14:19:00.570 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T14:19:00.571 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T14:19:00.919 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:19:00.943 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 31/33 2024-09-13T14:19:00.943 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:19:00.943 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T14:19:00.943 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:19:00.943 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T14:19:00.943 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:01.278 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:19:01.303 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 32/33 2024-09-13T14:19:01.303 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:19:01.303 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T14:19:01.303 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:19:01.303 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T14:19:01.303 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:01.366 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:19:01.467 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:19:01.468 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:19:01.469 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:01.470 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:01.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2024-09-13T14:19:02.414 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:47 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:02.506 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:02.548 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:02.549 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:02.549 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:02.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 33/33 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:18.2.4-843.g2c01458a.el9.x86_64 1/33 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2/33 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 3/33 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 4/33 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 5/33 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.n 6/33 2024-09-13T14:19:02.874 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T14:19:02.875 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T14:19:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T14:19:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T14:19:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T14:19:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T14:19:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T14:19:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T14:19:02.876 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T14:19:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T14:19:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T14:19:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T14:19:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T14:19:02.877 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T14:19:03.209 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:48 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:03.341 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:03.342 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:03.342 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:19:03.342 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:03.342 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:03.342 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 226 k 2024-09-13T14:19:03.342 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:03.342 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:03.343 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:03.343 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-09-13T14:19:03.343 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:03.343 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 226 k 2024-09-13T14:19:03.343 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 217 k 2024-09-13T14:19:03.343 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:03.582 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-18.2.4-843.g2c01458a.el9.noarch.rpm 949 kB/s | 226 kB 00:00 2024-09-13T14:19:03.582 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:03.582 INFO:teuthology.orchestra.run.smithi172.stdout:Total 943 kB/s | 226 kB 00:00 2024-09-13T14:19:03.582 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:03.587 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:03.588 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:03.590 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:03.591 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:03.628 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T14:19:03.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T14:19:03.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:03.631 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:03.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2024-09-13T14:19:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:04.168 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:04.512 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:04.556 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:48 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:04.651 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:04.693 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:04.694 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:04.694 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:04.755 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:04.755 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:04.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2024-09-13T14:19:05.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:19:05.334 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:49 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:05.466 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:05.467 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:05.467 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:19:05.467 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:05.467 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:05.467 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 226 k 2024-09-13T14:19:05.467 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:05.468 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:05.468 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:05.468 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T14:19:05.468 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:05.468 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 226 k 2024-09-13T14:19:05.468 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 217 k 2024-09-13T14:19:05.468 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:05.577 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:50 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:05.700 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-18.2.4-843.g2c01458a.el9.noarch.rpm 974 kB/s | 226 kB 00:00 2024-09-13T14:19:05.701 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:05.701 INFO:teuthology.orchestra.run.smithi154.stdout:Total 967 kB/s | 226 kB 00:00 2024-09-13T14:19:05.702 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:05.706 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:05.706 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:05.709 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:05.709 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:05.717 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:05.717 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:05.717 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-843.g2c01458a.el9 ceph 143 k 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 143 k 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 395 k 2024-09-13T14:19:05.718 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:05.783 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:05.929 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-18.2.4-843.g2c01458 681 kB/s | 143 kB 00:00 2024-09-13T14:19:05.929 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:05.929 INFO:teuthology.orchestra.run.smithi172.stdout:Total 676 kB/s | 143 kB 00:00 2024-09-13T14:19:05.930 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:05.937 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:05.937 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:05.994 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:05.994 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:19:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:06.221 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:19:06.221 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:19:06.221 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:19:06.221 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:06.651 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:06.946 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/1 2024-09-13T14:19:06.946 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:06.947 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:06.947 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:06.947 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:06.947 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:07.145 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:19:07.145 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:07.145 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:07.145 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:07.145 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:07.145 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:07.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T14:19:07.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2024-09-13T14:19:07.766 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:51 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:07.904 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-843.g2c01458a.el9 ceph 143 k 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T14:19:07.905 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:07.906 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 143 k 2024-09-13T14:19:07.906 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 395 k 2024-09-13T14:19:07.906 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:07.929 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:08.021 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:08.065 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:08.066 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:08.066 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:08.097 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-18.2.4-843.g2c01458 748 kB/s | 143 kB 00:00 2024-09-13T14:19:08.098 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:08.098 INFO:teuthology.orchestra.run.smithi154.stdout:Total 742 kB/s | 143 kB 00:00 2024-09-13T14:19:08.098 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:08.105 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:08.106 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:08.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:19:08.163 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:08.163 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:08.307 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:08.365 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:19:08.390 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:19:08.390 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:19:08.390 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T14:19:08.390 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:08.710 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:08.842 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:08.842 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:08.842 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:19:08.842 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 3.5 M 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 24 k 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 16 k 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout:Install 5 Packages 2024-09-13T14:19:08.843 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:08.844 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.8 M 2024-09-13T14:19:08.844 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 87 M 2024-09-13T14:19:08.844 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:09.148 INFO:teuthology.orchestra.run.smithi172.stdout:(1/5): ceph-prometheus-alerts-18.2.4-843.g2c014 111 kB/s | 16 kB 00:00 2024-09-13T14:19:09.173 INFO:teuthology.orchestra.run.smithi172.stdout:(2/5): ceph-grafana-dashboards-18.2.4-843.g2c01 141 kB/s | 24 kB 00:00 2024-09-13T14:19:09.215 INFO:teuthology.orchestra.run.smithi172.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 460 kB/s | 31 kB 00:00 2024-09-13T14:19:09.257 INFO:teuthology.orchestra.run.smithi172.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-13T14:19:09.280 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.e 1/1 2024-09-13T14:19:09.280 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:09.280 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:09.280 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:09.280 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:09.280 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:09.366 INFO:teuthology.orchestra.run.smithi172.stdout:(5/5): ceph-mgr-dashboard-18.2.4-843.g2c01458a. 9.7 MB/s | 3.5 MB 00:00 2024-09-13T14:19:09.370 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:09.370 INFO:teuthology.orchestra.run.smithi172.stdout:Total 7.2 MB/s | 3.8 MB 00:00 2024-09-13T14:19:09.370 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:09.377 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:09.377 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:09.451 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:09.452 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:09.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2024-09-13T14:19:09.706 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:09.789 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T14:19:09.820 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T14:19:09.849 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:19:10.082 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:10.178 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:10.221 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:10.222 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:10.222 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:10.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T14:19:10.867 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:10.907 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 4/5 2024-09-13T14:19:10.945 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:19:11.000 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:11.001 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:11.001 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:19:11.001 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:11.001 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:11.001 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 3.5 M 2024-09-13T14:19:11.001 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 24 k 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 16 k 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout:Install 5 Packages 2024-09-13T14:19:11.002 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:11.003 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.8 M 2024-09-13T14:19:11.003 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 87 M 2024-09-13T14:19:11.003 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:11.318 INFO:teuthology.orchestra.run.smithi154.stdout:(1/5): ceph-prometheus-alerts-18.2.4-843.g2c014 105 kB/s | 16 kB 00:00 2024-09-13T14:19:11.334 INFO:teuthology.orchestra.run.smithi154.stdout:(2/5): ceph-grafana-dashboards-18.2.4-843.g2c01 141 kB/s | 24 kB 00:00 2024-09-13T14:19:11.377 INFO:teuthology.orchestra.run.smithi154.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 526 kB/s | 31 kB 00:00 2024-09-13T14:19:11.418 INFO:teuthology.orchestra.run.smithi154.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-13T14:19:11.544 INFO:teuthology.orchestra.run.smithi154.stdout:(5/5): ceph-mgr-dashboard-18.2.4-843.g2c01458a. 9.2 MB/s | 3.5 MB 00:00 2024-09-13T14:19:11.547 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:11.547 INFO:teuthology.orchestra.run.smithi154.stdout:Total 6.9 MB/s | 3.8 MB 00:00 2024-09-13T14:19:11.548 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:11.555 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:11.555 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:11.644 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:11.645 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:11.892 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:11.966 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T14:19:12.007 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T14:19:12.027 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:19:12.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 1/5 2024-09-13T14:19:12.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2/5 2024-09-13T14:19:12.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:19:12.027 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T14:19:12.035 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:19:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T14:19:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:12.302 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:19:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:19:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:12.303 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:12.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:19:13.097 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 4/5 2024-09-13T14:19:13.131 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:19:13.232 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:13.233 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-09-13T14:19:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 7.4 M 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:19:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout:====================================================================================== 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout:Install 12 Packages 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:13.235 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 38 M 2024-09-13T14:19:13.236 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 212 M 2024-09-13T14:19:13.236 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:13.881 INFO:teuthology.orchestra.run.smithi172.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-13T14:19:13.931 INFO:teuthology.orchestra.run.smithi172.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-13T14:19:14.015 INFO:teuthology.orchestra.run.smithi172.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-13T14:19:14.065 INFO:teuthology.orchestra.run.smithi172.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-13T14:19:14.157 INFO:teuthology.orchestra.run.smithi172.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 420 kB/s | 38 kB 00:00 2024-09-13T14:19:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 5/5 2024-09-13T14:19:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.n 1/5 2024-09-13T14:19:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2/5 2024-09-13T14:19:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.no 3/5 2024-09-13T14:19:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T14:19:14.241 INFO:teuthology.orchestra.run.smithi172.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-84 15 MB/s | 7.4 MB 00:00 2024-09-13T14:19:14.439 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T14:19:14.439 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:14.439 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:14.439 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:14.439 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:14.439 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:14.440 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T14:19:14.440 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T14:19:14.440 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:14.440 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:14.441 INFO:teuthology.orchestra.run.smithi172.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-13T14:19:14.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T14:19:14.758 INFO:teuthology.orchestra.run.smithi172.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.6 MB/s | 3.0 MB 00:00 2024-09-13T14:19:14.908 INFO:teuthology.orchestra.run.smithi172.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 250 kB 00:00 2024-09-13T14:19:15.175 INFO:teuthology.orchestra.run.smithi172.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.9 MB/s | 4.9 MB 00:01 2024-09-13T14:19:15.225 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:15.359 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:15.359 INFO:teuthology.orchestra.run.smithi172.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.6 MB/s | 5.1 MB 00:00 2024-09-13T14:19:15.359 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 7.4 M 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T14:19:15.360 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout:Install 12 Packages 2024-09-13T14:19:15.361 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:15.362 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 38 M 2024-09-13T14:19:15.362 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 212 M 2024-09-13T14:19:15.362 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:15.651 INFO:teuthology.orchestra.run.smithi154.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-13T14:19:15.717 INFO:teuthology.orchestra.run.smithi154.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-13T14:19:15.968 INFO:teuthology.orchestra.run.smithi154.stdout:(3/12): ceph-mgr-diskprediction-local-18.2.4-84 16 MB/s | 7.4 MB 00:00 2024-09-13T14:19:15.993 INFO:teuthology.orchestra.run.smithi154.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 96 kB/s | 33 kB 00:00 2024-09-13T14:19:16.193 INFO:teuthology.orchestra.run.smithi154.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-13T14:19:16.252 INFO:teuthology.orchestra.run.smithi154.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-13T14:19:16.719 INFO:teuthology.orchestra.run.smithi154.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 524 kB/s | 245 kB 00:00 2024-09-13T14:19:16.794 INFO:teuthology.orchestra.run.smithi154.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-09-13T14:19:16.842 INFO:teuthology.orchestra.run.smithi172.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.4 MB/s | 16 MB 00:01 2024-09-13T14:19:16.843 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:16.843 INFO:teuthology.orchestra.run.smithi172.stdout:Total 11 MB/s | 38 MB 00:03 2024-09-13T14:19:16.911 INFO:teuthology.orchestra.run.smithi154.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-13T14:19:17.144 INFO:teuthology.orchestra.run.smithi154.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.2 MB/s | 4.9 MB 00:00 2024-09-13T14:19:17.195 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:17.221 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:17.221 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:17.733 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:17.733 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:18.509 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:18.581 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:19:18.615 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:19:18.706 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:19:18.741 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:19:18.778 INFO:teuthology.orchestra.run.smithi154.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.5 MB/s | 5.1 MB 00:02 2024-09-13T14:19:18.926 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:19:18.965 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:19:19.046 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:19:19.695 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:19:19.820 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:19:21.073 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:19:22.103 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:19:22.130 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:19:23.444 INFO:teuthology.orchestra.run.smithi154.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.5 MB/s | 16 MB 00:06 2024-09-13T14:19:23.445 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:23.445 INFO:teuthology.orchestra.run.smithi154.stdout:Total 4.7 MB/s | 38 MB 00:08 2024-09-13T14:19:23.799 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:23.826 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:23.826 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:24.323 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:24.323 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:19:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 1/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:19:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:19:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:19:25.112 INFO:teuthology.orchestra.run.smithi172.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:19:25.113 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:19:25.113 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:19:25.113 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:19:25.113 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:25.113 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T14:19:25.225 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:19:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:19:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T14:19:25.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:19:25.560 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T14:19:25.600 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T14:19:25.677 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T14:19:26.100 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:11 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:26.241 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:26.242 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:26.242 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:19:26.242 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:26.242 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 49 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:19:26.243 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:19:26.244 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:26.244 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2024-09-13T14:19:26.244 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.2 M 2024-09-13T14:19:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2024-09-13T14:19:26.245 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T14:19:26.445 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T14:19:26.780 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 272 kB/s | 19 kB 00:00 2024-09-13T14:19:26.805 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 268 kB/s | 26 kB 00:00 2024-09-13T14:19:26.872 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-13T14:19:26.897 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): ceph-mgr-rook-18.2.4-843.g2c01458a.el9. 262 kB/s | 49 kB 00:00 2024-09-13T14:19:26.922 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-13T14:19:26.947 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-09-13T14:19:26.981 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 639 kB/s | 54 kB 00:00 2024-09-13T14:19:26.998 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 426 kB/s | 32 kB 00:00 2024-09-13T14:19:27.023 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 206 kB/s | 14 kB 00:00 2024-09-13T14:19:27.057 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-13T14:19:27.099 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-09-13T14:19:27.124 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-13T14:19:27.174 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.0 MB/s | 1.0 MB 00:00 2024-09-13T14:19:27.177 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:27.177 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-13T14:19:27.241 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:27.257 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:27.257 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:27.346 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:27.346 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:19:27.864 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:19:28.126 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:19:28.182 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:19:28.218 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:19:28.260 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:19:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:19:28.464 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:19:28.511 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:19:28.714 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T14:19:28.749 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:19:29.247 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:19:29.291 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:19:29.325 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:19:29.382 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:19:29.403 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:19:30.208 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:19:30.208 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 1/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:19:30.209 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:19:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:30.560 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:30.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:19:31.285 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 12/12 2024-09-13T14:19:31.285 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-843.g2c0145 1/12 2024-09-13T14:19:31.285 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T14:19:31.285 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T14:19:31.285 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T14:19:31.285 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T14:19:31.285 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T14:19:31.286 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T14:19:31.286 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T14:19:31.286 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T14:19:31.286 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T14:19:31.286 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T14:19:31.393 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:16 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:31.528 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:31.528 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 139 k 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:31.529 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:31.530 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:31.530 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2024-09-13T14:19:31.530 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:31.530 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 7.0 M 2024-09-13T14:19:31.530 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 33 M 2024-09-13T14:19:31.530 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T14:19:31.692 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T14:19:31.693 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:31.694 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:31.909 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): ceph-mgr-cephadm-18.2.4-843.g2c01458a.el 731 kB/s | 139 kB 00:00 2024-09-13T14:19:32.009 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.4 MB/s | 548 kB 00:00 2024-09-13T14:19:32.059 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T14:19:32.084 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.0 MB/s | 74 kB 00:00 2024-09-13T14:19:32.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2024-09-13T14:19:32.518 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 310 kB/s | 248 kB 00:00 2024-09-13T14:19:32.665 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:16 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:32.799 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:32.800 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:32.800 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 49 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T14:19:32.801 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2024-09-13T14:19:32.802 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:32.803 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.2 M 2024-09-13T14:19:32.803 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 26 M 2024-09-13T14:19:32.803 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:32.860 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-13T14:19:32.862 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:32.862 INFO:teuthology.orchestra.run.smithi172.stdout:Total 5.3 MB/s | 7.0 MB 00:01 2024-09-13T14:19:32.935 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:32.947 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:32.947 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:33.047 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:33.047 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:33.100 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 163 kB/s | 19 kB 00:00 2024-09-13T14:19:33.125 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 179 kB/s | 26 kB 00:00 2024-09-13T14:19:33.150 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): ceph-mgr-rook-18.2.4-843.g2c01458a.el9. 293 kB/s | 49 kB 00:00 2024-09-13T14:19:33.209 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T14:19:33.226 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-09-13T14:19:33.260 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T14:19:33.293 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-13T14:19:33.319 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-09-13T14:19:33.349 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:33.361 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.3 MB/s | 220 kB 00:00 2024-09-13T14:19:33.385 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-13T14:19:33.419 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-13T14:19:33.461 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-13T14:19:33.520 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:19:33.528 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.9 MB/s | 1.0 MB 00:00 2024-09-13T14:19:33.531 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:33.531 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-13T14:19:33.570 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:19:33.595 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:33.610 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:33.610 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:33.704 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:33.704 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:34.103 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:19:34.190 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:19:34.203 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:34.260 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:19:34.290 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:19:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T14:19:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T14:19:34.485 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T14:19:34.522 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T14:19:34.563 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T14:19:34.673 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T14:19:34.775 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T14:19:34.832 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T14:19:35.041 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:19:35.041 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/6 2024-09-13T14:19:35.041 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:19:35.041 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:19:35.041 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:19:35.041 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:19:35.315 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:19:35.315 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:35.315 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:35.315 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:35.315 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:19:35.315 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:19:35.315 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:19:35.316 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:19:35.316 INFO:teuthology.orchestra.run.smithi172.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:19:35.316 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:35.316 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:35.561 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T14:19:35.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2024-09-13T14:19:35.612 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T14:19:35.645 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T14:19:35.694 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T14:19:35.724 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:19:36.143 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:21 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:36.279 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:36.279 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:36.279 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2024-09-13T14:19:36.279 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:36.279 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:36.279 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 852 k 2024-09-13T14:19:36.279 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-09-13T14:19:36.280 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:19:36.280 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:36.280 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:36.280 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:36.280 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2024-09-13T14:19:36.280 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:36.280 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 932 k 2024-09-13T14:19:36.281 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 2.7 M 2024-09-13T14:19:36.281 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 13/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 1/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T14:19:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T14:19:36.644 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 718 kB/s | 80 kB 00:00 2024-09-13T14:19:36.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T14:19:36.810 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:36.810 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:36.810 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:36.810 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:36.811 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:36.812 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-18.2.4-843.g2c01458a.el9.x86_6 3.0 MB/s | 852 kB 00:00 2024-09-13T14:19:36.812 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:36.812 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.7 MB/s | 932 kB 00:00 2024-09-13T14:19:36.824 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:36.844 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:36.844 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:36.925 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:36.925 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:37.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T14:19:37.117 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:19:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:19:37.641 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:21 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:37.778 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:18.2.4-843.g2c01458a.el9 ceph-noarch 139 k 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T14:19:37.779 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-09-13T14:19:37.780 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:37.781 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 7.0 M 2024-09-13T14:19:37.781 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 33 M 2024-09-13T14:19:37.781 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:19:37.983 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/2 2024-09-13T14:19:38.121 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): ceph-mgr-cephadm-18.2.4-843.g2c01458a.el 739 kB/s | 139 kB 00:00 2024-09-13T14:19:38.154 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-13T14:19:38.213 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-13T14:19:38.230 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 785 kB/s | 58 kB 00:00 2024-09-13T14:19:38.255 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.7 MB/s | 74 kB 00:00 2024-09-13T14:19:38.274 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:19:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:19:38.275 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:38.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2024-09-13T14:19:39.107 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:39.172 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-09-13T14:19:39.174 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:39.174 INFO:teuthology.orchestra.run.smithi154.stdout:Total 5.0 MB/s | 7.0 MB 00:01 2024-09-13T14:19:39.243 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:39.243 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:39.243 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 127 k 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-09-13T14:19:39.244 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:39.245 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:39.245 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 127 k 2024-09-13T14:19:39.245 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 456 k 2024-09-13T14:19:39.245 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:39.255 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:39.255 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:39.355 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:39.355 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:39.457 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-18.2.4-843.g2c01458a.el9.x86_64. 598 kB/s | 127 kB 00:00 2024-09-13T14:19:39.457 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:39.458 INFO:teuthology.orchestra.run.smithi172.stdout:Total 593 kB/s | 127 kB 00:00 2024-09-13T14:19:39.458 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:39.463 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:39.463 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:39.527 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:39.527 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:39.666 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:39.680 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:39.788 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:39.839 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T14:19:39.891 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T14:19:40.263 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:40.423 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T14:19:40.513 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T14:19:40.523 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:40.523 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:40.523 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:40.524 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:40.524 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:40.524 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:40.581 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T14:19:40.611 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:19:40.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2024-09-13T14:19:41.378 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:26 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:41.401 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 6/6 2024-09-13T14:19:41.401 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 1/6 2024-09-13T14:19:41.401 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T14:19:41.401 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T14:19:41.401 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T14:19:41.402 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T14:19:41.472 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:41.521 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:41.522 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:41.522 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:41.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2024-09-13T14:19:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T14:19:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:41.665 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:18.2.4-843.g2c01458a.el9.noarch 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:41.666 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:41.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2024-09-13T14:19:42.171 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:42.306 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:42.307 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:42.307 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-09-13T14:19:42.307 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:42.307 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:42.307 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 31 k 2024-09-13T14:19:42.308 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:42.308 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:42.308 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:42.308 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-09-13T14:19:42.308 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:42.308 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 31 k 2024-09-13T14:19:42.308 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 139 k 2024-09-13T14:19:42.309 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:42.469 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-18.2.4-843.g2c01458a.el9.x86_64 193 kB/s | 31 kB 00:00 2024-09-13T14:19:42.470 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:42.470 INFO:teuthology.orchestra.run.smithi172.stdout:Total 191 kB/s | 31 kB 00:00 2024-09-13T14:19:42.470 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:42.473 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:42.473 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:42.496 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:42.496 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:42.507 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:26 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:42.587 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:42.639 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 852 k 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T14:19:42.640 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:42.641 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:42.641 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:42.641 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2024-09-13T14:19:42.641 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:42.641 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 932 k 2024-09-13T14:19:42.641 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.7 M 2024-09-13T14:19:42.642 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:43.032 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:43.049 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): ceph-fuse-18.2.4-843.g2c01458a.el9.x86_6 3.0 MB/s | 852 kB 00:00 2024-09-13T14:19:43.224 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 176 kB/s | 80 kB 00:00 2024-09-13T14:19:43.224 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:43.224 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.6 MB/s | 932 kB 00:00 2024-09-13T14:19:43.236 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:43.258 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:43.259 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:43.272 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:43.272 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:43.273 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:43.273 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:43.339 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:43.340 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:43.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2024-09-13T14:19:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:43.664 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T14:19:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:19:44.066 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:44.157 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:44.200 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:44.201 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:44.201 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:44.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2024-09-13T14:19:44.415 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2/2 2024-09-13T14:19:44.415 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/2 2024-09-13T14:19:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T14:19:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:44.681 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T14:19:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:44.682 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:44.845 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:44.935 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:44.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2024-09-13T14:19:44.978 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:44.979 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:44.979 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:45.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2024-09-13T14:19:45.544 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:45.636 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:45.682 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 127 k 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:45.683 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T14:19:45.684 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:45.684 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 127 k 2024-09-13T14:19:45.684 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 456 k 2024-09-13T14:19:45.684 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:45.727 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:45.770 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:45.772 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:45.772 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:45.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2024-09-13T14:19:45.886 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-18.2.4-843.g2c01458a.el9.x86_64. 627 kB/s | 127 kB 00:00 2024-09-13T14:19:45.887 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:45.887 INFO:teuthology.orchestra.run.smithi154.stdout:Total 623 kB/s | 127 kB 00:00 2024-09-13T14:19:45.887 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:45.893 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:45.893 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:45.956 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:45.956 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:46.107 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:46.209 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:46.429 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:46.520 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:46.563 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:46.565 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:46.565 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:46.650 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:46.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2024-09-13T14:19:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:46.886 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:46.886 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:47.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2024-09-13T14:19:47.217 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:47.309 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:47.351 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:47.352 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:47.352 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:47.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2024-09-13T14:19:47.711 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:47.805 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:47.849 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:47.850 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:47.850 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:47.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2024-09-13T14:19:48.015 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:48.105 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:48.149 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:48.150 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2024-09-13T14:19:48.150 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:48.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2024-09-13T14:19:48.495 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:48.627 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:18.2.4-843.g2c01458a.el9 ceph 31 k 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:48.628 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:48.629 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T14:19:48.629 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:48.629 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 31 k 2024-09-13T14:19:48.629 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 139 k 2024-09-13T14:19:48.629 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:48.774 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-18.2.4-843.g2c01458a.el9.x86_64 215 kB/s | 31 kB 00:00 2024-09-13T14:19:48.775 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:48.775 INFO:teuthology.orchestra.run.smithi154.stdout:Total 213 kB/s | 31 kB 00:00 2024-09-13T14:19:48.775 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:48.778 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:48.778 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:48.796 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:48.796 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:48.817 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:48.887 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:48.951 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 87 k 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-09-13T14:19:48.952 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:48.953 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 87 k 2024-09-13T14:19:48.953 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 227 k 2024-09-13T14:19:48.953 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:48.987 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:49.147 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-18.2.4-843.g2c01458a.el9.x86_64.rpm 447 kB/s | 87 kB 00:00 2024-09-13T14:19:49.148 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:49.148 INFO:teuthology.orchestra.run.smithi172.stdout:Total 444 kB/s | 87 kB 00:00 2024-09-13T14:19:49.148 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:49.155 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:49.155 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:49.218 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:49.219 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:49.349 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:49.373 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:49.477 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:49.590 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:49.590 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:49.590 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:49.590 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:49.590 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:49.590 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:49.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2024-09-13T14:19:49.855 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:50.313 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:50.313 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:50.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2024-09-13T14:19:50.359 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:50.452 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:50.492 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:50.493 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:50.493 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:50.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2024-09-13T14:19:50.932 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:51.070 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:51.071 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:51.072 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-09-13T14:19:51.072 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:51.072 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.0 M 2024-09-13T14:19:51.072 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 12 M 2024-09-13T14:19:51.072 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:51.139 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:51.231 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:51.274 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:51.275 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:51.275 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:51.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2024-09-13T14:19:51.400 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-18.2.4-843.g2c01458a.el9.x86_64.rpm 9.1 MB/s | 3.0 MB 00:00 2024-09-13T14:19:51.401 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:51.401 INFO:teuthology.orchestra.run.smithi172.stdout:Total 9.1 MB/s | 3.0 MB 00:00 2024-09-13T14:19:51.401 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:51.410 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:51.410 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:51.495 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:51.495 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:51.923 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:52.009 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:52.014 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:52.034 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:52.034 INFO:teuthology.orchestra.run.smithi172.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:19:52.034 INFO:teuthology.orchestra.run.smithi172.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:19:52.034 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:19:52.034 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:19:52.034 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:52.056 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:52.058 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:52.058 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:52.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2024-09-13T14:19:52.704 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:52.793 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:52.837 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:52.838 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:52.838 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:52.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2024-09-13T14:19:53.129 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:53.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:53.129 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:53.129 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:53.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:53.129 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:53.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2024-09-13T14:19:53.470 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:53.561 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:53.604 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:53.605 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:53.605 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:53.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2024-09-13T14:19:53.973 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 02:18:15 PM UTC. 2024-09-13T14:19:54.105 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 172 k 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2024-09-13T14:19:54.106 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:54.107 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 172 k 2024-09-13T14:19:54.107 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 494 k 2024-09-13T14:19:54.107 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-09-13T14:19:54.243 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:54.319 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-18.2.4-843.g2c01458a.el9.x86_64.rpm 812 kB/s | 172 kB 00:00 2024-09-13T14:19:54.320 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:54.320 INFO:teuthology.orchestra.run.smithi172.stdout:Total 806 kB/s | 172 kB 00:00 2024-09-13T14:19:54.320 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-09-13T14:19:54.328 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-09-13T14:19:54.328 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-09-13T14:19:54.333 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:18.2.4-843.g2c01458a.el9.x86_64 is already installed. 2024-09-13T14:19:54.377 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:54.378 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T14:19:54.378 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:54.391 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-09-13T14:19:54.392 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-09-13T14:19:54.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2024-09-13T14:19:54.552 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-09-13T14:19:54.641 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:55.025 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:55.159 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:18.2.4-843.g2c01458a.el9 ceph 87 k 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T14:19:55.160 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:55.161 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 87 k 2024-09-13T14:19:55.161 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 227 k 2024-09-13T14:19:55.161 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:55.280 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:55.280 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:55.280 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-09-13T14:19:55.280 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:55.280 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:19:55.280 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-09-13T14:19:55.352 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-18.2.4-843.g2c01458a.el9.x86_64.rpm 455 kB/s | 87 kB 00:00 2024-09-13T14:19:55.352 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:55.352 INFO:teuthology.orchestra.run.smithi154.stdout:Total 452 kB/s | 87 kB 00:00 2024-09-13T14:19:55.353 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:55.359 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:55.359 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:55.423 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:55.424 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:55.517 DEBUG:teuthology.parallel:result is None 2024-09-13T14:19:55.583 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:56.063 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:56.329 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:56.329 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:56.329 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:56.330 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:56.330 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:56.330 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:56.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2024-09-13T14:19:57.158 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:19:57.295 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:18.2.4-843.g2c01458a.el9 ceph 3.0 M 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:57.296 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.0 M 2024-09-13T14:19:57.297 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 12 M 2024-09-13T14:19:57.297 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:19:57.637 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-18.2.4-843.g2c01458a.el9.x86_64.rpm 8.8 MB/s | 3.0 MB 00:00 2024-09-13T14:19:57.638 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:19:57.639 INFO:teuthology.orchestra.run.smithi154.stdout:Total 8.7 MB/s | 3.0 MB 00:00 2024-09-13T14:19:57.639 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:19:57.647 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:19:57.648 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:19:57.733 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:19:57.734 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:19:58.153 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:19:58.216 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T14:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T14:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T14:19:58.239 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:59.382 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:19:59.383 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:59.383 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:19:59.383 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:19:59.383 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:19:59.383 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:19:59.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2024-09-13T14:20:00.230 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 02:18:16 PM UTC. 2024-09-13T14:20:00.368 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T14:20:00.369 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:20:00.369 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T14:20:00.369 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:20:00.369 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T14:20:00.369 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:18.2.4-843.g2c01458a.el9 ceph 172 k 2024-09-13T14:20:00.369 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:20:00.369 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T14:20:00.370 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T14:20:00.370 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T14:20:00.370 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:20:00.370 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 172 k 2024-09-13T14:20:00.370 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 494 k 2024-09-13T14:20:00.370 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T14:20:00.608 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-18.2.4-843.g2c01458a.el9.x86_64.rpm 723 kB/s | 172 kB 00:00 2024-09-13T14:20:00.609 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:20:00.609 INFO:teuthology.orchestra.run.smithi154.stdout:Total 717 kB/s | 172 kB 00:00 2024-09-13T14:20:00.609 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T14:20:00.617 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T14:20:00.618 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T14:20:00.685 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T14:20:00.686 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T14:20:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T14:20:00.957 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:20:01.396 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:20:01.646 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 1/1 2024-09-13T14:20:01.646 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:20:01.646 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T14:20:01.646 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:18.2.4-843.g2c01458a.el9.x86_64 2024-09-13T14:20:01.646 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:20:01.646 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T14:20:01.923 DEBUG:teuthology.parallel:result is None 2024-09-13T14:20:01.923 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:20:01.923 INFO:teuthology.packaging:ref: None 2024-09-13T14:20:01.923 INFO:teuthology.packaging:tag: None 2024-09-13T14:20:01.923 INFO:teuthology.packaging:branch: reef 2024-09-13T14:20:01.923 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:01.924 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:20:02.199 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:20:02.221 INFO:teuthology.orchestra.run.smithi154.stdout:18.2.4-843.g2c01458a.el9 2024-09-13T14:20:02.222 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-843.g2c01458a.el9 2024-09-13T14:20:02.222 INFO:teuthology.task.install:The correct ceph version 18.2.4-843.g2c01458a is installed. 2024-09-13T14:20:02.224 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T14:20:02.224 INFO:teuthology.packaging:ref: None 2024-09-13T14:20:02.224 INFO:teuthology.packaging:tag: None 2024-09-13T14:20:02.224 INFO:teuthology.packaging:branch: reef 2024-09-13T14:20:02.224 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:02.224 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-13T14:20:02.492 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T14:20:02.519 INFO:teuthology.orchestra.run.smithi172.stdout:18.2.4-843.g2c01458a.el9 2024-09-13T14:20:02.519 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-843.g2c01458a.el9 2024-09-13T14:20:02.519 INFO:teuthology.task.install:The correct ceph version 18.2.4-843.g2c01458a is installed. 2024-09-13T14:20:02.521 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T14:20:02.521 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:20:02.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:20:02.550 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:20:02.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T14:20:02.590 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T14:20:02.590 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:20:02.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:20:02.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:20:02.682 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:20:02.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T14:20:02.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T14:20:02.780 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T14:20:02.780 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:20:02.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:20:02.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:20:02.870 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:20:02.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T14:20:02.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T14:20:02.964 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T14:20:02.965 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:20:02.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:20:02.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:20:03.054 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:20:03.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T14:20:03.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T14:20:03.151 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-13T14:20:03.245 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T14:20:03.246 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-13T14:20:03.246 INFO:tasks.cephadm:Cluster fsid is 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:20:03.246 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-13T14:20:03.247 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.154', 'mon.c': '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]', 'mon.b': '172.21.15.172'} 2024-09-13T14:20:03.247 INFO:tasks.cephadm:First mon is mon.a on smithi154 2024-09-13T14:20:03.247 INFO:tasks.cephadm:First mgr is y 2024-09-13T14:20:03.247 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-13T14:20:03.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-09-13T14:20:03.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo hostname $(hostname -s) 2024-09-13T14:20:03.301 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-13T14:20:03.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:20:03.453 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83028/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4697-gd58f7306', 'node_name': '172.21.4.75+confusa13', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4697.gd58f7306'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-12 23:59:16.034150', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-aclamk-cbt-combined', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2024-09-13T14:20:03.717 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef, sha1 2c01458affee2b2435cd231b6e392435c1c21661 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-13T14:20:03.719 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm 2024-09-13T14:20:03.719 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm 2024-09-13T14:20:03.719 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:20:04.064 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-r--r--. 1 ubuntu ubuntu 217218 Sep 13 14:20 /home/ubuntu/cephtest/cephadm 2024-09-13T14:20:04.064 DEBUG:teuthology.orchestra.run.smithi172:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/2c01458affee2b2435cd231b6e392435c1c21661/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T14:20:04.408 INFO:teuthology.orchestra.run.smithi172.stdout:-rw-r--r--. 1 ubuntu ubuntu 217218 Sep 13 14:20 /home/ubuntu/cephtest/cephadm 2024-09-13T14:20:04.409 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:20:04.428 DEBUG:teuthology.orchestra.run.smithi172:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T14:20:04.460 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-13T14:20:04.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-13T14:20:04.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-13T14:20:06.029 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:20:06.074 INFO:teuthology.orchestra.run.smithi172.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:21:19.314 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:21:19.314 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_version": "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)", 2024-09-13T14:21:19.314 INFO:teuthology.orchestra.run.smithi154.stdout: "image_id": "c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5", 2024-09-13T14:21:19.314 INFO:teuthology.orchestra.run.smithi154.stdout: "repo_digests": [ 2024-09-13T14:21:19.314 INFO:teuthology.orchestra.run.smithi154.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11" 2024-09-13T14:21:19.315 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:21:19.315 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:21:21.884 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:21:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_version": "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)", 2024-09-13T14:21:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "image_id": "c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5", 2024-09-13T14:21:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "repo_digests": [ 2024-09-13T14:21:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11" 2024-09-13T14:21:21.884 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:21:21.884 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:21:21.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-09-13T14:21:21.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph 2024-09-13T14:21:21.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-09-13T14:21:22.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /etc/ceph 2024-09-13T14:21:22.030 INFO:tasks.cephadm:Writing seed config... 2024-09-13T14:21:22.031 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-13T14:21:22.031 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-13T14:21:22.031 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-13T14:21:22.031 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-13T14:21:22.031 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-13T14:21:22.031 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-13T14:21:22.031 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-13T14:21:22.032 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-13T14:21:22.032 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-13T14:21:22.032 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-13T14:21:22.032 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-13T14:21:22.032 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:21:22.033 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-13T14:21:22.062 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 447b0cd2-71db-11ef-bceb-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-13T14:21:22.063 DEBUG:teuthology.orchestra.run.smithi154:mon.a> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service 2024-09-13T14:21:22.106 DEBUG:teuthology.orchestra.run.smithi154:mgr.y> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service 2024-09-13T14:21:22.149 INFO:tasks.cephadm:Bootstrapping... 2024-09-13T14:21:22.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.154 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:21:22.304 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T14:21:22.304 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '447b0cd2-71db-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.154', '--skip-admin-label'] 2024-09-13T14:21:22.328 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:21:22.328 INFO:teuthology.orchestra.run.smithi154.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-13T14:21:22.329 INFO:teuthology.orchestra.run.smithi154.stdout:Verifying podman|docker is present... 2024-09-13T14:21:22.350 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:21:22.350 INFO:teuthology.orchestra.run.smithi154.stdout:Verifying lvm2 is present... 2024-09-13T14:21:22.350 INFO:teuthology.orchestra.run.smithi154.stdout:Verifying time synchronization is in place... 2024-09-13T14:21:22.358 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:21:22.358 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:21:22.364 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:21:22.364 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout inactive 2024-09-13T14:21:22.371 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout enabled 2024-09-13T14:21:22.377 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout active 2024-09-13T14:21:22.377 INFO:teuthology.orchestra.run.smithi154.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:21:22.377 INFO:teuthology.orchestra.run.smithi154.stdout:Repeating the final host check... 2024-09-13T14:21:22.399 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stdout 5.2.2 2024-09-13T14:21:22.399 INFO:teuthology.orchestra.run.smithi154.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-13T14:21:22.399 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl is present 2024-09-13T14:21:22.400 INFO:teuthology.orchestra.run.smithi154.stdout:lvcreate is present 2024-09-13T14:21:22.406 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T14:21:22.406 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T14:21:22.412 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T14:21:22.412 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout inactive 2024-09-13T14:21:22.418 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout enabled 2024-09-13T14:21:22.425 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout active 2024-09-13T14:21:22.425 INFO:teuthology.orchestra.run.smithi154.stdout:Unit chronyd.service is enabled and running 2024-09-13T14:21:22.425 INFO:teuthology.orchestra.run.smithi154.stdout:Host looks OK 2024-09-13T14:21:22.425 INFO:teuthology.orchestra.run.smithi154.stdout:Cluster fsid: 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:22.426 INFO:teuthology.orchestra.run.smithi154.stdout:Acquiring lock 140124752674384 on /run/cephadm/447b0cd2-71db-11ef-bceb-c7b262605968.lock 2024-09-13T14:21:22.426 INFO:teuthology.orchestra.run.smithi154.stdout:Lock 140124752674384 acquired on /run/cephadm/447b0cd2-71db-11ef-bceb-c7b262605968.lock 2024-09-13T14:21:22.426 INFO:teuthology.orchestra.run.smithi154.stdout:Verifying IP 172.21.15.154 port 3300 ... 2024-09-13T14:21:22.426 INFO:teuthology.orchestra.run.smithi154.stdout:Verifying IP 172.21.15.154 port 6789 ... 2024-09-13T14:21:22.426 INFO:teuthology.orchestra.run.smithi154.stdout:Base mon IP(s) is [172.21.15.154:3300, 172.21.15.154:6789], mon addrv is [v2:172.21.15.154:3300,v1:172.21.15.154:6789] 2024-09-13T14:21:22.429 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.154 metric 100 2024-09-13T14:21:22.429 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.154 metric 100 2024-09-13T14:21:22.432 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-13T14:21:22.432 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-13T14:21:22.432 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7761/64 scope link noprefixroute 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:Mon IP `172.21.15.154` is in CIDR network `172.21.0.0/20` 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:Mon IP `172.21.15.154` is in CIDR network `172.21.0.0/20` 2024-09-13T14:21:22.435 INFO:teuthology.orchestra.run.smithi154.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-13T14:21:22.436 INFO:teuthology.orchestra.run.smithi154.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-13T14:21:22.436 INFO:teuthology.orchestra.run.smithi154.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:21:22.990 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stdout c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 2024-09-13T14:21:22.990 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-13T14:21:22.991 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stderr Getting image source signatures 2024-09-13T14:21:22.991 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-13T14:21:22.991 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stderr Copying blob sha256:f9a1328d66f49e17fe78ed666f97dd3c34cc07be60bd7ae6ba8f2402a2a33f7f 2024-09-13T14:21:22.991 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stderr Copying config sha256:c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 2024-09-13T14:21:22.991 INFO:teuthology.orchestra.run.smithi154.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-13T14:21:23.821 INFO:teuthology.orchestra.run.smithi154.stdout:ceph: stdout ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable) 2024-09-13T14:21:23.821 INFO:teuthology.orchestra.run.smithi154.stdout:Ceph version: ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable) 2024-09-13T14:21:23.821 INFO:teuthology.orchestra.run.smithi154.stdout:Extracting ceph user uid/gid from container image... 2024-09-13T14:21:24.802 INFO:teuthology.orchestra.run.smithi154.stdout:stat: stdout 167 167 2024-09-13T14:21:24.802 INFO:teuthology.orchestra.run.smithi154.stdout:Creating initial keys... 2024-09-13T14:21:25.808 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph-authtool: stdout AQBlSuRm9NkBCBAAu6aeESvBcAymcr0VVXcMCw== 2024-09-13T14:21:26.781 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph-authtool: stdout AQBmSuRm7HT0BRAA16BxlXVpL8y/uKa72c79Tw== 2024-09-13T14:21:27.604 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph-authtool: stdout AQBnSuRmGGdYBhAAE6yWaeByBemIyhmV0RdK9w== 2024-09-13T14:21:27.605 INFO:teuthology.orchestra.run.smithi154.stdout:Creating initial monmap... 2024-09-13T14:21:28.460 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:21:28.460 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout:monmaptool for a [v2:172.21.15.154:3300,v1:172.21.15.154:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout:setting min_mon_release = pacific 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/monmaptool: set fsid to 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:21:28.461 INFO:teuthology.orchestra.run.smithi154.stdout:Creating mon... 2024-09-13T14:21:29.443 INFO:teuthology.orchestra.run.smithi154.stdout:create mon.a on 2024-09-13T14:21:29.650 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-13T14:21:29.792 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-13T14:21:29.974 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-447b0cd2-71db-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-447b0cd2-71db-11ef-bceb-c7b262605968.target. 2024-09-13T14:21:29.974 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-447b0cd2-71db-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-447b0cd2-71db-11ef-bceb-c7b262605968.target. 2024-09-13T14:21:30.211 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a 2024-09-13T14:21:30.211 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Failed to reset failed state of unit ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-13T14:21:30.385 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-447b0cd2-71db-11ef-bceb-c7b262605968.target.wants/ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-447b0cd2-71db-11ef-bceb-c7b262605968@.service. 2024-09-13T14:21:30.445 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 systemd[1]: Starting Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:21:30.697 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 podman[28605]: 2024-09-13 14:21:30.539690925 +0000 UTC m=+0.021385614 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph:reef 2024-09-13T14:21:30.697 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 podman[28605]: 2024-09-13 14:21:30.658559858 +0000 UTC m=+0.140254544 container create 24320b91c7ac889f234062acde5756a3afe6fa2ca4579f6556bb970143055678 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-13T14:21:30.953 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 podman[28605]: 2024-09-13 14:21:30.869353935 +0000 UTC m=+0.351048622 container init 24320b91c7ac889f234062acde5756a3afe6fa2ca4579f6556bb970143055678 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-2c01458, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:21:30.953 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 podman[28605]: 2024-09-13 14:21:30.872666897 +0000 UTC m=+0.354361584 container start 24320b91c7ac889f234062acde5756a3afe6fa2ca4579f6556bb970143055678 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-13T14:21:30.953 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 ceph-mon[28632]: mkfs 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:30.953 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 ceph-mon[28632]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:21:30.967 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:21:30.967 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout disabled 2024-09-13T14:21:30.975 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:21:30.975 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout inactive 2024-09-13T14:21:30.975 INFO:teuthology.orchestra.run.smithi154.stdout:firewalld.service is not enabled 2024-09-13T14:21:30.975 INFO:teuthology.orchestra.run.smithi154.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:21:30.975 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for mon to start... 2024-09-13T14:21:30.976 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for mon... 2024-09-13T14:21:31.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 bash[28605]: 24320b91c7ac889f234062acde5756a3afe6fa2ca4579f6556bb970143055678 2024-09-13T14:21:31.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:30 smithi154 systemd[1]: Started Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:21:32.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:31 smithi154 ceph-mon[28632]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:21:32.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:31 smithi154 ceph-mon[28632]: monmap e1: 1 mons at {a=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:21:32.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:31 smithi154 ceph-mon[28632]: fsmap 2024-09-13T14:21:32.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:31 smithi154 ceph-mon[28632]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:21:32.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:31 smithi154 ceph-mon[28632]: mgrmap e1: no daemons active 2024-09-13T14:21:32.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:31 smithi154 ceph-mon[28632]: from='client.? 172.21.15.154:0/597606290' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout cluster: 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout id: 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout services: 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.526946s) 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-13T14:21:32.015 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout data: 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout pgs: 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:mon is available 2024-09-13T14:21:32.016 INFO:teuthology.orchestra.run.smithi154.stdout:Assimilating anything we can from ceph.conf... 2024-09-13T14:21:34.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:34 smithi154 ceph-mon[28632]: from='client.? 172.21.15.154:0/1242266' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:21:34.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:34 smithi154 ceph-mon[28632]: from='client.? 172.21.15.154:0/1242266' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-13T14:21:41.098 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:41.099 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:21:41.099 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout fsid = 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:41.099 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.154:3300,v1:172.21.15.154:6789] 2024-09-13T14:21:41.099 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:21:41.099 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:21:41.099 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:21:41.100 INFO:teuthology.orchestra.run.smithi154.stdout:Generating new minimal ceph.conf... 2024-09-13T14:21:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:41 smithi154 ceph-mon[28632]: from='client.? 172.21.15.154:0/3974253172' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:21:42.488 INFO:teuthology.orchestra.run.smithi154.stdout:Restarting the monitor... 2024-09-13T14:21:42.756 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:42 smithi154 systemd[1]: Stopping Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:21:43.096 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:42 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[28628]: 2024-09-13T14:21:42.754+0000 7f1bc4fe0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:21:43.097 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:42 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[28628]: 2024-09-13T14:21:42.754+0000 7f1bc4fe0640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-13T14:21:43.097 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:42 smithi154 podman[28875]: 2024-09-13 14:21:42.897897188 +0000 UTC m=+0.250279505 container died 24320b91c7ac889f234062acde5756a3afe6fa2ca4579f6556bb970143055678 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:21:43.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 podman[28875]: 2024-09-13 14:21:43.204042195 +0000 UTC m=+0.556424511 container cleanup 24320b91c7ac889f234062acde5756a3afe6fa2ca4579f6556bb970143055678 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux ) 2024-09-13T14:21:43.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 bash[28875]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:21:43.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 podman[28887]: 2024-09-13 14:21:43.39674671 +0000 UTC m=+0.499993326 container remove 24320b91c7ac889f234062acde5756a3afe6fa2ca4579f6556bb970143055678 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=reef-2c01458, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T14:21:43.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:21:43.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 systemd[1]: Stopped Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:21:43.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 systemd[1]: Starting Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:21:43.940 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 podman[28981]: 2024-09-13 14:21:43.822865457 +0000 UTC m=+0.105708677 container create d8d448f265b63d0d0361fbb76de9c445c0acf7cb21d3bb1ba45b8b5142d56317 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-13T14:21:43.940 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:43 smithi154 podman[28981]: 2024-09-13 14:21:43.733074317 +0000 UTC m=+0.015917546 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph:reef 2024-09-13T14:21:44.130 INFO:teuthology.orchestra.run.smithi154.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-13T14:21:44.345 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 podman[28981]: 2024-09-13 14:21:44.041914941 +0000 UTC m=+0.324758158 container init d8d448f265b63d0d0361fbb76de9c445c0acf7cb21d3bb1ba45b8b5142d56317 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T14:21:44.345 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 podman[28981]: 2024-09-13 14:21:44.045119036 +0000 UTC m=+0.327962253 container start d8d448f265b63d0d0361fbb76de9c445c0acf7cb21d3bb1ba45b8b5142d56317 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T14:21:44.345 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:21:44.345 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable), process ceph-mon, pid 2 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: pidfile_write: ignore empty --pid-file 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: load: jerasure load: lrc 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Git sha 0 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Compile date 2024-09-12 19:46:18 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: DB SUMMARY 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: DB Session ID: YKD5BVHVIL5PGNTASCBF 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:21:44.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:21:44.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-13T14:21:44.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-13T14:21:44.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84546 ; 2024-09-13T14:21:44.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:21:44.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:21:44.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:21:44.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:21:44.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:21:44.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:21:44.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.env: 0x55aafb9adbc0 2024-09-13T14:21:44.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:21:44.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.info_log: 0x55aafd362be0 2024-09-13T14:21:44.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:21:44.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.statistics: (nil) 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.use_fsync: 0 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:21:44.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.db_log_dir: 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.wal_dir: 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:21:44.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.write_buffer_manager: 0x55aafd370320 2024-09-13T14:21:44.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:21:44.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.unordered_write: 0 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.row_cache: None 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.wal_filter: None 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:21:44.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.wal_compression: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:21:44.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:21:44.358 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_open_files: -1 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:21:44.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Compression algorithms supported: 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kZSTD supported: 0 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:21:44.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.merge_operator: 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_filter: None 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aafd3627e0) 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout: index_type: 0 2024-09-13T14:21:44.361 INFO:journalctl@ceph.mon.a.smithi154.stdout: data_block_index_type: 0 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: index_shortening: 1 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: checksum: 4 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: no_block_cache: 0 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache: 0x55aafd357350 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache_options: 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: capacity : 536870912 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: num_shard_bits : 4 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: strict_capacity_limit : 0 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache_compressed: (nil) 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: persistent_cache: (nil) 2024-09-13T14:21:44.362 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_size: 4096 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_size_deviation: 10 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_restart_interval: 16 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: index_block_restart_interval: 1 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: metadata_block_size: 4096 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: partition_filters: 0 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: use_delta_encoding: 1 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: filter_policy: bloomfilter 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: whole_key_filtering: 1 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: verify_compression: 0 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: format_version: 5 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: enable_index_compression: 1 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_align: 0 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: max_auto_readahead_size: 262144 2024-09-13T14:21:44.363 INFO:journalctl@ceph.mon.a.smithi154.stdout: prepopulate_block_cache: 0 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression: NoCompression 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:21:44.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.num_levels: 7 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:21:44.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:21:44.366 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:21:44.367 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:21:44.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:21:44.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.ttl: 2592000 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:21:44.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 296d8068-e1f6-4def-acd2-8bd759767a90 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237304070481, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237304071335, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81096, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 256, "table_properties": {"data_size": 79246, "index_size": 234, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10267, "raw_average_key_size": 47, "raw_value_size": 73447, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 215, "num_filter_entries": 215, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726237304, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "296d8068-e1f6-4def-acd2-8bd759767a90", "db_session_id": "YKD5BVHVIL5PGNTASCBF", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726237304071430, "job": 1, "event": "recovery_finished"} 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55aafd382e00 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: DB pointer 0x55aafd476000 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:21:44.371 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** DB Stats ** 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** Compaction Stats [default] ** 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:21:44.372 INFO:journalctl@ceph.mon.a.smithi154.stdout: L0 2/0 81.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: Sum 2/0 81.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** Compaction Stats [default] ** 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:21:44.373 INFO:journalctl@ceph.mon.a.smithi154.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative compaction: 0.00 GB write, 26.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval compaction: 0.00 GB write, 26.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: Block cache BinnedLRUCache@0x55aafd357350#2 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-09-13T14:21:44.374 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: starting mon.a rank 0 at public addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] at bind addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???) e1 preinit fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???).mds e1 new map 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???).mds e1 print_map 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout: e1 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout: legacy client fscid: -1 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:21:44.375 INFO:journalctl@ceph.mon.a.smithi154.stdout: No filesystems configured 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 bash[28981]: d8d448f265b63d0d0361fbb76de9c445c0acf7cb21d3bb1ba45b8b5142d56317 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 systemd[1]: Started Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: monmap e1: 1 mons at {a=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: fsmap 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T14:21:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:44 smithi154 ceph-mon[29001]: mgrmap e1: no daemons active 2024-09-13T14:21:45.135 INFO:teuthology.orchestra.run.smithi154.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-13T14:21:45.148 INFO:teuthology.orchestra.run.smithi154.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:21:45.148 INFO:teuthology.orchestra.run.smithi154.stdout:Creating mgr... 2024-09-13T14:21:45.149 INFO:teuthology.orchestra.run.smithi154.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-13T14:21:45.149 INFO:teuthology.orchestra.run.smithi154.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-13T14:21:45.371 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y 2024-09-13T14:21:45.371 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Failed to reset failed state of unit ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-13T14:21:45.525 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-447b0cd2-71db-11ef-bceb-c7b262605968.target.wants/ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-447b0cd2-71db-11ef-bceb-c7b262605968@.service. 2024-09-13T14:21:45.880 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:45 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1497551086' entity='client.admin' 2024-09-13T14:21:46.134 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:21:46.134 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout disabled 2024-09-13T14:21:46.141 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:21:46.141 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout inactive 2024-09-13T14:21:46.141 INFO:teuthology.orchestra.run.smithi154.stdout:firewalld.service is not enabled 2024-09-13T14:21:46.141 INFO:teuthology.orchestra.run.smithi154.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T14:21:46.150 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:21:46.150 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout disabled 2024-09-13T14:21:46.150 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:46 smithi154 systemd[1]: Started Ceph mgr.y for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:21:46.157 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:21:46.157 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout inactive 2024-09-13T14:21:46.158 INFO:teuthology.orchestra.run.smithi154.stdout:firewalld.service is not enabled 2024-09-13T14:21:46.158 INFO:teuthology.orchestra.run.smithi154.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-13T14:21:46.158 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for mgr to start... 2024-09-13T14:21:46.158 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for mgr... 2024-09-13T14:21:46.402 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:46 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:46.149+0000 7fa4f97d2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:21:46.403 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:46 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:46.209+0000 7fa4f97d2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:21:46.403 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:46 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:46.359+0000 7fa4f97d2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:21:46.713 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:46 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:46.428+0000 7fa4f97d2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:21:46.714 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:46 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:46.574+0000 7fa4f97d2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:21:46.714 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:46 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:46.712+0000 7fa4f97d2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:21:47.345 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:47 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:47.022+0000 7fa4f97d2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:21:47.345 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:47 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:47.081+0000 7fa4f97d2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:21:47.345 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:47 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:47.262+0000 7fa4f97d2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:21:47.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:47 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2695438336' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:21:48.394 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:48 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:48.147+0000 7fa4f97d2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:21:48.394 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:48 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:48.213+0000 7fa4f97d2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:21:49.193 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:48 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:48.930+0000 7fa4f97d2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:21:49.193 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.046+0000 7fa4f97d2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:21:49.193 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.191+0000 7fa4f97d2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:21:49.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.255+0000 7fa4f97d2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:21:49.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.408+0000 7fa4f97d2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:21:49.771 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.479+0000 7fa4f97d2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:21:49.771 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.542+0000 7fa4f97d2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:21:49.771 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.702+0000 7fa4f97d2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:21:49.771 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.770+0000 7fa4f97d2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:21:50.046 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.839+0000 7fa4f97d2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:21:50.046 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.923+0000 7fa4f97d2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:21:50.046 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:49 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:49.984+0000 7fa4f97d2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:21:50.046 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:50.044+0000 7fa4f97d2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:21:50.153 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:50.153 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:21:50.153 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "fsid": "447b0cd2-71db-11ef-bceb-c7b262605968", 2024-09-13T14:21:50.153 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:21:50.153 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:21:50.153 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:21:50.153 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-13T14:21:50.154 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:21:50.155 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:21:50.156 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:21:30.909742+0000", 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:50.157 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:21:50.158 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:21:50.158 INFO:teuthology.orchestra.run.smithi154.stdout:mgr not available, waiting (1/15)... 2024-09-13T14:21:51.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:50.830+0000 7fa4f97d2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:21:51.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: Activating manager daemon y 2024-09-13T14:21:51.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: mgrmap e2: y(active, starting, since 0.0031518s) 2024-09-13T14:21:51.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: Manager daemon y is now available 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' 2024-09-13T14:21:51.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:50 smithi154 ceph-mon[29001]: from='mgr.14100 172.21.15.154:0/491100399' entity='mgr.y' 2024-09-13T14:21:53.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:52 smithi154 ceph-mon[29001]: mgrmap e3: y(active, since 1.00726s) 2024-09-13T14:21:53.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:52 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2940259698' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T14:21:53.423 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:53.423 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:21:53.423 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "fsid": "447b0cd2-71db-11ef-bceb-c7b262605968", 2024-09-13T14:21:53.423 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T14:21:53.423 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T14:21:53.423 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T14:21:53.423 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 0 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "a" 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-13T14:21:53.424 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T14:21:53.425 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:53.425 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T14:21:53.425 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:53.425 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T14:21:53.425 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T14:21:53.425 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T14:21:53.425 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T14:21:53.428 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T14:21:53.429 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T14:21:53.430 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T14:21:53.431 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T14:21:53.431 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T14:21:53.431 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout ], 2024-09-13T14:21:53.431 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:21:53.431 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:53.431 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T14:21:53.432 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T14:21:53.432 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T14:21:30.909742+0000", 2024-09-13T14:21:53.432 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T14:21:53.432 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout }, 2024-09-13T14:21:53.432 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T14:21:53.433 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:21:53.433 INFO:teuthology.orchestra.run.smithi154.stdout:mgr is available 2024-09-13T14:21:54.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:53 smithi154 ceph-mon[29001]: mgrmap e4: y(active, since 2s) 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout fsid = 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:54.579 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T14:21:54.580 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T14:21:54.580 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 2024-09-13T14:21:54.580 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T14:21:54.580 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T14:21:54.580 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T14:21:54.580 INFO:teuthology.orchestra.run.smithi154.stdout:Enabling cephadm module... 2024-09-13T14:21:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:54 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3897052413' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T14:21:56.148 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:55 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1684562193' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-13T14:21:56.148 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setuser ceph since I am not root 2024-09-13T14:21:56.148 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setgroup ceph since I am not root 2024-09-13T14:21:56.148 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:55.983+0000 7fcf66e2d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:21:56.148 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:56.039+0000 7fcf66e2d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:21:56.400 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:56.189+0000 7fcf66e2d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:21:56.400 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:56.254+0000 7fcf66e2d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:21:56.400 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:56.398+0000 7fcf66e2d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:21:56.661 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:56.549+0000 7fcf66e2d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:21:56.933 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1684562193' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-13T14:21:56.934 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-mon[29001]: mgrmap e5: y(active, since 5s) 2024-09-13T14:21:56.934 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:56.870+0000 7fcf66e2d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:21:57.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:56.932+0000 7fcf66e2d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:21:57.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:57.122+0000 7fcf66e2d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:21:57.644 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:21:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-13T14:21:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:21:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:21:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:21:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:21:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for the mgr to restart... 2024-09-13T14:21:57.645 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for mgr epoch 5... 2024-09-13T14:21:57.952 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:21:57 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/421669110' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:21:58.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:58.005+0000 7fcf66e2d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:21:58.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:58.075+0000 7fcf66e2d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:21:59.030 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:58.775+0000 7fcf66e2d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:21:59.030 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:58.888+0000 7fcf66e2d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:21:59.303 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.028+0000 7fcf66e2d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:21:59.303 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.089+0000 7fcf66e2d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:21:59.303 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.234+0000 7fcf66e2d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:21:59.304 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.301+0000 7fcf66e2d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:21:59.578 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.359+0000 7fcf66e2d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:21:59.578 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.513+0000 7fcf66e2d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:21:59.578 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.576+0000 7fcf66e2d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:21:59.830 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.641+0000 7fcf66e2d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:21:59.830 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.721+0000 7fcf66e2d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:21:59.830 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.779+0000 7fcf66e2d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:22:00.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:21:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:21:59.841+0000 7fcf66e2d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:22:00.675 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:00.620+0000 7fcf66e2d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:22:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: Active manager daemon y restarted 2024-09-13T14:22:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: Activating manager daemon y 2024-09-13T14:22:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: osdmap e2: 0 total, 0 up, 0 in 2024-09-13T14:22:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: mgrmap e6: y(active, starting, since 0.00379423s) 2024-09-13T14:22:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: Manager daemon y is now available 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:00 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:01.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:01 smithi154 ceph-mon[29001]: Found migration_current of "None". Setting to last migration. 2024-09-13T14:22:01.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:01 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:01.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:01 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:01.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:01 smithi154 ceph-mon[29001]: mgrmap e7: y(active, since 1.00564s) 2024-09-13T14:22:02.200 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:22:02.200 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-13T14:22:02.200 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:22:02.200 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:22:02.200 INFO:teuthology.orchestra.run.smithi154.stdout:mgr epoch 5 is available 2024-09-13T14:22:02.200 INFO:teuthology.orchestra.run.smithi154.stdout:Setting orchestrator backend to cephadm... 2024-09-13T14:22:02.451 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:02.220+0000 7fceceb0d640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:22:02] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:02.451 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: Traceback (most recent call last): 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._connections.run(self.expiration_interval) 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._run(expiration_interval) 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: s = self.context.wrap_socket( 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:02.452 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: return self.sslsocket_class._create( 2024-09-13T14:22:02.453 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:02.453 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self.do_handshake() 2024-09-13T14:22:02.453 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:02.453 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._sslobj.do_handshake() 2024-09-13T14:22:02.453 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:02.453 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-mon[29001]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:22:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-mon[29001]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:22:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:02 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:03.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:02] ENGINE Bus STARTING 2024-09-13T14:22:03.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:02] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:22:03.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:02] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:22:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:02] ENGINE Bus STARTED 2024-09-13T14:22:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:02] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:22:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:22:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:22:03.972 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:03.972 INFO:journalctl@ceph.mon.a.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:03.972 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:03.972 INFO:journalctl@ceph.mon.a.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:03.972 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:03.972 INFO:journalctl@ceph.mon.a.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout: self.do_handshake() 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: mgrmap e8: y(active, since 2s) 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:03.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:03.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:03 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:04.522 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-13T14:22:04.522 INFO:teuthology.orchestra.run.smithi154.stdout:Generating ssh key... 2024-09-13T14:22:05.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:04 smithi154 ceph-mon[29001]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:05.775 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: Generating public/private rsa key pair. 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: Your identification has been saved in /tmp/tmp58sbwv0h/key 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: Your public key has been saved in /tmp/tmp58sbwv0h/key.pub 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: The key fingerprint is: 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: SHA256:q/2w4CKU5Rf+XsM0QrpjvazQzkMjvw98CRDcBCRpvns ceph-447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: The key's randomart image is: 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: +---[RSA 3072]----+ 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | .+++. | 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | o.... | 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | o . . | 2024-09-13T14:22:05.776 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | . ...o | 2024-09-13T14:22:05.777 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | = .o.S o | 2024-09-13T14:22:05.777 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | + oo=+ * . | 2024-09-13T14:22:05.777 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | . ..=Oo* + | 2024-09-13T14:22:05.777 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | o E*oO.= . | 2024-09-13T14:22:05.777 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: | o .B*B.. | 2024-09-13T14:22:05.777 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: +----[SHA256]-----+ 2024-09-13T14:22:06.784 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:06 smithi154 ceph-mon[29001]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:06.784 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:06 smithi154 ceph-mon[29001]: Generating ssh key... 2024-09-13T14:22:06.784 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:06 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:06.784 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:06 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:07.353 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpxq7CpmqGBY9qvW3IsJxOU26USFP+j98Z3gkHVzL4HJyVYKAcV6jHRrkw78JRu2Dijj9mojFrXjdimljGF0Uc6WxboqG+WvwD7/dgt7zpDDOGhWTliMQt4CF8kJ2a/cbzJwLRvs+61oYzeUSiIwGtfbVXY13U7YEniwHOoJ4FwRwHIp1dUdAG3+v6Mx7tPipO3sfXMADt/eM7Poo+vHXUgVicwNvAFDXn8/yYD1l1s+s9o0kicpqOrxLZwZI62kAvHy+pjryfKgWjzEHlrrgW8Nn3wJnuqcrpYTD5q8T0VR+j1Bl1cYE0EF3tMi/xjqt+a8eIOmIaTtnK/Do7inM1qkNc8jeFCFW1yftABNk8v141E3dNSjoXMXSpF0L/VrGDQdP2zShu+l0vIfeBdQN7d0fzmQ4V/HOxx8nRS5lIoX4hRLbEAp6y9iRkMpJtzAyaiWx6Wi6bJReGNbWtXl58YEI+CVLVvmknyvJP3HJv4RntybTFF2oCQD/P3RaOOCU= ceph-447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:22:07.353 INFO:teuthology.orchestra.run.smithi154.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-13T14:22:07.353 INFO:teuthology.orchestra.run.smithi154.stdout:Adding key to root@localhost authorized_keys... 2024-09-13T14:22:07.357 INFO:teuthology.orchestra.run.smithi154.stdout:Adding host smithi154... 2024-09-13T14:22:07.607 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:07 smithi154 ceph-mon[29001]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:08 smithi154 ceph-mon[29001]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "addr": "172.21.15.154", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:08.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:08 smithi154 ceph-mon[29001]: Deploying cephadm binary to smithi154 2024-09-13T14:22:11.135 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout Added host 'smithi154' with addr '172.21.15.154' 2024-09-13T14:22:11.136 INFO:teuthology.orchestra.run.smithi154.stdout:Deploying unmanaged mon service... 2024-09-13T14:22:11.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:11 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:11.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:11 smithi154 ceph-mon[29001]: Added host smithi154 2024-09-13T14:22:11.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:11 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:12.478 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-13T14:22:12.479 INFO:teuthology.orchestra.run.smithi154.stdout:Deploying unmanaged mgr service... 2024-09-13T14:22:12.821 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:12 smithi154 ceph-mon[29001]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:12.821 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:12 smithi154 ceph-mon[29001]: Saving service mon spec with placement count:5 2024-09-13T14:22:12.821 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:12 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:14.196 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:14 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:14.196 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:14 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:14.196 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:14 smithi154 ceph-mon[29001]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:14.196 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:14 smithi154 ceph-mon[29001]: Saving service mgr spec with placement count:2 2024-09-13T14:22:14.196 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:14 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:14.702 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-13T14:22:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:16 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:16 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1414046127' entity='client.admin' 2024-09-13T14:22:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:16 smithi154 ceph-mon[29001]: from='mgr.14116 172.21.15.154:0/3386715100' entity='mgr.y' 2024-09-13T14:22:17.107 INFO:teuthology.orchestra.run.smithi154.stdout:Enabling the dashboard module... 2024-09-13T14:22:17.652 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:17 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1783985329' entity='client.admin' 2024-09-13T14:22:18.625 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:18 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3375660195' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-13T14:22:18.625 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:18 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setuser ceph since I am not root 2024-09-13T14:22:18.625 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:18 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setgroup ceph since I am not root 2024-09-13T14:22:18.625 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:18 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:18.623+0000 7fa1748e4040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:22:18.901 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:18 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:18.681+0000 7fa1748e4040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:22:18.901 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:18 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:18.833+0000 7fa1748e4040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:22:18.901 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:18 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:18.900+0000 7fa1748e4040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:22:19.194 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:19 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:19.046+0000 7fa1748e4040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:22:19.194 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:19 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:19.192+0000 7fa1748e4040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:22:19.729 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:19 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3375660195' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-13T14:22:19.729 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:19 smithi154 ceph-mon[29001]: mgrmap e9: y(active, since 17s) 2024-09-13T14:22:19.729 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:19 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:19.511+0000 7fa1748e4040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:22:19.729 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:19 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:19.574+0000 7fa1748e4040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:22:20.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:19 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:19.764+0000 7fa1748e4040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:22:20.302 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:22:20.302 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-13T14:22:20.302 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T14:22:20.302 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-13T14:22:20.303 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T14:22:20.303 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:22:20.303 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for the mgr to restart... 2024-09-13T14:22:20.303 INFO:teuthology.orchestra.run.smithi154.stdout:Waiting for mgr epoch 9... 2024-09-13T14:22:20.527 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:20 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/164585546' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T14:22:20.793 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:20 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:20.661+0000 7fa1748e4040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:22:20.793 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:20 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:20.732+0000 7fa1748e4040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:22:21.738 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:21.473+0000 7fa1748e4040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:22:21.738 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:21.591+0000 7fa1748e4040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:22:22.017 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:21.736+0000 7fa1748e4040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:22:22.017 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:21.800+0000 7fa1748e4040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:22:22.017 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:21.947+0000 7fa1748e4040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:22:22.018 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.013+0000 7fa1748e4040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:22:22.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.072+0000 7fa1748e4040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:22:22.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.229+0000 7fa1748e4040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:22:22.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.292+0000 7fa1748e4040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:22:22.557 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.358+0000 7fa1748e4040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:22:22.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.439+0000 7fa1748e4040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:22:22.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.498+0000 7fa1748e4040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:22:22.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:22.556+0000 7fa1748e4040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:22:23.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:23.329+0000 7fa1748e4040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:22:23.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: Active manager daemon y restarted 2024-09-13T14:22:23.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: Activating manager daemon y 2024-09-13T14:22:23.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: osdmap e3: 0 total, 0 up, 0 in 2024-09-13T14:22:23.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: mgrmap e10: y(active, starting, since 0.00299862s) 2024-09-13T14:22:23.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: Manager daemon y is now available 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:22:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:22:24.961 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout { 2024-09-13T14:22:24.961 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-13T14:22:24.962 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T14:22:24.962 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout } 2024-09-13T14:22:24.962 INFO:teuthology.orchestra.run.smithi154.stdout:mgr epoch 9 is available 2024-09-13T14:22:24.962 INFO:teuthology.orchestra.run.smithi154.stdout:Generating a dashboard self-signed certificate... 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:24.840+0000 7fa0dad6e640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:22:24] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: Traceback (most recent call last): 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._connections.run(self.expiration_interval) 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._run(expiration_interval) 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:25.082 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: s = self.context.wrap_socket( 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: return self.sslsocket_class._create( 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self.do_handshake() 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._sslobj.do_handshake() 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:25.083 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:22:24 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:22:25.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:25 smithi154 ceph-mon[29001]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T14:22:25.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:25 smithi154 ceph-mon[29001]: mgrmap e11: y(active, since 1.00563s) 2024-09-13T14:22:25.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:25 smithi154 ceph-mon[29001]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T14:22:26.168 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-13T14:22:26.168 INFO:teuthology.orchestra.run.smithi154.stdout:Creating initial admin user... 2024-09-13T14:22:26.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:24] ENGINE Bus STARTING 2024-09-13T14:22:26.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:24] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:24] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:24] ENGINE Bus STARTED 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: [13/Sep/2024:14:22:24] ENGINE Error in HTTPServer.serve 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:22:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: self.do_handshake() 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: mgrmap e12: y(active, since 2s) 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:26.621 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:26.622 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:26.622 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:26.622 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:26 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:28.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:28 smithi154 ceph-mon[29001]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:28.255 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:28.257 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$hP5VmNWtWth6VzzDlYlu3OTMg40pLsju4wNDC0Qh5ruZmhxbJ.48C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726237347, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-13T14:22:28.257 INFO:teuthology.orchestra.run.smithi154.stdout:Fetching dashboard port number... 2024-09-13T14:22:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:29 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2600031565' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-13T14:22:29.464 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stdout 8443 2024-09-13T14:22:29.474 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T14:22:29.474 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout disabled 2024-09-13T14:22:29.481 INFO:teuthology.orchestra.run.smithi154.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T14:22:29.481 INFO:teuthology.orchestra.run.smithi154.stdout:systemctl: stdout inactive 2024-09-13T14:22:29.481 INFO:teuthology.orchestra.run.smithi154.stdout:firewalld.service is not enabled 2024-09-13T14:22:29.481 INFO:teuthology.orchestra.run.smithi154.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-13T14:22:29.483 INFO:teuthology.orchestra.run.smithi154.stdout:Ceph Dashboard is now available at: 2024-09-13T14:22:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: URL: https://smithi154.front.sepia.ceph.com:8443/ 2024-09-13T14:22:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: User: admin 2024-09-13T14:22:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: Password: i4gwj8fmtr 2024-09-13T14:22:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:29.484 INFO:teuthology.orchestra.run.smithi154.stdout:Saving cluster configuration to /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config directory 2024-09-13T14:22:29.484 INFO:teuthology.orchestra.run.smithi154.stdout:Enabling autotune for osd_memory_target 2024-09-13T14:22:31.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:31 smithi154 ceph-mon[29001]: mgrmap e13: y(active, since 6s) 2024-09-13T14:22:31.828 INFO:teuthology.orchestra.run.smithi154.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-13T14:22:31.828 INFO:teuthology.orchestra.run.smithi154.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-13T14:22:31.828 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.828 INFO:teuthology.orchestra.run.smithi154.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:31.828 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.828 INFO:teuthology.orchestra.run.smithi154.stdout:Or, if you are only running a single cluster on this host: 2024-09-13T14:22:31.828 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: ceph telemetry on 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout:For more information see: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:31.829 INFO:teuthology.orchestra.run.smithi154.stdout:Bootstrap complete. 2024-09-13T14:22:31.864 INFO:tasks.cephadm:Fetching config... 2024-09-13T14:22:31.864 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:22:31.864 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-13T14:22:31.880 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-13T14:22:31.880 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:22:31.881 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-13T14:22:31.936 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-13T14:22:31.937 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:22:31.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-13T14:22:32.002 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-13T14:22:32.002 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:22:32.002 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-13T14:22:32.059 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-13T14:22:32.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpxq7CpmqGBY9qvW3IsJxOU26USFP+j98Z3gkHVzL4HJyVYKAcV6jHRrkw78JRu2Dijj9mojFrXjdimljGF0Uc6WxboqG+WvwD7/dgt7zpDDOGhWTliMQt4CF8kJ2a/cbzJwLRvs+61oYzeUSiIwGtfbVXY13U7YEniwHOoJ4FwRwHIp1dUdAG3+v6Mx7tPipO3sfXMADt/eM7Poo+vHXUgVicwNvAFDXn8/yYD1l1s+s9o0kicpqOrxLZwZI62kAvHy+pjryfKgWjzEHlrrgW8Nn3wJnuqcrpYTD5q8T0VR+j1Bl1cYE0EF3tMi/xjqt+a8eIOmIaTtnK/Do7inM1qkNc8jeFCFW1yftABNk8v141E3dNSjoXMXSpF0L/VrGDQdP2zShu+l0vIfeBdQN7d0fzmQ4V/HOxx8nRS5lIoX4hRLbEAp6y9iRkMpJtzAyaiWx6Wi6bJReGNbWtXl58YEI+CVLVvmknyvJP3HJv4RntybTFF2oCQD/P3RaOOCU= ceph-447b0cd2-71db-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:22:32.134 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpxq7CpmqGBY9qvW3IsJxOU26USFP+j98Z3gkHVzL4HJyVYKAcV6jHRrkw78JRu2Dijj9mojFrXjdimljGF0Uc6WxboqG+WvwD7/dgt7zpDDOGhWTliMQt4CF8kJ2a/cbzJwLRvs+61oYzeUSiIwGtfbVXY13U7YEniwHOoJ4FwRwHIp1dUdAG3+v6Mx7tPipO3sfXMADt/eM7Poo+vHXUgVicwNvAFDXn8/yYD1l1s+s9o0kicpqOrxLZwZI62kAvHy+pjryfKgWjzEHlrrgW8Nn3wJnuqcrpYTD5q8T0VR+j1Bl1cYE0EF3tMi/xjqt+a8eIOmIaTtnK/Do7inM1qkNc8jeFCFW1yftABNk8v141E3dNSjoXMXSpF0L/VrGDQdP2zShu+l0vIfeBdQN7d0fzmQ4V/HOxx8nRS5lIoX4hRLbEAp6y9iRkMpJtzAyaiWx6Wi6bJReGNbWtXl58YEI+CVLVvmknyvJP3HJv4RntybTFF2oCQD/P3RaOOCU= ceph-447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:22:32.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpxq7CpmqGBY9qvW3IsJxOU26USFP+j98Z3gkHVzL4HJyVYKAcV6jHRrkw78JRu2Dijj9mojFrXjdimljGF0Uc6WxboqG+WvwD7/dgt7zpDDOGhWTliMQt4CF8kJ2a/cbzJwLRvs+61oYzeUSiIwGtfbVXY13U7YEniwHOoJ4FwRwHIp1dUdAG3+v6Mx7tPipO3sfXMADt/eM7Poo+vHXUgVicwNvAFDXn8/yYD1l1s+s9o0kicpqOrxLZwZI62kAvHy+pjryfKgWjzEHlrrgW8Nn3wJnuqcrpYTD5q8T0VR+j1Bl1cYE0EF3tMi/xjqt+a8eIOmIaTtnK/Do7inM1qkNc8jeFCFW1yftABNk8v141E3dNSjoXMXSpF0L/VrGDQdP2zShu+l0vIfeBdQN7d0fzmQ4V/HOxx8nRS5lIoX4hRLbEAp6y9iRkMpJtzAyaiWx6Wi6bJReGNbWtXl58YEI+CVLVvmknyvJP3HJv4RntybTFF2oCQD/P3RaOOCU= ceph-447b0cd2-71db-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T14:22:32.209 INFO:teuthology.orchestra.run.smithi172.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpxq7CpmqGBY9qvW3IsJxOU26USFP+j98Z3gkHVzL4HJyVYKAcV6jHRrkw78JRu2Dijj9mojFrXjdimljGF0Uc6WxboqG+WvwD7/dgt7zpDDOGhWTliMQt4CF8kJ2a/cbzJwLRvs+61oYzeUSiIwGtfbVXY13U7YEniwHOoJ4FwRwHIp1dUdAG3+v6Mx7tPipO3sfXMADt/eM7Poo+vHXUgVicwNvAFDXn8/yYD1l1s+s9o0kicpqOrxLZwZI62kAvHy+pjryfKgWjzEHlrrgW8Nn3wJnuqcrpYTD5q8T0VR+j1Bl1cYE0EF3tMi/xjqt+a8eIOmIaTtnK/Do7inM1qkNc8jeFCFW1yftABNk8v141E3dNSjoXMXSpF0L/VrGDQdP2zShu+l0vIfeBdQN7d0fzmQ4V/HOxx8nRS5lIoX4hRLbEAp6y9iRkMpJtzAyaiWx6Wi6bJReGNbWtXl58YEI+CVLVvmknyvJP3HJv4RntybTFF2oCQD/P3RaOOCU= ceph-447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:22:32.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-13T14:22:32.415 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:32 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1630422427' entity='client.admin' 2024-09-13T14:22:35.268 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-13T14:22:35.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-13T14:22:35.513 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:35 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1274086472' entity='client.admin' 2024-09-13T14:22:35.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:35.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:35.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:35.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:35.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:35.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:35.725 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:38.678 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi172 2024-09-13T14:22:38.678 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:22:38.678 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.conf 2024-09-13T14:22:38.698 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:22:38.698 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:38.755 INFO:tasks.cephadm:Adding host smithi172 to orchestrator... 2024-09-13T14:22:38.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch host add smithi172 2024-09-13T14:22:38.954 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:39.063 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:39.064 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:39.994 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:39 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:41.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:41 smithi154 ceph-mon[29001]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:42 smithi154 ceph-mon[29001]: Deploying cephadm binary to smithi172 2024-09-13T14:22:43.312 INFO:teuthology.orchestra.run.smithi154.stdout:Added host 'smithi172' with addr '172.21.15.172' 2024-09-13T14:22:43.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-13T14:22:44.124 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:44.540 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:44.540 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:44 smithi154 ceph-mon[29001]: Added host smithi172 2024-09-13T14:22:44.540 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:44.540 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:44 smithi154 ceph-mon[29001]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:22:44.540 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:44.540 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:45.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:45 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:45.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:45 smithi154 ceph-mon[29001]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:22:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:22:45.818 INFO:teuthology.orchestra.run.smithi154.stdout:[{"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}, {"addr": "172.21.15.172", "hostname": "smithi172", "labels": [], "status": ""}] 2024-09-13T14:22:46.402 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-13T14:22:46.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-13T14:22:46.594 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:22:46.714 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:46 smithi154 ceph-mon[29001]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:22:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:49.214 INFO:teuthology.orchestra.run.smithi154.stderr:adjusted tunables profile to default 2024-09-13T14:22:49.464 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:49 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:22:49.464 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:49 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:22:49.464 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:49 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/4003364751' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T14:22:49.860 INFO:tasks.cephadm:Adding mon.a on smithi154 2024-09-13T14:22:49.860 INFO:tasks.cephadm:Adding mon.c on smithi154 2024-09-13T14:22:49.860 INFO:tasks.cephadm:Adding mon.b on smithi172 2024-09-13T14:22:49.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi154:172.21.15.154=a;smithi154:[v2:172.21.15.154:3301,v1:172.21.15.154:6790]=c;smithi172:172.21.15.172=b' 2024-09-13T14:22:50.062 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:50 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/4003364751' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T14:22:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:50 smithi154 ceph-mon[29001]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:22:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:50 smithi154 ceph-mon[29001]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:22:51.673 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mon update... 2024-09-13T14:22:52.230 DEBUG:teuthology.orchestra.run.smithi154:mon.c> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.c.service 2024-09-13T14:22:52.234 DEBUG:teuthology.orchestra.run.smithi172:mon.b> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.b.service 2024-09-13T14:22:52.236 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:22:52.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:22:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:22:52.917 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:22:53.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:53 smithi154 ceph-mon[29001]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi154:172.21.15.154=a;smithi154:[v2:172.21.15.154:3301,v1:172.21.15.154:6790]=c;smithi172:172.21.15.172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:22:53.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:53 smithi154 ceph-mon[29001]: Saving service mon spec with placement smithi154:172.21.15.154=a;smithi154:[v2:172.21.15.154:3301,v1:172.21.15.154:6790]=c;smithi172:172.21.15.172=b;count:3 2024-09-13T14:22:53.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:53 smithi154 ceph-mon[29001]: Deploying daemon mon.b on smithi172 2024-09-13T14:22:54.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:54 smithi154 ceph-mon[29001]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:22:56.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:22:56.303 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","modified":"2024-09-13T14:21:27.916625Z","created":"2024-09-13T14:21:27.916625Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T14:22:56.303 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-09-13T14:22:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:56 smithi154 ceph-mon[29001]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:22:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:22:56 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/730089972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:22:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:22:56 smithi172 systemd[1]: Started Ceph mon.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:22:58.129 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:22:58.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:22:58.345 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:23:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: mon.a calling monitor election 2024-09-13T14:23:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: mon.b calling monitor election 2024-09-13T14:23:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:23:01.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: monmap e2: 2 mons at {a=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:23:01.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: fsmap 2024-09-13T14:23:01.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:23:01.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: mgrmap e13: y(active, since 38s) 2024-09-13T14:23:01.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: overall HEALTH_OK 2024-09-13T14:23:01.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:01.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:01.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:01.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:01.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: mon.a calling monitor election 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: mon.b calling monitor election 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:02.073 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: monmap e2: 2 mons at {a=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: fsmap 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: mgrmap e13: y(active, since 38s) 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: overall HEALTH_OK 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:02.074 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:23:02.982 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":2,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","modified":"2024-09-13T14:22:56.672916Z","created":"2024-09-13T14:21:27.916625Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T14:23:02.982 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 2 2024-09-13T14:23:02.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:02 smithi172 ceph-mon[32794]: Deploying daemon mon.c on smithi154 2024-09-13T14:23:02.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:02 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:02.998 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:02 smithi154 ceph-mon[29001]: Deploying daemon mon.c on smithi154 2024-09-13T14:23:02.998 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:02 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:03.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:03 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/3979522219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:23:03.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:03 smithi172 ceph-mon[32794]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:04.092 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:03 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/3979522219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:23:04.093 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:03 smithi154 ceph-mon[29001]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:04.566 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-13T14:23:04.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T14:23:04.765 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:23:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:04 smithi154 systemd[1]: Started Ceph mon.c for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:23:10.041 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:23:10.041 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":3,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","modified":"2024-09-13T14:23:04.765616Z","created":"2024-09-13T14:21:27.916625Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3301","nonce":0},{"type":"v1","addr":"172.21.15.154:6790","nonce":0}]},"addr":"172.21.15.154:6790/0","public_addr":"172.21.15.154:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-13T14:23:10.041 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 3 2024-09-13T14:23:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: mon.a calling monitor election 2024-09-13T14:23:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: mon.b calling monitor election 2024-09-13T14:23:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: mon.c calling monitor election 2024-09-13T14:23:10.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:23:10.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: monmap e3: 3 mons at {a=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:23:10.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: fsmap 2024-09-13T14:23:10.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:23:10.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: mgrmap e13: y(active, since 46s) 2024-09-13T14:23:10.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: overall HEALTH_OK 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: mon.a calling monitor election 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: mon.b calling monitor election 2024-09-13T14:23:10.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: mon.c calling monitor election 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: monmap e3: 3 mons at {a=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: fsmap 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:23:10.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: mgrmap e13: y(active, since 46s) 2024-09-13T14:23:10.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: overall HEALTH_OK 2024-09-13T14:23:10.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:23:10.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: mon.a calling monitor election 2024-09-13T14:23:10.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:23:10.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: mon.b calling monitor election 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: mon.c calling monitor election 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: monmap e3: 3 mons at {a=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: fsmap 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T14:23:10.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: mgrmap e13: y(active, since 46s) 2024-09-13T14:23:10.246 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: overall HEALTH_OK 2024-09-13T14:23:10.246 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.246 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.246 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.246 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:10.609 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-13T14:23:10.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-13T14:23:10.912 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:11.102 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:10 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:11.102 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:10 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/222919599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:23:11.102 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:10 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:11.142 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:10 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:11.142 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:10 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/222919599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:23:11.142 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:10 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:11.143 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:10 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:11.143 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:10 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/222919599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T14:23:11.143 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:10 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:23:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:12.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:12.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:12.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:12.944 INFO:teuthology.orchestra.run.smithi154.stdout:# minimal ceph.conf for 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:23:12.945 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2024-09-13T14:23:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:23:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: mon_host = [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] 2024-09-13T14:23:13.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[29001]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[29001]: Reconfiguring daemon mon.a on smithi154 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2971805420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[39074]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[39074]: Reconfiguring daemon mon.a on smithi154 2024-09-13T14:23:13.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:12 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2971805420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:12 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:12 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:23:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:12 smithi172 ceph-mon[32794]: Reconfiguring mon.a (unknown last config time)... 2024-09-13T14:23:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:12 smithi172 ceph-mon[32794]: Reconfiguring daemon mon.a on smithi154 2024-09-13T14:23:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:12 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2971805420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:13.781 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-13T14:23:13.782 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:23:13.782 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:23:13.812 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:23:13.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:13.879 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:23:13.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T14:23:13.908 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:23:13.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:23:13.975 INFO:tasks.cephadm:Adding mgr.y on smithi154 2024-09-13T14:23:13.975 INFO:tasks.cephadm:Adding mgr.x on smithi172 2024-09-13T14:23:13.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi154=y;smithi172=x' 2024-09-13T14:23:14.204 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:23:14.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:14 smithi172 ceph-mon[32794]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:14.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:14.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:14.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[29001]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[39074]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:14.619 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:14.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:14.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:14.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:15.527 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:15 smithi172 ceph-mon[32794]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:23:15.527 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:15 smithi172 ceph-mon[32794]: Reconfiguring daemon mon.c on smithi154 2024-09-13T14:23:15.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:15 smithi154 ceph-mon[29001]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:23:15.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:15 smithi154 ceph-mon[29001]: Reconfiguring daemon mon.c on smithi154 2024-09-13T14:23:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:15 smithi154 ceph-mon[39074]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:23:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:15 smithi154 ceph-mon[39074]: Reconfiguring daemon mon.c on smithi154 2024-09-13T14:23:15.910 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mgr update... 2024-09-13T14:23:16.472 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:16 smithi172 ceph-mon[32794]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:16.473 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:16 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.473 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:16 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.473 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:16 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:16.473 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:16 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:16.473 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:16 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:16.473 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:16 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[29001]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:16.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:16.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:16.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[39074]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:16.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:16.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:23:16.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:23:16.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:16.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:16 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.393 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:23:17.393 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: Reconfiguring daemon mon.b on smithi172 2024-09-13T14:23:17.393 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi154=y;smithi172=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:17.393 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: Saving service mgr spec with placement smithi154=y;smithi172=x;count:2 2024-09-13T14:23:17.393 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:17.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:17.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:17.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.395 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:17.395 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:23:17.395 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:17.395 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:17.413 DEBUG:teuthology.orchestra.run.smithi172:mgr.x> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service 2024-09-13T14:23:17.416 DEBUG:tasks.cephadm:set 0 configs 2024-09-13T14:23:17.416 INFO:tasks.cephadm:Deploying OSDs... 2024-09-13T14:23:17.416 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:23:17.417 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:23:17.437 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:23:17.437 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-09-13T14:23:17.496 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:23:17.496 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:17.497 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-13T14:23:17.497 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:17.497 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:17.497 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 14:23:04.253230789 +0000 2024-09-13T14:23:17.497 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 14:17:52.520467355 +0000 2024-09-13T14:23:17.497 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 14:17:52.520467355 +0000 2024-09-13T14:23:17.497 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 14:17:52.520467355 +0000 2024-09-13T14:23:17.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:23:17.565 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T14:23:17.565 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T14:23:17.565 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000229276 s, 2.2 MB/s 2024-09-13T14:23:17.566 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:23:17.625 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-09-13T14:23:17.682 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 970 Links: 1 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 14:23:04.253230789 +0000 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 14:17:52.520467355 +0000 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 14:17:52.520467355 +0000 2024-09-13T14:23:17.683 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 14:17:52.520467355 +0000 2024-09-13T14:23:17.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:23:17.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:23:17.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: Reconfiguring daemon mon.b on smithi172 2024-09-13T14:23:17.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi154=y;smithi172=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: Saving service mgr spec with placement smithi154=y;smithi172=x;count:2 2024-09-13T14:23:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:17.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:17.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:23:17.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:17.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: Reconfiguring daemon mon.b on smithi172 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi154=y;smithi172=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: Saving service mgr spec with placement smithi154=y;smithi172=x;count:2 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:17.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T14:23:17.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:17.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:17.749 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T14:23:17.749 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T14:23:17.749 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.00020442 s, 2.5 MB/s 2024-09-13T14:23:17.751 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:23:17.809 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 956 Links: 1 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 14:23:04.254230763 +0000 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 14:17:52.519467381 +0000 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 14:17:52.519467381 +0000 2024-09-13T14:23:17.867 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 14:17:52.519467381 +0000 2024-09-13T14:23:17.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:23:17.932 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T14:23:17.933 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T14:23:17.933 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000231614 s, 2.2 MB/s 2024-09-13T14:23:17.934 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:23:17.991 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-09-13T14:23:18.047 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:23:18.047 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:18.047 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 957 Links: 1 2024-09-13T14:23:18.047 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:18.048 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:18.048 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 14:23:04.254230763 +0000 2024-09-13T14:23:18.048 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 14:17:52.519467381 +0000 2024-09-13T14:23:18.048 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 14:17:52.519467381 +0000 2024-09-13T14:23:18.048 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 14:17:52.519467381 +0000 2024-09-13T14:23:18.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:23:18.111 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T14:23:18.112 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T14:23:18.112 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000193149 s, 2.7 MB/s 2024-09-13T14:23:18.113 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:23:18.171 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:23:18.171 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T14:23:18.188 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T14:23:18.188 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-09-13T14:23:18.244 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T14:23:18.244 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:18.245 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-13T14:23:18.245 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:18.245 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:18.245 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-13 14:22:55.800874580 +0000 2024-09-13T14:23:18.245 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-13 14:17:50.904055958 +0000 2024-09-13T14:23:18.245 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-13 14:17:50.904055958 +0000 2024-09-13T14:23:18.245 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-09-13 14:17:50.904055958 +0000 2024-09-13T14:23:18.246 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T14:23:18.309 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-13T14:23:18.309 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-13T14:23:18.309 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000273649 s, 1.9 MB/s 2024-09-13T14:23:18.310 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T14:23:18.367 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-09-13T14:23:18.423 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T14:23:18.423 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:18.423 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-13T14:23:18.423 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:18.423 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:18.423 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-13 14:22:55.800874580 +0000 2024-09-13T14:23:18.424 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.424 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.424 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T14:23:18.613 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:18 smithi172 ceph-mon[32794]: Deploying daemon mgr.x on smithi172 2024-09-13T14:23:18.614 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:18 smithi172 ceph-mon[32794]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:18.631 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-13T14:23:18.631 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-13T14:23:18.631 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000188535 s, 2.7 MB/s 2024-09-13T14:23:18.633 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T14:23:18.650 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-09-13T14:23:18.710 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T14:23:18.710 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:18.710 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-13T14:23:18.710 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:18.711 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:18.711 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-13 14:23:18.585349450 +0000 2024-09-13T14:23:18.711 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.711 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.711 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T14:23:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:18 smithi154 ceph-mon[29001]: Deploying daemon mgr.x on smithi172 2024-09-13T14:23:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:18 smithi154 ceph-mon[29001]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:18 smithi154 ceph-mon[39074]: Deploying daemon mgr.x on smithi172 2024-09-13T14:23:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:18 smithi154 ceph-mon[39074]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:18.811 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-13T14:23:18.812 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-13T14:23:18.812 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000191549 s, 2.7 MB/s 2024-09-13T14:23:18.813 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T14:23:18.829 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-09-13T14:23:18.886 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T14:23:18.886 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T14:23:18.886 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-13T14:23:18.887 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T14:23:18.887 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T14:23:18.887 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-13 14:23:18.768345225 +0000 2024-09-13T14:23:18.887 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.887 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.887 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-09-13 14:17:50.903055980 +0000 2024-09-13T14:23:18.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T14:23:18.950 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:18 smithi172 systemd[1]: Starting Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:23:18.954 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-13T14:23:18.954 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-13T14:23:18.954 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000193359 s, 2.6 MB/s 2024-09-13T14:23:18.956 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T14:23:19.012 INFO:tasks.cephadm:Deploying osd.0 on smithi154 with /dev/vg_nvme/lv_4... 2024-09-13T14:23:19.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:23:19.208 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:19.215 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 podman[34580]: 2024-09-13 14:23:18.943418125 +0000 UTC m=+0.015769301 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T14:23:19.216 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 podman[34580]: 2024-09-13 14:23:19.092535726 +0000 UTC m=+0.164886894 container create 8b093da69eb833ffe48af01ab9c64579bfab8d6e2d95c482a1fb0dd56a295ef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:23:19.658 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 podman[34580]: 2024-09-13 14:23:19.303055994 +0000 UTC m=+0.375407163 container init 8b093da69eb833ffe48af01ab9c64579bfab8d6e2d95c482a1fb0dd56a295ef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-13T14:23:19.659 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 podman[34580]: 2024-09-13 14:23:19.306338474 +0000 UTC m=+0.378689640 container start 8b093da69eb833ffe48af01ab9c64579bfab8d6e2d95c482a1fb0dd56a295ef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458) 2024-09-13T14:23:19.659 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 ceph-mgr[34635]: -- 172.21.15.172:0/2960419954 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b7d3a64d00 con 0x55b7d3a39000 2024-09-13T14:23:19.659 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 bash[34580]: 8b093da69eb833ffe48af01ab9c64579bfab8d6e2d95c482a1fb0dd56a295ef8 2024-09-13T14:23:19.659 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 systemd[1]: Started Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:23:19.659 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:19.440+0000 7ffac1b8b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:23:19.659 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:19.498+0000 7ffac1b8b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:23:19.659 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:19.657+0000 7ffac1b8b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:23:19.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:19.734+0000 7ffac1b8b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:23:19.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:19.904+0000 7ffac1b8b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:23:20.366 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:20.047+0000 7ffac1b8b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:23:20.366 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:20.365+0000 7ffac1b8b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:23:20.625 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:20.435+0000 7ffac1b8b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:23:20.626 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-mon[32794]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:20.626 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.626 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.626 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.626 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.626 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:20.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[29001]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:20.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[39074]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:20.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:20.990 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:20.624+0000 7ffac1b8b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:23:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[29001]: Reconfiguring daemon mgr.y on smithi154 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:23:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:21.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:21.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:21.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:21 smithi154 ceph-mon[39074]: Reconfiguring daemon mgr.y on smithi154 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:21.486+0000 7ffac1b8b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:21.551+0000 7ffac1b8b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: Reconfiguring mgr.y (unknown last config time)... 2024-09-13T14:23:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:23:21.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:23:21.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:21.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:21 smithi172 ceph-mon[32794]: Reconfiguring daemon mgr.y on smithi154 2024-09-13T14:23:21.767 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:23:21.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2024-09-13T14:23:22.514 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-mon[32794]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:22.515 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:22.261+0000 7ffac1b8b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:23:22.515 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:22.374+0000 7ffac1b8b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:23:22.515 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:22.513+0000 7ffac1b8b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:23:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:22 smithi154 ceph-mon[29001]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:22.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:22 smithi154 ceph-mon[39074]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:22.785 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:22.574+0000 7ffac1b8b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:23:22.785 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:22.718+0000 7ffac1b8b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:23:22.785 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:22.784+0000 7ffac1b8b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:23:22.830 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:23.067 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:22.844+0000 7ffac1b8b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:23:23.067 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:23.002+0000 7ffac1b8b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:23:23.067 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:23.066+0000 7ffac1b8b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:23:23.330 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:23.131+0000 7ffac1b8b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:23:23.330 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:23.212+0000 7ffac1b8b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:23:23.330 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:23.270+0000 7ffac1b8b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:23:23.330 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:23.329+0000 7ffac1b8b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:23.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:23.928 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:23 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:24.097 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:23 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:24.490 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:23:24.096+0000 7ffac1b8b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:23:24.999 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: Standby manager daemon x started 2024-09-13T14:23:24.999 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:24.999 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:24.999 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:24.999 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:24.999 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:24.999 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: Standby manager daemon x started 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:23:25.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:23:25.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:25.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: Standby manager daemon x started 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4001013093' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:23:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:26.115 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:25 smithi154 ceph-mon[29001]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-13T14:23:26.115 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:25 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:23:26.115 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:25 smithi154 ceph-mon[29001]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:26.115 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:25 smithi154 ceph-mon[39074]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-13T14:23:26.115 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:25 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:23:26.115 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:25 smithi154 ceph-mon[39074]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:26.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:25 smithi172 ceph-mon[32794]: mgrmap e14: y(active, since 61s), standbys: x 2024-09-13T14:23:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:25 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:23:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:25 smithi172 ceph-mon[32794]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3500673572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]: dispatch 2024-09-13T14:23:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]: dispatch 2024-09-13T14:23:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]': finished 2024-09-13T14:23:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[29001]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:23:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3500673572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]: dispatch 2024-09-13T14:23:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]: dispatch 2024-09-13T14:23:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]': finished 2024-09-13T14:23:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[39074]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:23:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:26 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:27.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:26 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3500673572' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]: dispatch 2024-09-13T14:23:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:26 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]: dispatch 2024-09-13T14:23:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:26 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa9860db-d569-4402-8a9e-c41b083f2042"}]': finished 2024-09-13T14:23:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:26 smithi172 ceph-mon[32794]: osdmap e5: 1 total, 0 up, 1 in 2024-09-13T14:23:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:26 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:27 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/458047879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:23:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:27 smithi154 ceph-mon[39074]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:27 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/458047879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:23:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:27 smithi154 ceph-mon[29001]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:28.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:27 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/458047879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:23:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:27 smithi172 ceph-mon[32794]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:30 smithi154 ceph-mon[29001]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:30 smithi154 ceph-mon[39074]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:30.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:30 smithi172 ceph-mon[32794]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:32 smithi154 ceph-mon[29001]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:32.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:32 smithi154 ceph-mon[39074]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:32.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:32 smithi172 ceph-mon[32794]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:33.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:23:33.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:33.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:23:33.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:33.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T14:23:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:34.441 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:34 smithi154 ceph-mon[29001]: Deploying daemon osd.0 on smithi154 2024-09-13T14:23:34.441 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:34 smithi154 ceph-mon[29001]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:34.441 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:34 smithi154 ceph-mon[39074]: Deploying daemon osd.0 on smithi154 2024-09-13T14:23:34.441 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:34 smithi154 ceph-mon[39074]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:34 smithi172 ceph-mon[32794]: Deploying daemon osd.0 on smithi154 2024-09-13T14:23:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:34 smithi172 ceph-mon[32794]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:36 smithi154 ceph-mon[29001]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:36 smithi154 ceph-mon[39074]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:36 smithi172 ceph-mon[32794]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[39074]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[29001]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:38 smithi172 ceph-mon[32794]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[29001]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[39074]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:40 smithi172 ceph-mon[32794]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:40.983 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 0 on host 'smithi154' 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[39074]: from='osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[39074]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[29001]: from='osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[29001]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:23:41.652 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:41.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:41.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:41 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:41.671 DEBUG:teuthology.orchestra.run.smithi154:osd.0> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T14:23:41.674 INFO:tasks.cephadm:Deploying osd.1 on smithi154 with /dev/vg_nvme/lv_3... 2024-09-13T14:23:41.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:23:41.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:41 smithi172 ceph-mon[32794]: from='osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:23:41.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:41 smithi172 ceph-mon[32794]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T14:23:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:41 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:23:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:41 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:41 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:42.367 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:42.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[39074]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:42.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[39074]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:23:42.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[39074]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:23:42.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:42.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[39074]: from='osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:23:42.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[39074]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:23:42.715 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[29001]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:42.715 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[29001]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:23:42.715 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[29001]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:23:42.715 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:42.715 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[29001]: from='osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:23:42.715 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-mon[29001]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:23:42.715 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[42918]: 2024-09-13T14:23:42.430+0000 7fe3eb508640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T14:23:42.715 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:23:42 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[42918]: 2024-09-13T14:23:42.432+0000 7fe3e7331640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:23:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:42 smithi172 ceph-mon[32794]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:42 smithi172 ceph-mon[32794]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:23:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:42 smithi172 ceph-mon[32794]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T14:23:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:42 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:42 smithi172 ceph-mon[32794]: from='osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:23:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:42 smithi172 ceph-mon[32794]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:23:43.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[29001]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:23:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[29001]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:23:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[39074]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:23:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[39074]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:23:43.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:43.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:43 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:43.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:43 smithi172 ceph-mon[32794]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:23:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:43 smithi172 ceph-mon[32794]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T14:23:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:43 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:43 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:23:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:23:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[39074]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[39074]: osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819] boot 2024-09-13T14:23:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[39074]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:23:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:23:44.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:23:44.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[29001]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:44.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[29001]: osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819] boot 2024-09-13T14:23:44.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[29001]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:23:44.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:44 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:23:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:44 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:23:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:44 smithi172 ceph-mon[32794]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T14:23:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:44 smithi172 ceph-mon[32794]: osd.0 [v2:172.21.15.154:6802/1775775819,v1:172.21.15.154:6803/1775775819] boot 2024-09-13T14:23:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:44 smithi172 ceph-mon[32794]: osdmap e8: 1 total, 1 up, 1 in 2024-09-13T14:23:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:44 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:23:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:23:45.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2024-09-13T14:23:45.716 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:23:45.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:45 smithi154 ceph-mon[29001]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:23:45.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:45 smithi154 ceph-mon[39074]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:23:45.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:45 smithi172 ceph-mon[32794]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T14:23:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:46 smithi154 ceph-mon[29001]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:46.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:46 smithi154 ceph-mon[39074]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:46.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:46 smithi172 ceph-mon[32794]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:48.607 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi154 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi154 to 16064M 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:48.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi154 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi154 to 16064M 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:48.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.610 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:48.610 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:23:48.610 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:23:48.610 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:48 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi154 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi154 to 16064M 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:23:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:48.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:23:48.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:23:48.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:48 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:49.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:49 smithi154 ceph-mon[39074]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:49 smithi154 ceph-mon[29001]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:49 smithi172 ceph-mon[32794]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:23:50.703 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[29001]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:50.703 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/932601449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]: dispatch 2024-09-13T14:23:50.703 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]: dispatch 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]': finished 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[29001]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[39074]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/932601449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]: dispatch 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]: dispatch 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]': finished 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[39074]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:23:50.704 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:50 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:23:50.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:50 smithi172 ceph-mon[32794]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:50 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/932601449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]: dispatch 2024-09-13T14:23:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:50 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]: dispatch 2024-09-13T14:23:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:50 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "539dd807-6e38-4237-8d23-28b57f0f1b4f"}]': finished 2024-09-13T14:23:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:50 smithi172 ceph-mon[32794]: osdmap e10: 2 total, 1 up, 2 in 2024-09-13T14:23:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:50 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:23:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:51 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2682845318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:23:51.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:51 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2682845318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:23:51.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:51 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2682845318' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:23:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:52 smithi154 ceph-mon[29001]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:52.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:52 smithi154 ceph-mon[39074]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:52 smithi172 ceph-mon[32794]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:54.468 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:54 smithi154 ceph-mon[29001]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:54.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:54 smithi154 ceph-mon[39074]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:54.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:54 smithi172 ceph-mon[32794]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:56.628 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:56 smithi154 ceph-mon[29001]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:56.628 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:56 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:23:56.628 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:56 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:56.628 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:56 smithi154 ceph-mon[39074]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:56.628 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:56 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:23:56.629 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:56 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:56 smithi172 ceph-mon[32794]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:56 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T14:23:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:56 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:23:57.600 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:57 smithi154 ceph-mon[29001]: Deploying daemon osd.1 on smithi154 2024-09-13T14:23:57.600 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:57 smithi154 ceph-mon[39074]: Deploying daemon osd.1 on smithi154 2024-09-13T14:23:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:57 smithi172 ceph-mon[32794]: Deploying daemon osd.1 on smithi154 2024-09-13T14:23:58.646 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:23:58 smithi154 ceph-mon[29001]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:58.647 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:23:58 smithi154 ceph-mon[39074]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:23:58.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:23:58 smithi172 ceph-mon[32794]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:00 smithi154 ceph-mon[29001]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:00 smithi154 ceph-mon[39074]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:00.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:00 smithi172 ceph-mon[32794]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:01.487 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:01.487 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:01.487 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:01.488 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:01 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:01.488 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:01 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:01.488 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:01 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:01.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:02.568 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:02 smithi154 ceph-mon[29001]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:02.568 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:02 smithi154 ceph-mon[39074]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:02.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:02 smithi172 ceph-mon[32794]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:04.135 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 1 on host 'smithi154' 2024-09-13T14:24:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:24:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:24:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:24:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T14:24:04.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:04.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:04.773 DEBUG:teuthology.orchestra.run.smithi154:osd.1> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.1.service 2024-09-13T14:24:04.776 INFO:tasks.cephadm:Deploying osd.2 on smithi154 with /dev/vg_nvme/lv_2... 2024-09-13T14:24:04.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:24:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:05 smithi172 ceph-mon[32794]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:24:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:05 smithi172 ceph-mon[32794]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:24:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:05 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:05 smithi172 ceph-mon[32794]: from='osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:05 smithi172 ceph-mon[32794]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:05.881 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[29001]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:24:05.881 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[29001]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:24:05.881 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:05.881 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[29001]: from='osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:05.881 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[29001]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:05.882 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[39074]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:24:05.882 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[39074]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T14:24:05.882 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:05.882 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[39074]: from='osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:05.882 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-mon[39074]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:06.153 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[48623]: 2024-09-13T14:24:05.898+0000 7f8d658dc640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T14:24:06.153 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[48623]: 2024-09-13T14:24:05.904+0000 7f8d5cee6640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:24:06.358 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:06.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:06 smithi172 ceph-mon[32794]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:06 smithi172 ceph-mon[32794]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:06 smithi172 ceph-mon[32794]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:24:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:06 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:06 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:06 smithi172 ceph-mon[32794]: from='osd.1 ' entity='osd.1' 2024-09-13T14:24:06.788 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[29001]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:06.788 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[29001]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[29001]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[29001]: from='osd.1 ' entity='osd.1' 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[39074]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[39074]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[39074]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:06.789 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:06 smithi154 ceph-mon[39074]: from='osd.1 ' entity='osd.1' 2024-09-13T14:24:07.685 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:24:07.685 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:24:07.685 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:07.685 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[29001]: osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336] boot 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[29001]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[39074]: osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336] boot 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[39074]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:24:07.686 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:07 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:07 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:24:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:07 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:24:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:07 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:07 smithi172 ceph-mon[32794]: osd.1 [v2:172.21.15.154:6810/3038007336,v1:172.21.15.154:6811/3038007336] boot 2024-09-13T14:24:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:07 smithi172 ceph-mon[32794]: osdmap e13: 2 total, 2 up, 2 in 2024-09-13T14:24:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:07 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:24:08.600 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:08.600 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.600 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[29001]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.601 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:08.602 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:08.602 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.602 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:08.602 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:08.602 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.602 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:08 smithi154 ceph-mon[39074]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:24:08.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:08 smithi172 ceph-mon[32794]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T14:24:09.491 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:24:09.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2024-09-13T14:24:09.708 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:09.828 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:09 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi154 2024-09-13T14:24:09.828 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:09 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi154 to 8032M 2024-09-13T14:24:09.828 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:09 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi154 2024-09-13T14:24:09.828 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:09 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi154 to 8032M 2024-09-13T14:24:09.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:09 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi154 2024-09-13T14:24:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:09 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi154 to 8032M 2024-09-13T14:24:10.777 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:10 smithi154 ceph-mon[29001]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:10.778 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:10 smithi154 ceph-mon[39074]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:10 smithi172 ceph-mon[32794]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:11.798 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:11.799 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:11.799 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:11 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:11.799 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:11.799 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:11.799 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:11 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:11.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:11 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:12.779 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:12 smithi154 ceph-mon[39074]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:12.779 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:12 smithi154 ceph-mon[39074]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:12.779 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:12 smithi154 ceph-mon[29001]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:12.779 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:12 smithi154 ceph-mon[29001]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:12 smithi172 ceph-mon[32794]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:12 smithi172 ceph-mon[32794]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[29001]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1649784329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba93fa41-fa08-4d94-b499-149ded9151f0"}]: dispatch 2024-09-13T14:24:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1649784329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba93fa41-fa08-4d94-b499-149ded9151f0"}]': finished 2024-09-13T14:24:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[29001]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:24:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/872563995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:24:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[39074]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1649784329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba93fa41-fa08-4d94-b499-149ded9151f0"}]: dispatch 2024-09-13T14:24:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1649784329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba93fa41-fa08-4d94-b499-149ded9151f0"}]': finished 2024-09-13T14:24:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[39074]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:24:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:14 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/872563995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:24:14.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:14 smithi172 ceph-mon[32794]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:14 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1649784329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba93fa41-fa08-4d94-b499-149ded9151f0"}]: dispatch 2024-09-13T14:24:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:14 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1649784329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba93fa41-fa08-4d94-b499-149ded9151f0"}]': finished 2024-09-13T14:24:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:14 smithi172 ceph-mon[32794]: osdmap e15: 3 total, 2 up, 3 in 2024-09-13T14:24:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:14 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/872563995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:24:16.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:16 smithi154 ceph-mon[39074]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:16 smithi154 ceph-mon[29001]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:16.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:16 smithi172 ceph-mon[32794]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:18 smithi154 ceph-mon[29001]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:18 smithi154 ceph-mon[39074]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:18.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:18 smithi172 ceph-mon[32794]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:20 smithi154 ceph-mon[29001]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:24:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:20 smithi154 ceph-mon[39074]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:24:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:20.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:20 smithi172 ceph-mon[32794]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T14:24:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:21.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:21 smithi154 ceph-mon[29001]: Deploying daemon osd.2 on smithi154 2024-09-13T14:24:21.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:21 smithi154 ceph-mon[39074]: Deploying daemon osd.2 on smithi154 2024-09-13T14:24:21.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:21 smithi172 ceph-mon[32794]: Deploying daemon osd.2 on smithi154 2024-09-13T14:24:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:22 smithi154 ceph-mon[29001]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:22.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:22 smithi154 ceph-mon[39074]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:22 smithi172 ceph-mon[32794]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:24.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:24 smithi154 ceph-mon[29001]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:24.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:24 smithi154 ceph-mon[39074]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:24.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:24 smithi172 ceph-mon[32794]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:25.794 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:25 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:25.794 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:25 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:25.794 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:25 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:25.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:25 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:25.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:25 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:25.795 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:25 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:25.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:25 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:25 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:25 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:26.613 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:26 smithi154 ceph-mon[29001]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:26.613 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:26 smithi154 ceph-mon[39074]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:26.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:26 smithi172 ceph-mon[32794]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:27.985 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 2 on host 'smithi154' 2024-09-13T14:24:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:28.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:24:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:24:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:28 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:28 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:28.575 DEBUG:teuthology.orchestra.run.smithi154:osd.2> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T14:24:28.580 INFO:tasks.cephadm:Deploying osd.3 on smithi154 with /dev/vg_nvme/lv_1... 2024-09-13T14:24:28.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:24:29.259 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[29001]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:24:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[29001]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:24:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[29001]: from='osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[29001]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:29.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[39074]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:24:29.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[39074]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:24:29.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:29.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[39074]: from='osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:29.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-mon[39074]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:29.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:29 smithi172 ceph-mon[32794]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:24:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:29 smithi172 ceph-mon[32794]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T14:24:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:29 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:29 smithi172 ceph-mon[32794]: from='osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:29 smithi172 ceph-mon[32794]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:29.970 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[54339]: 2024-09-13T14:24:29.648+0000 7f9e06d5f640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T14:24:29.970 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:29 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[54339]: 2024-09-13T14:24:29.651+0000 7f9dfeb80640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:24:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[29001]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[29001]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:24:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[29001]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[29001]: from='osd.2 ' entity='osd.2' 2024-09-13T14:24:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[39074]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[39074]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:24:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[39074]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:30 smithi154 ceph-mon[39074]: from='osd.2 ' entity='osd.2' 2024-09-13T14:24:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:30 smithi172 ceph-mon[32794]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:30 smithi172 ceph-mon[32794]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T14:24:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:30 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:30 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:30 smithi172 ceph-mon[32794]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T14:24:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:30 smithi172 ceph-mon[32794]: from='osd.2 ' entity='osd.2' 2024-09-13T14:24:31.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[29001]: osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304] boot 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[29001]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:31.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[39074]: osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304] boot 2024-09-13T14:24:31.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[39074]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:24:31.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:31 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:31.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:31 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:24:31.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:31 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:24:31.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:31 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:31.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:31 smithi172 ceph-mon[32794]: osd.2 [v2:172.21.15.154:6818/1952703304,v1:172.21.15.154:6819/1952703304] boot 2024-09-13T14:24:31.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:31 smithi172 ceph-mon[32794]: osdmap e18: 3 total, 3 up, 3 in 2024-09-13T14:24:31.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:31 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:24:32.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[29001]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:32.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:24:32.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T14:24:32.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[29001]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:24:32.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:24:32.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[39074]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:32.306 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:24:32.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T14:24:32.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[39074]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:24:32.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:32 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:24:32.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:32 smithi172 ceph-mon[32794]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:32 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:24:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:32 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T14:24:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:32 smithi172 ceph-mon[32794]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T14:24:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:32 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T14:24:33.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58234]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:24:33.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58234]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:33.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58234]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:33.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58234]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:33.644 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi154 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi154 to 5354M 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:24:33.646 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[39074]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:24:33.646 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58240]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:24:33.646 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:33.646 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58240]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:33.646 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58240]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:33.646 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi154 2024-09-13T14:24:33.646 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.646 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi154 to 5354M 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:24:33.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:33 smithi154 ceph-mon[29001]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:24:33.663 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:24:33.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi154:vg_nvme/lv_1 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi154 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi154 to 5354M 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:33.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:24:33.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:33 smithi172 ceph-mon[32794]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T14:24:33.881 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:24:33.914 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58243]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:24:33.914 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58243]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:33.914 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58243]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:33.914 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58243]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:34.164 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58316]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:24:34.164 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58316]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:34.164 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58316]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:34.164 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:24:33 smithi154 sudo[58316]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:34.429 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58325]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:24:34.429 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58325]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:34.429 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58325]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:34.429 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58325]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[29001]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[29001]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[29001]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[39074]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[39074]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:24:34.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:34 smithi154 ceph-mon[39074]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:24:34.679 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58345]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:24:34.680 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58341]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:24:34.680 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58341]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:34.680 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58341]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:34.680 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58341]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:34.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:34 smithi172 ceph-mon[32794]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:34 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:34 smithi172 ceph-mon[32794]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T14:24:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:34 smithi172 ceph-mon[32794]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-13T14:24:34.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58345]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:34.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58345]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:34.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58345]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:35.198 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58351]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:24:35.198 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58351]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:35.198 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58351]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:35.198 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:34 smithi154 sudo[58351]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:35.440 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35274]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:24:35.440 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35274]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:35.441 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35274]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:35.441 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35274]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:35.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35277]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:24:35.744 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35277]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:35.744 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35277]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:35.744 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 sudo[35277]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:35.744 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:35.744 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:35.744 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:35.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:35.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:35.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:35.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:35.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:35 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:35.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58467]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T14:24:35.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:35.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58467]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:35.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58467]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:36.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58524]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T14:24:36.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58524]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T14:24:36.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58524]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T14:24:36.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:35 smithi154 sudo[58524]: pam_unix(sudo:session): session closed for user root 2024-09-13T14:24:36.666 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:36.666 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:36.666 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:36.666 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:36.666 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[29001]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:36.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:36.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:36 smithi154 ceph-mon[39074]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:36.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:36.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:36 smithi172 ceph-mon[32794]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T14:24:37.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:37 smithi154 ceph-mon[39074]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:37 smithi154 ceph-mon[29001]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:37 smithi172 ceph-mon[32794]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:38.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[29001]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2781900875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]: dispatch 2024-09-13T14:24:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]: dispatch 2024-09-13T14:24:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]': finished 2024-09-13T14:24:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[29001]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:24:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1805368626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:24:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[39074]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2781900875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]: dispatch 2024-09-13T14:24:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]: dispatch 2024-09-13T14:24:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]': finished 2024-09-13T14:24:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[39074]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:24:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:38 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1805368626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:24:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:38 smithi172 ceph-mon[32794]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:38 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2781900875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]: dispatch 2024-09-13T14:24:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:38 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]: dispatch 2024-09-13T14:24:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:38 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4422727c-268e-4356-83fe-118d25e6cb4d"}]': finished 2024-09-13T14:24:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:38 smithi172 ceph-mon[32794]: osdmap e22: 4 total, 3 up, 4 in 2024-09-13T14:24:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:38 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1805368626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:24:40.714 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:40 smithi154 ceph-mon[29001]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:40.715 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:40 smithi154 ceph-mon[39074]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:40.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:40 smithi172 ceph-mon[32794]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:42.608 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:42 smithi154 ceph-mon[29001]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:42.609 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:42 smithi154 ceph-mon[39074]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:42.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:42 smithi172 ceph-mon[32794]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:44 smithi154 ceph-mon[29001]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:24:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:44 smithi154 ceph-mon[39074]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:44.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:44 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:24:44.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:44 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:44.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:44 smithi172 ceph-mon[32794]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:44 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T14:24:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:44 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:45.558 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:45 smithi154 ceph-mon[29001]: Deploying daemon osd.3 on smithi154 2024-09-13T14:24:45.558 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:45 smithi154 ceph-mon[39074]: Deploying daemon osd.3 on smithi154 2024-09-13T14:24:45.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:45 smithi172 ceph-mon[32794]: Deploying daemon osd.3 on smithi154 2024-09-13T14:24:46.589 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:46 smithi154 ceph-mon[29001]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:46.589 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:46 smithi154 ceph-mon[39074]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:46.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:46 smithi172 ceph-mon[32794]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:48.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:48 smithi154 ceph-mon[29001]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:48.586 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:48 smithi154 ceph-mon[39074]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:48 smithi172 ceph-mon[32794]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:49.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:49 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:49.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:49 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:49.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:49 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:49.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:49 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:49.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:49 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:49.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:49 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:49.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:49 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:49 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:49 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:50 smithi172 ceph-mon[32794]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:50 smithi154 ceph-mon[29001]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:50.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:50 smithi154 ceph-mon[39074]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:52.191 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 3 on host 'smithi154' 2024-09-13T14:24:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:52.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:24:52.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:52.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:52.826 DEBUG:teuthology.orchestra.run.smithi154:osd.3> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.3.service 2024-09-13T14:24:52.830 INFO:tasks.cephadm:Deploying osd.4 on smithi172 with /dev/vg_nvme/lv_4... 2024-09-13T14:24:52.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-13T14:24:53.034 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:24:54.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:53 smithi172 ceph-mon[32794]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:24:54.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:53 smithi172 ceph-mon[32794]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:24:54.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:53 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:54.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:53 smithi172 ceph-mon[32794]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:54.171 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:53 smithi172 ceph-mon[32794]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:54.174 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[39074]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[39074]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[39074]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[39074]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[29001]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[29001]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[29001]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-13T14:24:54.175 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:53 smithi154 ceph-mon[29001]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:54.458 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[60271]: 2024-09-13T14:24:54.350+0000 7f7eef29f640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T14:24:54.458 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[60271]: 2024-09-13T14:24:54.354+0000 7f7ee70c0640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:24:55.169 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:54 smithi172 ceph-mon[32794]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:55.169 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:54 smithi172 ceph-mon[32794]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:24:55.169 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:54 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:55.169 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:54 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:55.169 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:54 smithi172 ceph-mon[32794]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' 2024-09-13T14:24:55.206 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:24:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[29001]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[29001]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:24:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[29001]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' 2024-09-13T14:24:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[39074]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-13T14:24:55.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[39074]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T14:24:55.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:55.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:55.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:54 smithi154 ceph-mon[39074]: from='osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703]' entity='osd.3' 2024-09-13T14:24:55.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_4 2024-09-13T14:24:55.431 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:24:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:24:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:24:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703] boot 2024-09-13T14:24:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:56.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703] boot 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:56.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:56.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:55 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: osd.3 [v2:172.21.15.154:6826/1896570703,v1:172.21.15.154:6827/1896570703] boot 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: osdmap e25: 4 total, 4 up, 4 in 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T14:24:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:56.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:24:56.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:55 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:24:57.678 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:57 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi154 2024-09-13T14:24:57.678 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:57 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi154 to 4016M 2024-09-13T14:24:57.678 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:57 smithi172 ceph-mon[32794]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:24:57.678 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:57 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:57.678 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:57 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:57.679 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:57 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi154 2024-09-13T14:24:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi154 to 4016M 2024-09-13T14:24:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[29001]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:24:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:57.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi154 2024-09-13T14:24:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi154 to 4016M 2024-09-13T14:24:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[39074]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T14:24:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:24:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:24:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:57 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:24:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:58 smithi172 ceph-mon[32794]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:24:58 smithi172 ceph-mon[32794]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:24:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:58 smithi154 ceph-mon[29001]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:24:58 smithi154 ceph-mon[29001]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:24:58.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:58 smithi154 ceph-mon[39074]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:24:58.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:24:58 smithi154 ceph-mon[39074]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[29001]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/59700950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]: dispatch 2024-09-13T14:25:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]: dispatch 2024-09-13T14:25:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]': finished 2024-09-13T14:25:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[29001]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:25:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/363798305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[39074]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/59700950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]: dispatch 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]: dispatch 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]': finished 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[39074]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:00 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/363798305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:00 smithi172 ceph-mon[32794]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:00 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/59700950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]: dispatch 2024-09-13T14:25:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:00 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]: dispatch 2024-09-13T14:25:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:00 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d3372f5-f7f2-4798-a221-5adc2acfc3c6"}]': finished 2024-09-13T14:25:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:00 smithi172 ceph-mon[32794]: osdmap e27: 5 total, 4 up, 5 in 2024-09-13T14:25:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:00 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/363798305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:02 smithi172 ceph-mon[32794]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:02 smithi154 ceph-mon[29001]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:02.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:02 smithi154 ceph-mon[39074]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:04.395 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:04 smithi172 ceph-mon[32794]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:04 smithi154 ceph-mon[29001]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:04 smithi154 ceph-mon[39074]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:05.633 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:05 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:25:05.633 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:05 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:05.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:05 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:25:05.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:05 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:05.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:05 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T14:25:05.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:05 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:06.635 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:06 smithi172 ceph-mon[32794]: Deploying daemon osd.4 on smithi172 2024-09-13T14:25:06.635 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:06 smithi172 ceph-mon[32794]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:06.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:06 smithi154 ceph-mon[29001]: Deploying daemon osd.4 on smithi172 2024-09-13T14:25:06.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:06 smithi154 ceph-mon[29001]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:06.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:06 smithi154 ceph-mon[39074]: Deploying daemon osd.4 on smithi172 2024-09-13T14:25:06.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:06 smithi154 ceph-mon[39074]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:08.633 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:08 smithi172 ceph-mon[32794]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:08 smithi154 ceph-mon[29001]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:08.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:08 smithi154 ceph-mon[39074]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:10.638 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:10 smithi172 ceph-mon[32794]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:10.638 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:10 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:10.638 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:10 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:10.638 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:10 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[29001]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:10.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[39074]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:10 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:12.560 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:12 smithi172 ceph-mon[32794]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:12.560 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:12 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:12.561 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:12 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:12 smithi154 ceph-mon[29001]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:12 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:12 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:12.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:12 smithi154 ceph-mon[39074]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:12 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:12 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:13.430 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:13 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:13.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:13 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:13.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:13 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:13.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:13 smithi172 ceph-mon[32794]: from='osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:25:13.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:13 smithi172 ceph-mon[32794]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:25:13.465 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 4 on host 'smithi172' 2024-09-13T14:25:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[29001]: from='osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:25:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[29001]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:25:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[39074]: from='osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:25:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:13 smithi154 ceph-mon[39074]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T14:25:14.032 DEBUG:teuthology.orchestra.run.smithi172:osd.4> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T14:25:14.034 INFO:tasks.cephadm:Deploying osd.5 on smithi172 with /dev/vg_nvme/lv_3... 2024-09-13T14:25:14.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-13T14:25:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: from='osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[29001]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:25:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:25:14.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:14.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: from='osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:14.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:14 smithi154 ceph-mon[39074]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: from='osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:14 smithi172 ceph-mon[32794]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:14.783 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:25:15.149 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T14:25:15.035+0000 7f441abf1640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T14:25:15.149 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T14:25:15.038+0000 7f4412a12640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:25:15.711 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-mon[32794]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:25:15.711 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-mon[32794]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:25:15.712 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:15.712 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:15.712 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-mon[32794]: from='osd.4 ' entity='osd.4' 2024-09-13T14:25:15.712 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:15 smithi172 ceph-mon[32794]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[29001]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:25:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[29001]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:25:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[29001]: from='osd.4 ' entity='osd.4' 2024-09-13T14:25:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[29001]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[39074]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:25:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[39074]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T14:25:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[39074]: from='osd.4 ' entity='osd.4' 2024-09-13T14:25:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:15 smithi154 ceph-mon[39074]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T14:25:16.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:16 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:25:16.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:16 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:25:16.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:16 smithi172 ceph-mon[32794]: osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689] boot 2024-09-13T14:25:16.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:16 smithi172 ceph-mon[32794]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:25:16.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:16 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:25:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:25:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[29001]: osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689] boot 2024-09-13T14:25:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[29001]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:25:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:25:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:25:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[39074]: osd.4 [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689] boot 2024-09-13T14:25:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[39074]: osdmap e30: 5 total, 5 up, 5 in 2024-09-13T14:25:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:16 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:25:17.835 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:25:17.835 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi172 2024-09-13T14:25:17.835 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.835 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.835 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:17.835 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi172 to 17088M 2024-09-13T14:25:17.836 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.836 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:17.836 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:17.836 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.836 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:17 smithi172 ceph-mon[32794]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:25:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi172 2024-09-13T14:25:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi172 to 17088M 2024-09-13T14:25:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[29001]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi172 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi172 to 17088M 2024-09-13T14:25:17.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:17.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:17.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:17.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:17 smithi154 ceph-mon[39074]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:18.538 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:25:18.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_3 2024-09-13T14:25:18.751 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:25:18.872 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:18 smithi172 ceph-mon[32794]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:25:18.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:18 smithi154 ceph-mon[29001]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:25:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:18 smithi154 ceph-mon[39074]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T14:25:19.898 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:19 smithi172 ceph-mon[32794]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:25:19.898 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:19 smithi172 ceph-mon[32794]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:19.899 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:19 smithi172 ceph-mon[32794]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T14:25:19.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:19 smithi154 ceph-mon[29001]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:25:19.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:19 smithi154 ceph-mon[29001]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:19.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:19 smithi154 ceph-mon[29001]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T14:25:19.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:19 smithi154 ceph-mon[39074]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T14:25:19.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:19 smithi154 ceph-mon[39074]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:19.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:19 smithi154 ceph-mon[39074]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T14:25:20.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:20 smithi172 ceph-mon[32794]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:20.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:25:20.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:25:20.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[39074]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[29001]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:25:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:21.844 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:21 smithi172 ceph-mon[32794]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:21.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:21 smithi154 ceph-mon[29001]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:21.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:21 smithi154 ceph-mon[39074]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/1745647325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]: dispatch 2024-09-13T14:25:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]: dispatch 2024-09-13T14:25:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]': finished 2024-09-13T14:25:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[29001]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:25:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:22.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/1745647325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]: dispatch 2024-09-13T14:25:22.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]: dispatch 2024-09-13T14:25:22.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]': finished 2024-09-13T14:25:22.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[39074]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:25:22.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:22 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:22 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/1745647325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]: dispatch 2024-09-13T14:25:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:22 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]: dispatch 2024-09-13T14:25:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:22 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6458ef05-8c2f-4ef0-a0a1-080908418a07"}]': finished 2024-09-13T14:25:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:22 smithi172 ceph-mon[32794]: osdmap e35: 6 total, 5 up, 6 in 2024-09-13T14:25:22.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:22 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:23.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:23 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/1174972735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:23.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:23 smithi154 ceph-mon[29001]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-09-13T14:25:23.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:23 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/1174972735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:23.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:23 smithi154 ceph-mon[39074]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-09-13T14:25:23.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:23 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/1174972735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:23.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:23 smithi172 ceph-mon[32794]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-09-13T14:25:26.587 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:26 smithi172 ceph-mon[32794]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-13T14:25:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:26 smithi154 ceph-mon[29001]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-13T14:25:26.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:26 smithi154 ceph-mon[39074]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-13T14:25:28.471 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:28 smithi172 ceph-mon[32794]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:25:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:28 smithi154 ceph-mon[29001]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:25:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:28 smithi154 ceph-mon[39074]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T14:25:29.717 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:29 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:25:29.717 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:29 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:29.717 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:29 smithi172 ceph-mon[32794]: Deploying daemon osd.5 on smithi172 2024-09-13T14:25:29.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:29 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:25:29.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:29 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:29.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:29 smithi154 ceph-mon[29001]: Deploying daemon osd.5 on smithi172 2024-09-13T14:25:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:29 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T14:25:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:29 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:29 smithi154 ceph-mon[39074]: Deploying daemon osd.5 on smithi172 2024-09-13T14:25:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:30 smithi154 ceph-mon[39074]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-13T14:25:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:30 smithi154 ceph-mon[29001]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-13T14:25:30.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:30 smithi172 ceph-mon[32794]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-13T14:25:32.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:32 smithi172 ceph-mon[32794]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:25:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:32 smithi154 ceph-mon[29001]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:25:32.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:32 smithi154 ceph-mon[39074]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:25:33.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:33.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:33.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:33 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:33.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:33.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:33.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:33 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:33.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:33.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:33.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:33 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:34.687 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:34 smithi172 ceph-mon[32794]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:34 smithi154 ceph-mon[29001]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:34.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:34 smithi154 ceph-mon[39074]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:36.655 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 5 on host 'smithi172' 2024-09-13T14:25:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[29001]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:36.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[39074]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:36 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:36 smithi172 ceph-mon[32794]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:36 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:37.315 DEBUG:teuthology.orchestra.run.smithi172:osd.5> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.5.service 2024-09-13T14:25:37.317 INFO:tasks.cephadm:Deploying osd.6 on smithi172 with /dev/vg_nvme/lv_2... 2024-09-13T14:25:37.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-13T14:25:37.636 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:37 smithi172 ceph-mon[32794]: from='osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:25:37.636 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:37 smithi172 ceph-mon[32794]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:25:37.636 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:37 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:37.636 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:37 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:37.636 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:37 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[29001]: from='osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[29001]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[39074]: from='osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[39074]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:37.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:37 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:38.701 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-mon[32794]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:38.701 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-mon[32794]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:25:38.702 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-mon[32794]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:25:38.702 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:38.702 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-mon[32794]: from='osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:38.702 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-mon[32794]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[29001]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[29001]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:25:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[29001]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:25:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[29001]: from='osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[29001]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[39074]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[39074]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T14:25:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[39074]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T14:25:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[39074]: from='osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:38 smithi154 ceph-mon[39074]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:25:38.891 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:25:38.991 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T14:25:38.907+0000 7f1519c43640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T14:25:38.991 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 14:25:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T14:25:38.914+0000 7f1511a64640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: from='osd.5 ' entity='osd.5' 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433] boot 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:25:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: from='osd.5 ' entity='osd.5' 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433] boot 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:25:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:39 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: from='osd.5 ' entity='osd.5' 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: osd.5 [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433] boot 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: osdmap e38: 6 total, 6 up, 6 in 2024-09-13T14:25:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:39 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:40 smithi172 ceph-mon[32794]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:25:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:25:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:25:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[29001]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:40 smithi154 ceph-mon[39074]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T14:25:41.698 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:25:41.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_2 2024-09-13T14:25:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:41 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi172 2024-09-13T14:25:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:41 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi172 to 8544M 2024-09-13T14:25:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:41 smithi154 ceph-mon[29001]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T14:25:41.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:41 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi172 2024-09-13T14:25:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:41 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi172 to 8544M 2024-09-13T14:25:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:41 smithi154 ceph-mon[39074]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T14:25:41.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:41 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi172 2024-09-13T14:25:41.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:41 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi172 to 8544M 2024-09-13T14:25:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:41 smithi172 ceph-mon[32794]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T14:25:41.912 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:25:42.701 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:42 smithi172 ceph-mon[32794]: pgmap v129: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:42.701 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:42 smithi172 ceph-mon[32794]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T14:25:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:42 smithi154 ceph-mon[29001]: pgmap v129: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:42 smithi154 ceph-mon[29001]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T14:25:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:42 smithi154 ceph-mon[39074]: pgmap v129: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:42 smithi154 ceph-mon[39074]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T14:25:44.341 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:44 smithi172 ceph-mon[32794]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T14:25:44.341 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:44 smithi172 ceph-mon[32794]: pgmap v132: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:44.341 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:44 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:25:44.341 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:44 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:25:44.341 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:44 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[29001]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T14:25:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[29001]: pgmap v132: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:25:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:25:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:44.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[39074]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T14:25:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[39074]: pgmap v132: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:25:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:25:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:44 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:45.206 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:45 smithi172 ceph-mon[32794]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:45 smithi154 ceph-mon[29001]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:45.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:45 smithi154 ceph-mon[39074]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:25:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[29001]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-09-13T14:25:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/1251633686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]: dispatch 2024-09-13T14:25:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]: dispatch 2024-09-13T14:25:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]': finished 2024-09-13T14:25:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[29001]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:25:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:25:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[39074]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-09-13T14:25:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/1251633686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]: dispatch 2024-09-13T14:25:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]: dispatch 2024-09-13T14:25:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]': finished 2024-09-13T14:25:46.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[39074]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:25:46.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:46 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:25:46.477 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:46 smithi172 ceph-mon[32794]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-09-13T14:25:46.477 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:46 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/1251633686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]: dispatch 2024-09-13T14:25:46.477 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:46 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]: dispatch 2024-09-13T14:25:46.477 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:46 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e768676-0ffa-46e3-8a03-db3fdde2951c"}]': finished 2024-09-13T14:25:46.478 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:46 smithi172 ceph-mon[32794]: osdmap e43: 7 total, 6 up, 7 in 2024-09-13T14:25:46.478 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:46 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:25:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:47 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/2756354054' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:47.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:47 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/2756354054' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:47.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:47 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/2756354054' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:25:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:48 smithi154 ceph-mon[29001]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:25:48.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:48 smithi154 ceph-mon[39074]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:25:48.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:48 smithi172 ceph-mon[32794]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:25:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:50 smithi154 ceph-mon[29001]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-13T14:25:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:50 smithi154 ceph-mon[39074]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-13T14:25:50.739 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:50 smithi172 ceph-mon[32794]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-13T14:25:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:52 smithi154 ceph-mon[29001]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:25:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:25:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:52 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:52.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:52 smithi154 ceph-mon[39074]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:25:52.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:25:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:52 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:52 smithi172 ceph-mon[32794]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:25:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T14:25:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:52 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:25:53.627 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:53 smithi172 ceph-mon[32794]: Deploying daemon osd.6 on smithi172 2024-09-13T14:25:53.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:53 smithi154 ceph-mon[29001]: Deploying daemon osd.6 on smithi172 2024-09-13T14:25:53.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:53 smithi154 ceph-mon[39074]: Deploying daemon osd.6 on smithi172 2024-09-13T14:25:54.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:54 smithi154 ceph-mon[29001]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:25:54.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:54 smithi154 ceph-mon[39074]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:25:54.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:54 smithi172 ceph-mon[32794]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:25:56.693 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:56 smithi172 ceph-mon[32794]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:56 smithi154 ceph-mon[29001]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:56.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:56 smithi154 ceph-mon[39074]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:57 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:57 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:57 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:57 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:57 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:57 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:57.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:57 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:25:57.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:57 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:57 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:25:58.680 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:25:58 smithi172 ceph-mon[32794]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:25:58 smithi154 ceph-mon[29001]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:25:58.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:25:58 smithi154 ceph-mon[39074]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:26:00.186 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 6 on host 'smithi172' 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:00 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.809 DEBUG:teuthology.orchestra.run.smithi172:osd.6> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T14:26:00.852 INFO:tasks.cephadm:Deploying osd.7 on smithi172 with /dev/vg_nvme/lv_1... 2024-09-13T14:26:00.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-13T14:26:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:26:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:00.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:26:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T14:26:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:00 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:01.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:01 smithi172 ceph-mon[32794]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:26:01.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:01 smithi172 ceph-mon[32794]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T14:26:01.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:01 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:01.821 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:01 smithi172 ceph-mon[32794]: from='osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:01.821 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:01 smithi172 ceph-mon[32794]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:01.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[29001]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:26:01.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[29001]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T14:26:01.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:01.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[29001]: from='osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:01.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[29001]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:01.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[39074]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T14:26:01.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[39074]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T14:26:01.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:01.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[39074]: from='osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:01.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:01 smithi154 ceph-mon[39074]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:02.079 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 14:26:01 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T14:26:01.943+0000 7f7fc89cf640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T14:26:02.079 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 14:26:01 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T14:26:01.949+0000 7f7fbffd9640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:26:02.408 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:26:02.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:02 smithi172 ceph-mon[32794]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:26:02.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:02 smithi172 ceph-mon[32794]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:26:02.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:02 smithi172 ceph-mon[32794]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T14:26:02.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:02 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:02.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:02 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:02.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:02 smithi172 ceph-mon[32794]: from='osd.6 ' entity='osd.6' 2024-09-13T14:26:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[29001]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:26:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[29001]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:26:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[29001]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T14:26:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[29001]: from='osd.6 ' entity='osd.6' 2024-09-13T14:26:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[39074]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T14:26:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[39074]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:26:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[39074]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T14:26:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:02 smithi154 ceph-mon[39074]: from='osd.6 ' entity='osd.6' 2024-09-13T14:26:03.709 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:03 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:26:03.709 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:03 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:26:03.709 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:03 smithi172 ceph-mon[32794]: osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288] boot 2024-09-13T14:26:03.709 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:03 smithi172 ceph-mon[32794]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:26:03.709 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:03 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:03.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:26:03.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:26:03.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[39074]: osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288] boot 2024-09-13T14:26:03.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[39074]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:26:03.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:26:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:26:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[29001]: osd.6 [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288] boot 2024-09-13T14:26:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[29001]: osdmap e46: 7 total, 7 up, 7 in 2024-09-13T14:26:03.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:03 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:04.506 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:04.506 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T14:26:04.507 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.507 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.507 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T14:26:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:04 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:04 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:05.262 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-13T14:26:05.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi172:vg_nvme/lv_1 2024-09-13T14:26:05.476 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:26:05.597 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:05 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi172 2024-09-13T14:26:05.597 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:05 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi172 to 5696M 2024-09-13T14:26:05.597 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:05 smithi172 ceph-mon[32794]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T14:26:05.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:05 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi172 2024-09-13T14:26:05.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:05 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi172 to 5696M 2024-09-13T14:26:05.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:05 smithi154 ceph-mon[29001]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T14:26:05.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:05 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi172 2024-09-13T14:26:05.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:05 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi172 to 5696M 2024-09-13T14:26:05.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:05 smithi154 ceph-mon[39074]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T14:26:06.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:06 smithi172 ceph-mon[32794]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:06.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:06 smithi172 ceph-mon[32794]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T14:26:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:06 smithi154 ceph-mon[29001]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:06 smithi154 ceph-mon[29001]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T14:26:06.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:06 smithi154 ceph-mon[39074]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:06.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:06 smithi154 ceph-mon[39074]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T14:26:07.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:07 smithi172 ceph-mon[32794]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T14:26:07.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:07 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:26:07.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:07 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:26:07.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:07 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:07.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[39074]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T14:26:07.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:26:07.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:26:07.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:07.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[29001]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T14:26:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T14:26:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T14:26:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:07 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:08.637 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:08 smithi172 ceph-mon[32794]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:08.637 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:08 smithi172 ceph-mon[32794]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:08 smithi154 ceph-mon[29001]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:08 smithi154 ceph-mon[29001]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:08.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:08 smithi154 ceph-mon[39074]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:08.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:08 smithi154 ceph-mon[39074]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 589 MiB used, 625 GiB / 626 GiB avail 2024-09-13T14:26:09.909 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:09 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/344257765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]: dispatch 2024-09-13T14:26:09.909 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:09 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]: dispatch 2024-09-13T14:26:09.909 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:09 smithi172 ceph-mon[32794]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]': finished 2024-09-13T14:26:09.909 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:09 smithi172 ceph-mon[32794]: osdmap e51: 8 total, 7 up, 8 in 2024-09-13T14:26:09.909 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:09 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/344257765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]: dispatch 2024-09-13T14:26:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]: dispatch 2024-09-13T14:26:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[29001]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]': finished 2024-09-13T14:26:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[29001]: osdmap e51: 8 total, 7 up, 8 in 2024-09-13T14:26:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:09.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/344257765' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]: dispatch 2024-09-13T14:26:09.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]: dispatch 2024-09-13T14:26:09.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[39074]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef36cdfb-77d4-4974-aa46-d64d79a2b0e0"}]': finished 2024-09-13T14:26:09.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[39074]: osdmap e51: 8 total, 7 up, 8 in 2024-09-13T14:26:09.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:09 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:10 smithi154 ceph-mon[29001]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-13T14:26:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:10 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/922837688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:26:10.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:10 smithi154 ceph-mon[39074]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-13T14:26:10.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:10 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/922837688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:26:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:10 smithi172 ceph-mon[32794]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-13T14:26:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:10 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/922837688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T14:26:12.716 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:12 smithi172 ceph-mon[32794]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:26:12.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:12 smithi154 ceph-mon[39074]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:26:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:12 smithi154 ceph-mon[29001]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T14:26:14.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:14 smithi154 ceph-mon[39074]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:26:14.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:14 smithi154 ceph-mon[29001]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:26:14.979 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:14 smithi172 ceph-mon[32794]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-09-13T14:26:15.858 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:15 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:26:15.858 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:15 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:15 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:26:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:15 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:15.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:15 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T14:26:15.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:15 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:16.795 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:16 smithi172 ceph-mon[32794]: Deploying daemon osd.7 on smithi172 2024-09-13T14:26:16.796 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:16 smithi172 ceph-mon[32794]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-13T14:26:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:16 smithi154 ceph-mon[29001]: Deploying daemon osd.7 on smithi172 2024-09-13T14:26:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:16 smithi154 ceph-mon[29001]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-13T14:26:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:16 smithi154 ceph-mon[39074]: Deploying daemon osd.7 on smithi172 2024-09-13T14:26:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:16 smithi154 ceph-mon[39074]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-13T14:26:18.630 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:18 smithi172 ceph-mon[32794]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:26:18.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:18 smithi154 ceph-mon[29001]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:26:18.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:18 smithi154 ceph-mon[39074]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:26:20.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:20 smithi172 ceph-mon[32794]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:20.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:20.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:20.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:20 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[29001]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:20.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:20.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:20.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[39074]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:20.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:20.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:20.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:20 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:22.877 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:22 smithi172 ceph-mon[32794]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:22 smithi154 ceph-mon[29001]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:22.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:22 smithi154 ceph-mon[39074]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:23.813 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 7 on host 'smithi172' 2024-09-13T14:26:24.376 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:24 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.397 DEBUG:teuthology.orchestra.run.smithi172:osd.7> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.7.service 2024-09-13T14:26:24.400 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-13T14:26:24.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T14:26:24.428 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.428 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.428 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:24.428 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:24.428 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:24.429 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:26:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T14:26:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:24 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:24.613 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:25.807 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[29001]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:26:25.807 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[29001]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T14:26:25.807 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:25.807 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[29001]: from='osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:25.807 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[29001]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:25.808 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[39074]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:26:25.808 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[39074]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T14:26:25.808 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:25.808 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[39074]: from='osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:25.808 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:25 smithi154 ceph-mon[39074]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:25.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:25 smithi172 ceph-mon[32794]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T14:26:25.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:25 smithi172 ceph-mon[32794]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T14:26:25.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:25 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:25.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:25 smithi172 ceph-mon[32794]: from='osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:25.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:25 smithi172 ceph-mon[32794]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-09-13T14:26:25.831 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 14:26:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T14:26:25.516+0000 7fd606085640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T14:26:25.831 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 14:26:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T14:26:25.523+0000 7fd601eae640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:26:26.449 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:26:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: purged_snaps scrub starts 2024-09-13T14:26:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: purged_snaps scrub ok 2024-09-13T14:26:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:26:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3841944324' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: purged_snaps scrub starts 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: purged_snaps scrub ok 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:26 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3841944324' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:26:26.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: purged_snaps scrub starts 2024-09-13T14:26:26.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: purged_snaps scrub ok 2024-09-13T14:26:26.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:26.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-09-13T14:26:26.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T14:26:26.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:26.907 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:26.907 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:26 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3841944324' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:26:27.277 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1726237562,"num_in_osds":8,"osd_in_since":1726237569,"num_remapped_pgs":0} 2024-09-13T14:26:27.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:27.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567] boot 2024-09-13T14:26:27.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:26:27.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:27.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567] boot 2024-09-13T14:26:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:27 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:27.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: osd.7 [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567] boot 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: osdmap e54: 8 total, 8 up, 8 in 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:27 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:28.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T14:26:28.490 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:28.611 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[39074]: Detected new or changed devices on smithi172 2024-09-13T14:26:28.611 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi172 to 4272M 2024-09-13T14:26:28.611 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[39074]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:28.611 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[39074]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:26:28.611 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[29001]: Detected new or changed devices on smithi172 2024-09-13T14:26:28.612 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi172 to 4272M 2024-09-13T14:26:28.612 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[29001]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:28.612 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:28 smithi154 ceph-mon[29001]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:26:28.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:28 smithi172 ceph-mon[32794]: Detected new or changed devices on smithi172 2024-09-13T14:26:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:28 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi172 to 4272M 2024-09-13T14:26:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:28 smithi172 ceph-mon[32794]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-13T14:26:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:28 smithi172 ceph-mon[32794]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T14:26:29.848 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:29 smithi154 ceph-mon[29001]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:26:29.848 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:29 smithi154 ceph-mon[29001]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:26:29.848 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:29 smithi154 ceph-mon[39074]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:26:29.848 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:29 smithi154 ceph-mon[39074]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:26:29.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:29 smithi172 ceph-mon[32794]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T14:26:29.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:29 smithi172 ceph-mon[32794]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T14:26:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:26:30.845 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:30 smithi154 ceph-mon[29001]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:30.846 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:30 smithi154 ceph-mon[29001]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:26:30.846 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:30 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2611525640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:26:30.846 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:30 smithi154 ceph-mon[39074]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:30.846 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:30 smithi154 ceph-mon[39074]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:26:30.846 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:30 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2611525640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:26:30.866 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1726237586,"num_in_osds":8,"osd_in_since":1726237569,"num_remapped_pgs":0} 2024-09-13T14:26:30.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:26:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:30 smithi172 ceph-mon[32794]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:30 smithi172 ceph-mon[32794]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T14:26:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:30 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2611525640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T14:26:31.071 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:32.793 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:32 smithi172 ceph-mon[32794]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:32.810 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:26:32.810 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":58,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T14:26:30.120026+0000","last_up_change":"2024-09-13T14:26:26.512111+0000","last_in_change":"2024-09-13T14:26:09.317846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6803","nonce":1775775819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6805","nonce":1775775819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6809","nonce":1775775819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6807","nonce":1775775819}]},"public_addr":"172.21.15.154:6803/1775775819","cluster_addr":"172.21.15.154:6805/1775775819","heartbeat_back_addr":"172.21.15.154:6809/1775775819","heartbeat_front_addr":"172.21.15.154:6807/1775775819","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6811","nonce":3038007336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6813","nonce":3038007336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6817","nonce":3038007336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6815","nonce":3038007336}]},"public_addr":"172.21.15.154:6811/3038007336","cluster_addr":"172.21.15.154:6813/3038007336","heartbeat_back_addr":"172.21.15.154:6817/3038007336","heartbeat_front_addr":"172.21.15.154:6815/3038007336","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6819","nonce":1952703304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6821","nonce":1952703304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6825","nonce":1952703304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6823","nonce":1952703304}]},"public_addr":"172.21.15.154:6819/1952703304","cluster_addr":"172.21.15.154:6821/1952703304","heartbeat_back_addr":"172.21.15.154:6825/1952703304","heartbeat_front_addr":"172.21.15.154:6823/1952703304","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6827","nonce":1896570703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6829","nonce":1896570703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6833","nonce":1896570703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6831","nonce":1896570703}]},"public_addr":"172.21.15.154:6827/1896570703","cluster_addr":"172.21.15.154:6829/1896570703","heartbeat_back_addr":"172.21.15.154:6833/1896570703","heartbeat_front_addr":"172.21.15.154:6831/1896570703","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:26:32.811 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:32 smithi154 ceph-mon[29001]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:32.811 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:32 smithi154 ceph-mon[39074]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:33.399 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T14:24:31.633615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T14:26:33.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-13T14:26:33.598 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:33.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:33 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1012387007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:26:33.719 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:33 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1012387007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:26:33.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:33 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1012387007' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:26:34.861 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:34 smithi154 ceph-mon[29001]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-13T14:26:34.861 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:34 smithi154 ceph-mon[39074]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-13T14:26:34.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:34 smithi172 ceph-mon[32794]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-13T14:26:35.230 INFO:teuthology.orchestra.run.smithi154.stdout:pg_num: 1 2024-09-13T14:26:35.865 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:35 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2293248904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:26:35.865 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:35 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2293248904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:26:35.887 INFO:tasks.cephadm:Adding prometheus.a on smithi172 2024-09-13T14:26:35.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi172=a' 2024-09-13T14:26:35.917 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:35 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2293248904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T14:26:36.097 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:26:36.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:36 smithi154 ceph-mon[29001]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T14:26:36.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:36 smithi154 ceph-mon[39074]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T14:26:36.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:36 smithi172 ceph-mon[32794]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T14:26:37.821 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled prometheus update... 2024-09-13T14:26:38.425 DEBUG:teuthology.orchestra.run.smithi172:prometheus.a> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-13T14:26:38.468 INFO:tasks.cephadm:Adding node-exporter.a on smithi154 2024-09-13T14:26:38.468 INFO:tasks.cephadm:Adding node-exporter.b on smithi172 2024-09-13T14:26:38.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi154=a;smithi172=b' 2024-09-13T14:26:38.709 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:26:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:38 smithi172 ceph-mon[32794]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:26:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:38 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[29001]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:26:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:38.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[39074]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T14:26:38.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:38.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:38.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:26:38.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:26:38.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:38 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:39.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:39 smithi154 ceph-mon[39074]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:39.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:39 smithi154 ceph-mon[39074]: Saving service prometheus spec with placement smithi172=a;count:1 2024-09-13T14:26:39.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:39 smithi154 ceph-mon[39074]: Deploying daemon prometheus.a on smithi172 2024-09-13T14:26:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:39 smithi154 ceph-mon[29001]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:39.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:39 smithi154 ceph-mon[29001]: Saving service prometheus spec with placement smithi172=a;count:1 2024-09-13T14:26:39.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:39 smithi154 ceph-mon[29001]: Deploying daemon prometheus.a on smithi172 2024-09-13T14:26:39.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:39 smithi172 ceph-mon[32794]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:39.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:39 smithi172 ceph-mon[32794]: Saving service prometheus spec with placement smithi172=a;count:1 2024-09-13T14:26:39.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:39 smithi172 ceph-mon[32794]: Deploying daemon prometheus.a on smithi172 2024-09-13T14:26:40.907 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled node-exporter update... 2024-09-13T14:26:40.907 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:40 smithi172 ceph-mon[32794]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:26:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:40 smithi154 ceph-mon[29001]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:26:40.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:40 smithi154 ceph-mon[39074]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T14:26:41.901 DEBUG:teuthology.orchestra.run.smithi154:node-exporter.a> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-13T14:26:41.904 DEBUG:teuthology.orchestra.run.smithi172:node-exporter.b> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-13T14:26:41.906 INFO:tasks.cephadm:Adding alertmanager.a on smithi154 2024-09-13T14:26:41.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi154=a' 2024-09-13T14:26:42.139 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:26:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[29001]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi154=a;smithi172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[29001]: Saving service node-exporter spec with placement smithi154=a;smithi172=b;count:2 2024-09-13T14:26:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[29001]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-13T14:26:42.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[39074]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi154=a;smithi172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:42.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[39074]: Saving service node-exporter spec with placement smithi154=a;smithi172=b;count:2 2024-09-13T14:26:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:41 smithi154 ceph-mon[39074]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-13T14:26:42.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:41 smithi172 ceph-mon[32794]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi154=a;smithi172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:41 smithi172 ceph-mon[32794]: Saving service node-exporter spec with placement smithi154=a;smithi172=b;count:2 2024-09-13T14:26:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:41 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:41 smithi172 ceph-mon[32794]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-13T14:26:44.444 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled alertmanager update... 2024-09-13T14:26:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:44 smithi154 ceph-mon[29001]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:26:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:44 smithi154 ceph-mon[39074]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:26:44.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:44 smithi172 ceph-mon[32794]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:26:45.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:45 smithi154 ceph-mon[29001]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:45.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:45 smithi154 ceph-mon[29001]: Saving service alertmanager spec with placement smithi154=a;count:1 2024-09-13T14:26:45.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:45 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:45.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:45 smithi154 ceph-mon[39074]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:45.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:45 smithi154 ceph-mon[39074]: Saving service alertmanager spec with placement smithi154=a;count:1 2024-09-13T14:26:45.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:45 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:45.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:45 smithi172 ceph-mon[32794]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:45.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:45 smithi172 ceph-mon[32794]: Saving service alertmanager spec with placement smithi154=a;count:1 2024-09-13T14:26:45.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:45 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:45.881 DEBUG:teuthology.orchestra.run.smithi154:alertmanager.a> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-13T14:26:45.884 INFO:tasks.cephadm:Adding grafana.a on smithi172 2024-09-13T14:26:45.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi172=a' 2024-09-13T14:26:46.343 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:26:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:46 smithi154 ceph-mon[29001]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:46.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:46 smithi154 ceph-mon[39074]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:46.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:46 smithi172 ceph-mon[32794]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:48 smithi154 ceph-mon[29001]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:48.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:48 smithi154 ceph-mon[39074]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:48.733 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:48 smithi172 ceph-mon[32794]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:50 smithi154 ceph-mon[29001]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:50 smithi154 ceph-mon[39074]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:50.722 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:50 smithi172 ceph-mon[32794]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:50.731 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled grafana update... 2024-09-13T14:26:51.491 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:26:51 smithi172 systemd[1]: Started Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:26:51.723 DEBUG:teuthology.orchestra.run.smithi172:grafana.a> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@grafana.a.service 2024-09-13T14:26:51.726 INFO:tasks.cephadm:Setting up client nodes... 2024-09-13T14:26:51.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:26:51.938 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:51.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: Saving service grafana spec with placement smithi172=a;count:1 2024-09-13T14:26:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:26:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:51 smithi172 ceph-mon[32794]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:52.059 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: Saving service grafana spec with placement smithi172=a;count:1 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[29001]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: Saving service grafana spec with placement smithi172=a;count:1 2024-09-13T14:26:52.060 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.061 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.061 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.061 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' 2024-09-13T14:26:52.061 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T14:26:52.061 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:51 smithi154 ceph-mon[39074]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:26:52.641 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: ignoring --setuser ceph since I am not root 2024-09-13T14:26:52.641 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: ignoring --setgroup ceph since I am not root 2024-09-13T14:26:52.642 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-mgr[34635]: -- 172.21.15.172:0/3012703067 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559caf72ad00 con 0x559caf6ff000 2024-09-13T14:26:52.642 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:52.435+0000 7fa787a55040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:26:52.642 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:52.492+0000 7fa787a55040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:26:52.642 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:52.640+0000 7fa787a55040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:26:52.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setuser ceph since I am not root 2024-09-13T14:26:52.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setgroup ceph since I am not root 2024-09-13T14:26:52.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:52.439+0000 7f7c8aaaa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:26:52.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:52.495+0000 7f7c8aaaa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:26:52.643 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:52.642+0000 7f7c8aaaa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:26:52.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:52.709+0000 7f7c8aaaa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:26:52.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:52.855+0000 7f7c8aaaa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:26:52.990 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:52.706+0000 7fa787a55040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:26:52.990 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:52.851+0000 7fa787a55040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:26:53.295 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:52.998+0000 7f7c8aaaa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:26:53.296 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:52 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:52.992+0000 7fa787a55040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:26:53.568 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:53 smithi154 ceph-mon[29001]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:26:53.568 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:53 smithi154 ceph-mon[29001]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:26:53.568 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:53 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:53.316+0000 7f7c8aaaa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:26:53.568 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:53 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:53.378+0000 7f7c8aaaa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:26:53.569 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:53 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:53.566+0000 7f7c8aaaa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:26:53.569 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:53 smithi154 ceph-mon[39074]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:26:53.569 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:53 smithi154 ceph-mon[39074]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:26:53.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:53 smithi172 ceph-mon[32794]: from='mgr.14148 172.21.15.154:0/2697089273' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T14:26:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:53 smithi172 ceph-mon[32794]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-13T14:26:53.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:53 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:53.301+0000 7fa787a55040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:26:53.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:53 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:53.360+0000 7fa787a55040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:26:53.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:53 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:53.542+0000 7fa787a55040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:26:53.954 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2024-09-13T14:26:53.954 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCtS+Rm9KGkOBAA4tXFI+foGOq4WQQczHmGcg== 2024-09-13T14:26:54.680 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1760815299' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:54.699 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:54.699 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:26:54.699 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:54.471+0000 7f7c8aaaa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:26:54.699 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:54.538+0000 7f7c8aaaa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:26:54.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1760815299' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:54.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:54.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:54 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:26:54.701 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:26:54.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-13T14:26:54.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T14:26:54.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T14:26:54.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:54 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1760815299' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:54.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:54 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:54.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:54 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:26:54.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:54 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:54.414+0000 7fa787a55040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:26:54.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:54 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:54.480+0000 7fa787a55040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:26:54.942 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:26:55.341 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.224+0000 7fa787a55040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:26:55.515 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:55.258+0000 7f7c8aaaa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:26:55.515 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:55.374+0000 7f7c8aaaa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:26:55.702 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.346+0000 7fa787a55040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:26:55.703 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.488+0000 7fa787a55040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:26:55.703 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.548+0000 7fa787a55040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:26:55.703 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.701+0000 7fa787a55040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:26:55.788 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:55.513+0000 7f7c8aaaa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:26:55.788 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:55.575+0000 7f7c8aaaa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:26:55.788 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:55.720+0000 7f7c8aaaa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:26:55.788 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:55.787+0000 7f7c8aaaa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:26:55.989 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.770+0000 7fa787a55040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:26:55.989 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.830+0000 7fa787a55040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:26:55.989 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:55.988+0000 7fa787a55040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:26:56.066 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:55.847+0000 7f7c8aaaa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:26:56.066 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:56.001+0000 7f7c8aaaa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:26:56.066 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:56.065+0000 7f7c8aaaa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:26:56.240 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:56.058+0000 7fa787a55040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:26:56.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:56.127+0000 7fa787a55040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:26:56.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:56.211+0000 7fa787a55040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:26:56.332 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:56.131+0000 7f7c8aaaa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:26:56.333 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:56.212+0000 7f7c8aaaa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:26:56.333 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:56.272+0000 7f7c8aaaa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:26:56.333 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:56.331+0000 7f7c8aaaa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:26:56.540 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:56.271+0000 7fa787a55040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:26:56.540 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:56.347+0000 7fa787a55040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:26:56.788 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2024-09-13T14:26:56.788 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCwS+Rml5+3LhAARZqrDKE8KC4XE/vdNsfIBw== 2024-09-13T14:26:57.102 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-mon[32794]: from='client.? 172.21.15.172:0/3726377675' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:57.102 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:57.102 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:56 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:26:57.116 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-mon[29001]: from='client.? 172.21.15.172:0/3726377675' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:57.116 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:57.116 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:26:57.116 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-mon[39074]: from='client.? 172.21.15.172:0/3726377675' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:57.117 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T14:26:57.117 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:56 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T14:26:57.354 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:57.167+0000 7fa787a55040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:26:57.354 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: [13/Sep/2024:14:26:57] ENGINE Bus STARTING 2024-09-13T14:26:57.354 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: CherryPy Checker: 2024-09-13T14:26:57.354 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: The Application mounted at '' has an empty config. 2024-09-13T14:26:57.354 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: 2024-09-13T14:26:57.354 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: [13/Sep/2024:14:26:57] ENGINE Serving on http://:::9283 2024-09-13T14:26:57.354 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[34631]: [13/Sep/2024:14:26:57] ENGINE Bus STARTED 2024-09-13T14:26:57.381 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:57.115+0000 7f7c8aaaa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:26:57.381 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:26:57] ENGINE Bus STARTING 2024-09-13T14:26:57.381 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: CherryPy Checker: 2024-09-13T14:26:57.381 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: The Application mounted at '' has an empty config. 2024-09-13T14:26:57.381 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:57.381 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:26:57] ENGINE Serving on http://:::9283 2024-09-13T14:26:57.382 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:26:57] ENGINE Bus STARTED 2024-09-13T14:26:57.382 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:26:57] "GET /metrics HTTP/1.1" 200 23688 "" "Prometheus/2.43.0" 2024-09-13T14:26:57.432 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-13T14:26:57.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-13T14:26:57.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T14:26:57.469 INFO:tasks.cephadm:Creating RBD pool 2024-09-13T14:26:57.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool create rbd 8 2024-09-13T14:26:57.676 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:26:58.106 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: Active manager daemon y restarted 2024-09-13T14:26:58.106 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: Activating manager daemon y 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: mgrmap e17: y(active, starting, since 0.00641496s), standbys: x 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:26:58.107 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: Manager daemon y is now available 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:26:58.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: Standby manager daemon x restarted 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: Standby manager daemon x started 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:58.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:57 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:58.126 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: Active manager daemon y restarted 2024-09-13T14:26:58.126 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: Activating manager daemon y 2024-09-13T14:26:58.126 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:26:58.126 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: mgrmap e17: y(active, starting, since 0.00641496s), standbys: x 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:58.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: Manager daemon y is now available 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: Standby manager daemon x restarted 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: Standby manager daemon x started 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:58.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: Active manager daemon y restarted 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: Activating manager daemon y 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: mgrmap e17: y(active, starting, since 0.00641496s), standbys: x 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:26:58.129 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:26:58.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: Manager daemon y is now available 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: Standby manager daemon x restarted 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: Standby manager daemon x started 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2854569532' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:26:58.132 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:58.132 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:57 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:58.440+0000 7f7be3019640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:26:58] ENGINE Error in HTTPServer.serve 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: Traceback (most recent call last): 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._connections.run(self.expiration_interval) 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._run(expiration_interval) 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:26:58.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: s = self.context.wrap_socket( 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: return self.sslsocket_class._create( 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self.do_handshake() 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: self._sslobj.do_handshake() 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:26:58.722 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:26:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:26:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:59 smithi154 ceph-mon[29001]: mgrmap e18: y(active, since 1.01478s), standbys: x 2024-09-13T14:26:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:59 smithi154 ceph-mon[29001]: [13/Sep/2024:14:26:58] ENGINE Bus STARTING 2024-09-13T14:26:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:26:59 smithi154 ceph-mon[29001]: [13/Sep/2024:14:26:58] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:26:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:59 smithi154 ceph-mon[39074]: mgrmap e18: y(active, since 1.01478s), standbys: x 2024-09-13T14:26:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:59 smithi154 ceph-mon[39074]: [13/Sep/2024:14:26:58] ENGINE Bus STARTING 2024-09-13T14:26:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:26:59 smithi154 ceph-mon[39074]: [13/Sep/2024:14:26:58] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:26:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:59 smithi172 ceph-mon[32794]: mgrmap e18: y(active, since 1.01478s), standbys: x 2024-09-13T14:26:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:59 smithi172 ceph-mon[32794]: [13/Sep/2024:14:26:58] ENGINE Bus STARTING 2024-09-13T14:26:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:26:59 smithi172 ceph-mon[32794]: [13/Sep/2024:14:26:58] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:27:00.149 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[29001]: [13/Sep/2024:14:26:58] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:27:00.149 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[39074]: [13/Sep/2024:14:26:58] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:27:00.149 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[39074]: [13/Sep/2024:14:26:58] ENGINE Bus STARTED 2024-09-13T14:27:00.153 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'rbd' created 2024-09-13T14:27:00.423 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:00 smithi172 ceph-mon[32794]: [13/Sep/2024:14:26:58] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:27:00.423 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:00 smithi172 ceph-mon[32794]: [13/Sep/2024:14:26:58] ENGINE Bus STARTED 2024-09-13T14:27:00.423 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:00 smithi172 ceph-mon[32794]: [13/Sep/2024:14:26:58] ENGINE Error in HTTPServer.serve 2024-09-13T14:27:00.423 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-09-13T14:27:00.423 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._run(expiration_interval) 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: s = self.context.wrap_socket( 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:27:00.424 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.sslsocket_class._create( 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout: self.do_handshake() 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._sslobj.do_handshake() 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:00 smithi172 ceph-mon[32794]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:00 smithi172 ceph-mon[32794]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:00 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2856677920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:27:00.425 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:00 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:27:00.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[29001]: [13/Sep/2024:14:26:58] ENGINE Bus STARTED 2024-09-13T14:27:00.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[29001]: [13/Sep/2024:14:26:58] ENGINE Error in HTTPServer.serve 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:27:00.432 INFO:journalctl@ceph.mon.a.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout: self.do_handshake() 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[29001]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[29001]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2856677920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[39074]: [13/Sep/2024:14:26:58] ENGINE Error in HTTPServer.serve 2024-09-13T14:27:00.433 INFO:journalctl@ceph.mon.c.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:27:00.434 INFO:journalctl@ceph.mon.c.smithi154.stdout: self.do_handshake() 2024-09-13T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[39074]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[39074]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-13T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2856677920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:27:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:00 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-13T14:27:00.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- rbd pool init rbd 2024-09-13T14:27:01.005 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:01.378 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:27:01.378 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[29001]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:27:01.378 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.378 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.378 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.378 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.378 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:27:01.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[39074]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:27:01.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:01 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:01 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-13T14:27:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:01 smithi172 ceph-mon[32794]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T14:27:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:01 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:01 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:01 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:01 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:02.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:27:02.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi172 to 4016M 2024-09-13T14:27:02.411 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.412 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.412 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.412 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:02.412 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:27:02.412 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:27:02.412 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:02 smithi172 ceph-mon[32794]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:27:02.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:02.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:27:02.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi172 to 4016M 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.466 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[29001]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.467 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi172 to 4016M 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:27:02.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:27:02.469 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:27:02.469 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:02 smithi154 ceph-mon[39074]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-13T14:27:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:27:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:27:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:27:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2551980207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2551980207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:27:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:03 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:27:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2551980207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:27:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:03 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-13T14:27:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[29001]: Deploying daemon alertmanager.a on smithi154 2024-09-13T14:27:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[29001]: pgmap v9: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[29001]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[39074]: Deploying daemon alertmanager.a on smithi154 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[39074]: pgmap v9: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:27:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:04 smithi154 ceph-mon[39074]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:27:04.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:04 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:04 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:27:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:04 smithi172 ceph-mon[32794]: Deploying daemon alertmanager.a on smithi154 2024-09-13T14:27:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:04 smithi172 ceph-mon[32794]: pgmap v9: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:04 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-13T14:27:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:04 smithi172 ceph-mon[32794]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T14:27:05.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:05 smithi154 ceph-mon[39074]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:27:05.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:05 smithi154 ceph-mon[39074]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-13T14:27:05.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:05 smithi154 ceph-mon[29001]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:27:05.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:05 smithi154 ceph-mon[29001]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-13T14:27:05.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:05 smithi172 ceph-mon[32794]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T14:27:05.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:05 smithi172 ceph-mon[32794]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-09-13T14:27:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:06 smithi154 ceph-mon[29001]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:27:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:06 smithi154 ceph-mon[29001]: Cluster is now healthy 2024-09-13T14:27:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:06 smithi154 ceph-mon[29001]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:27:06.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:06 smithi154 ceph-mon[39074]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:27:06.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:06 smithi154 ceph-mon[39074]: Cluster is now healthy 2024-09-13T14:27:06.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:06 smithi154 ceph-mon[39074]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:27:06.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:06 smithi172 ceph-mon[32794]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:27:06.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:06 smithi172 ceph-mon[32794]: Cluster is now healthy 2024-09-13T14:27:06.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:06 smithi172 ceph-mon[32794]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T14:27:07.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph config log 1 --format=json 2024-09-13T14:27:07.659 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:27:07] "GET /metrics HTTP/1.1" 200 23688 "" "Prometheus/2.43.0" 2024-09-13T14:27:07.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:07 smithi154 ceph-mon[29001]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-13T14:27:07.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:07 smithi154 ceph-mon[39074]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-13T14:27:07.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:07 smithi172 ceph-mon[32794]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-09-13T14:27:09.575 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:10 smithi154 ceph-mon[29001]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 341 B/s wr, 19 op/s 2024-09-13T14:27:10.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:10 smithi154 ceph-mon[39074]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 341 B/s wr, 19 op/s 2024-09-13T14:27:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:10 smithi172 ceph-mon[32794]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 341 B/s wr, 19 op/s 2024-09-13T14:27:11.568 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 podman[67332]: 2024-09-13 14:27:11.423014452 +0000 UTC m=+0.259957692 container create 25176e6d1892e3caccdd5c3296a6926d95320505c0cb1050015746ad4794d7d9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:27:11.851 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 podman[67332]: 2024-09-13 14:27:11.750564374 +0000 UTC m=+0.587507572 container init 25176e6d1892e3caccdd5c3296a6926d95320505c0cb1050015746ad4794d7d9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:27:11.851 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 podman[67332]: 2024-09-13 14:27:11.753312145 +0000 UTC m=+0.590255343 container start 25176e6d1892e3caccdd5c3296a6926d95320505c0cb1050015746ad4794d7d9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:27:11.851 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.764Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:27:11.851 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.764Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:27:11.851 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.766Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.154 port=9094 2024-09-13T14:27:11.852 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.767Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:27:11.852 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.790Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:27:11.852 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.790Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:27:11.852 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.792Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:27:11.852 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:11.792Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:27:11.852 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 bash[67332]: 25176e6d1892e3caccdd5c3296a6926d95320505c0cb1050015746ad4794d7d9 2024-09-13T14:27:12.127 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:11 smithi154 systemd[1]: Started Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:27:12.127 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:27:11] ENGINE Bus STOPPING 2024-09-13T14:27:12.297 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:27:12.460 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 273 B/s wr, 0 op/s 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 273 B/s wr, 0 op/s 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:12.463 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:27:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:27:12.463 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:27:12] ENGINE Bus STOPPED 2024-09-13T14:27:12.463 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:27:12] ENGINE Bus STARTING 2024-09-13T14:27:12.463 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:27:12] ENGINE Serving on http://:::9283 2024-09-13T14:27:12.463 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:27:12] ENGINE Bus STARTED 2024-09-13T14:27:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 273 B/s wr, 0 op/s 2024-09-13T14:27:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:27:12.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:12.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:12 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:13.030 INFO:teuthology.orchestra.run.smithi154.stdout:[{"version":25,"timestamp":"2024-09-13T14:27:11.962251+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/GRAFANA_API_SSL_VERIFY","new_value":"false"}]}] 2024-09-13T14:27:13.030 INFO:tasks.ceph_manager:config epoch is 25 2024-09-13T14:27:13.031 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:27:13.031 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:27:13.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:27:13.235 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:13.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[29001]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T14:27:13.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:27:13.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[29001]: Deploying daemon grafana.a on smithi172 2024-09-13T14:27:13.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/975784135' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:27:13.341 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[39074]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T14:27:13.342 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:27:13.342 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[39074]: Deploying daemon grafana.a on smithi172 2024-09-13T14:27:13.342 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/975784135' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:27:13.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:13 smithi172 ceph-mon[32794]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T14:27:13.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:13 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T14:27:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:13 smithi172 ceph-mon[32794]: Deploying daemon grafana.a on smithi172 2024-09-13T14:27:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:13 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/975784135' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-13T14:27:13.970 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:13.767Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000237416s 2024-09-13T14:27:14.356 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:14 smithi154 ceph-mon[29001]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 241 B/s wr, 0 op/s 2024-09-13T14:27:14.356 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:14 smithi154 ceph-mon[29001]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:27:14.356 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:14 smithi154 ceph-mon[39074]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 241 B/s wr, 0 op/s 2024-09-13T14:27:14.357 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:14 smithi154 ceph-mon[39074]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:27:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:14 smithi172 ceph-mon[32794]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 241 B/s wr, 0 op/s 2024-09-13T14:27:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:14 smithi172 ceph-mon[32794]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T14:27:15.006 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:27:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:15 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/4101833444' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:27:15.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:15 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/4101833444' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:27:15.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:15 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/4101833444' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:27:15.686 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":20,"flags":0,"active_gid":24329,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":302860313},{"type":"v1","addr":"172.21.15.154:6801","nonce":302860313}]},"active_addr":"172.21.15.154:6801/302860313","active_change":"2024-09-13T14:26:57.118185+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24337,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":1379294727}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":3255496958}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":3924873105}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":18164567}]}]} 2024-09-13T14:27:15.692 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:27:15.693 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:27:15.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:27:15.890 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:16.292 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:16 smithi154 ceph-mon[29001]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 215 B/s wr, 0 op/s 2024-09-13T14:27:16.292 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:16 smithi154 ceph-mon[39074]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 215 B/s wr, 0 op/s 2024-09-13T14:27:16.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:16 smithi172 ceph-mon[32794]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 215 B/s wr, 0 op/s 2024-09-13T14:27:17.600 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:27:17.600 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":66,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T14:27:14.125984+0000","last_up_change":"2024-09-13T14:26:26.512111+0000","last_in_change":"2024-09-13T14:26:09.317846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6803","nonce":1775775819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6805","nonce":1775775819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6809","nonce":1775775819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6807","nonce":1775775819}]},"public_addr":"172.21.15.154:6803/1775775819","cluster_addr":"172.21.15.154:6805/1775775819","heartbeat_back_addr":"172.21.15.154:6809/1775775819","heartbeat_front_addr":"172.21.15.154:6807/1775775819","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6811","nonce":3038007336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6813","nonce":3038007336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6817","nonce":3038007336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6815","nonce":3038007336}]},"public_addr":"172.21.15.154:6811/3038007336","cluster_addr":"172.21.15.154:6813/3038007336","heartbeat_back_addr":"172.21.15.154:6817/3038007336","heartbeat_front_addr":"172.21.15.154:6815/3038007336","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6819","nonce":1952703304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6821","nonce":1952703304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6825","nonce":1952703304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6823","nonce":1952703304}]},"public_addr":"172.21.15.154:6819/1952703304","cluster_addr":"172.21.15.154:6821/1952703304","heartbeat_back_addr":"172.21.15.154:6825/1952703304","heartbeat_front_addr":"172.21.15.154:6823/1952703304","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6827","nonce":1896570703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6829","nonce":1896570703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6833","nonce":1896570703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6831","nonce":1896570703}]},"public_addr":"172.21.15.154:6827/1896570703","cluster_addr":"172.21.15.154:6829/1896570703","heartbeat_back_addr":"172.21.15.154:6833/1896570703","heartbeat_front_addr":"172.21.15.154:6831/1896570703","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:27:17.601 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:27:17] "GET /metrics HTTP/1.1" 200 28519 "" "Prometheus/2.43.0" 2024-09-13T14:27:18.352 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:27:18.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:27:18.474 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:18 smithi154 ceph-mon[29001]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:27:18.474 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:18 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3838454486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:27:18.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:18 smithi154 ceph-mon[39074]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:27:18.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:18 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3838454486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:27:18.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:18 smithi172 ceph-mon[32794]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-13T14:27:18.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:18 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3838454486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:27:18.550 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:20.290 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:20 smithi154 ceph-mon[29001]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:20.290 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:20 smithi154 ceph-mon[39074]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:20.455 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:27:20.456 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":66,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T14:27:14.125984+0000","last_up_change":"2024-09-13T14:26:26.512111+0000","last_in_change":"2024-09-13T14:26:09.317846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6803","nonce":1775775819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6805","nonce":1775775819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6809","nonce":1775775819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6807","nonce":1775775819}]},"public_addr":"172.21.15.154:6803/1775775819","cluster_addr":"172.21.15.154:6805/1775775819","heartbeat_back_addr":"172.21.15.154:6809/1775775819","heartbeat_front_addr":"172.21.15.154:6807/1775775819","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6811","nonce":3038007336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6813","nonce":3038007336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6817","nonce":3038007336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6815","nonce":3038007336}]},"public_addr":"172.21.15.154:6811/3038007336","cluster_addr":"172.21.15.154:6813/3038007336","heartbeat_back_addr":"172.21.15.154:6817/3038007336","heartbeat_front_addr":"172.21.15.154:6815/3038007336","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6819","nonce":1952703304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6821","nonce":1952703304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6825","nonce":1952703304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6823","nonce":1952703304}]},"public_addr":"172.21.15.154:6819/1952703304","cluster_addr":"172.21.15.154:6821/1952703304","heartbeat_back_addr":"172.21.15.154:6825/1952703304","heartbeat_front_addr":"172.21.15.154:6823/1952703304","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6827","nonce":1896570703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6829","nonce":1896570703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6833","nonce":1896570703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6831","nonce":1896570703}]},"public_addr":"172.21.15.154:6827/1896570703","cluster_addr":"172.21.15.154:6829/1896570703","heartbeat_back_addr":"172.21.15.154:6833/1896570703","heartbeat_front_addr":"172.21.15.154:6831/1896570703","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:27:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:20 smithi172 ceph-mon[32794]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:21.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:27:21.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:27:21.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:27:21.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:27:21.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:27:21.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:27:21.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:27:21.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:27:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:21 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/200925090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:27:21.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:21 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/200925090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:27:21.483 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.484 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:21 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/200925090' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:27:21.503 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.503 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.505 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.510 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.514 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.538 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:21.891 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:27:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:27:21.769Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002321282s 2024-09-13T14:27:22.403 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:22 smithi154 ceph-mon[29001]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:22.403 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:22 smithi154 ceph-mon[39074]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:22.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:22 smithi172 ceph-mon[32794]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:23.741 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:23 smithi172 systemd[1]: Starting Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:27:24.086 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:23 smithi172 podman[62015]: 2024-09-13 14:27:23.676331681 +0000 UTC m=+0.015399344 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-13T14:27:24.086 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:23 smithi172 podman[62015]: 2024-09-13 14:27:23.783169522 +0000 UTC m=+0.122237172 container create 9507837610ffa2ee36997c9564a119383e7cd9f58072197292bce9edcf6c88ac (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, version=8.5, name=ubi8, io.openshift.expose-services=, vendor=Red Hat, Inc., release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, maintainer=Paul Cuzner , vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public) 2024-09-13T14:27:24.086 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 podman[62015]: 2024-09-13 14:27:24.002093341 +0000 UTC m=+0.341160991 container init 9507837610ffa2ee36997c9564a119383e7cd9f58072197292bce9edcf6c88ac (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, vcs-type=git, distribution-scope=public, io.buildah.version=1.32.0, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, architecture=x86_64, maintainer=Paul Cuzner , version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-13T14:27:24.086 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 podman[62015]: 2024-09-13 14:27:24.004922542 +0000 UTC m=+0.343990192 container start 9507837610ffa2ee36997c9564a119383e7cd9f58072197292bce9edcf6c88ac (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, architecture=x86_64, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, version=8.5, vcs-type=git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-09-13T14:27:24.086 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 bash[62015]: 9507837610ffa2ee36997c9564a119383e7cd9f58072197292bce9edcf6c88ac 2024-09-13T14:27:24.338 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-mon[32794]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:24.338 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.338 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.338 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.338 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 systemd[1]: Started Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.162738152Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.162971912Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.162990815Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.16299813Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163004477Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163011108Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163017222Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163023321Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163030229Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:27:24.339 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163037197Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163042969Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163049231Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163055488Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=settings t=2024-09-13T14:27:24.163061876Z level=info msg="App mode production" 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=sqlstore t=2024-09-13T14:27:24.163113093Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=sqlstore t=2024-09-13T14:27:24.163130254Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.164009164Z level=info msg="Starting DB migrations" 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.164083512Z level=info msg="Executing migration" id="create migration_log table" 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.164592693Z level=info msg="Executing migration" id="create user table" 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.165027105Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.165969713Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.166884757Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-13T14:27:24.340 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.16759202Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.16848413Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.169483451Z level=info msg="Executing migration" id="create user table v2" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.170260266Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.171063289Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.171510261Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.172351719Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.173201418Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.174066362Z level=info msg="Executing migration" id="Update user table charset" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.174959374Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.175916268Z level=info msg="Executing migration" id="Add missing user data" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.176669512Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.17763968Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-13T14:27:24.341 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.178396145Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.18130389Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.184155517Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.18685215Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.189889127Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.192737387Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.195460715Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.198082796Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.200850711Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.201238766Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.204104756Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.204524691Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.206970379Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-13T14:27:24.342 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.209758432Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.212399995Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.215247946Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.218030252Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.220809701Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.221229771Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.223751759Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.224265566Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-13T14:27:24.343 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.224582344Z level=info msg="Executing migration" id="create star table" 2024-09-13T14:27:24.344 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.224956138Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-13T14:27:24.344 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.22757122Z level=info msg="Executing migration" id="create org table v1" 2024-09-13T14:27:24.344 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.23047088Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-13T14:27:24.344 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.233041401Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-13T14:27:24.344 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.235739694Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-13T14:27:24.344 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.236128684Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.238862241Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.241604158Z level=info msg="Executing migration" id="Update org table charset" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.244276772Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.24462645Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.2473052Z level=info msg="Executing migration" id="create dashboard table" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.250099176Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.252695866Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.253267866Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.254135272Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.254852316Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.255661012Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-13T14:27:24.345 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.256552205Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.257391266Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.258342075Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.259051671Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.259409246Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.260224543Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.260540798Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.261100226Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.261914101Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.262446319Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.263371748Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.264138228Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.264948721Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.265390317Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-13T14:27:24.346 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.266255914Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.266659238Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.267190069Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.268039514Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.268579401Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.269113632Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.270074482Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.270749943Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.271779639Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.272198941Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.27303222Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.273439103Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.274352952Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.275403122Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-13T14:27:24.347 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.276095979Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.277071873Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.277790571Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.278834715Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.279567769Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.280421575Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.281347474Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.282184031Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-13T14:27:24.348 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.283173187Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.28391653Z level=info msg="Executing migration" id="create data_source table" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.284999068Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.285504977Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.285952667Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.288820411Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.291735082Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.294799742Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.297565902Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.30029341Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.303295429Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.306065694Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.308774117Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-13T14:27:24.349 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.311775249Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.314386264Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.317418874Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.32031319Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.322945039Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.325726844Z level=info msg="Executing migration" id="Add uid column" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.328900824Z level=info msg="Executing migration" id="Update uid value" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.33223964Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.334846311Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.337722953Z level=info msg="Executing migration" id="create api_key table" 2024-09-13T14:27:24.350 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.340311831Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-13T14:27:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[29001]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[39074]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:24 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.343015495Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.345794855Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.348511842Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.351285653Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.354071876Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.356894329Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.35928913Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.359696983Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.362024663Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-13T14:27:24.589 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.36465109Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.367436865Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.37019508Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.372819438Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.375493925Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.37822646Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.380779367Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.383614125Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.384295735Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.385181215Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.386180625Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.386929897Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-13T14:27:24.590 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.387725739Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.38850355Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.389461002Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.390234265Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.391049706Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.391881055Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.392773318Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.393748392Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.394477545Z level=info msg="Executing migration" id="create quota table v1" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.395474895Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.395997527Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.396289674Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.396676614Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-13T14:27:24.591 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.397085214Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-13T14:27:24.592 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.39762131Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-13T14:27:24.592 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.397922346Z level=info msg="Executing migration" id="create session table" 2024-09-13T14:27:24.592 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.398333064Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-13T14:27:24.593 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.398597371Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-13T14:27:24.593 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.398904715Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-13T14:27:24.593 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.399285907Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-13T14:27:24.593 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.399657246Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-13T14:27:24.593 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.399951665Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-13T14:27:24.593 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.400205243Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.400463704Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.400735798Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.40112411Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.401382854Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.40199214Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.4022976Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.402927607Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.403498086Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.403764552Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.404173296Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.404605561Z level=info msg="Executing migration" id="create alert table v1" 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.405379232Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-13T14:27:24.594 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.40623188Z level=info msg="Executing migration" id="add index alert state" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.406658073Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.407103321Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.407468525Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.407888396Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.408283033Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.409723679Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.410098205Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.41050835Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.410854082Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.411200759Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.411545959Z level=info msg="Executing migration" id="Add column is_default" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.412196724Z level=info msg="Executing migration" id="Add column frequency" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.41285995Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-13T14:27:24.595 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.4135088Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.414174738Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.414602005Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.414900928Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.415186823Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.415564695Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.416002084Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.416374917Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.416743975Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.417150178Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.417757723Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.418383181Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-13T14:27:24.596 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.418685571Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.419106284Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.419479061Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.42013325Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.420438411Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.420855588Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.4212577Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.421557069Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.421998414Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.422397989Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.422819037Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.423276416Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.423741652Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-13T14:27:24.597 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.424607514Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.424917855Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.425567574Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.425997549Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.426664126Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.427519939Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.427977812Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.42882039Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.430448869Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.431326735Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.431777837Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.432721383Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.433442516Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-13T14:27:24.598 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.434301954Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.435038224Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.435729658Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.436626898Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.439461013Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.442266989Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.44298588Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.445787555Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.448782429Z level=info msg="Executing migration" id="Move region to single row" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.451534869Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-13T14:27:24.599 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.451997006Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.454807592Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.457614363Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.460547224Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.463330939Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.466142789Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.469078701Z level=info msg="Executing migration" id="create test_data table" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.471866599Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-13T14:27:24.600 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.474783795Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.477555833Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.480477842Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.483320682Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.486022595Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.488530349Z level=info msg="Executing migration" id="create team table" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.49132479Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.494068668Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.496804428Z level=info msg="Executing migration" id="create team member table" 2024-09-13T14:27:24.601 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.499569264Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.50213262Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.504858394Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.507572265Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.510109393Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.512860706Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.51547595Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.51835754Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-13T14:27:24.602 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.520808566Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.523583196Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.526328339Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.528880337Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.529732214Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.530644335Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.531385319Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.532251321Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-13T14:27:24.603 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.532757676Z level=info msg="Executing migration" id="create tag table" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.533159537Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.533979797Z level=info msg="Executing migration" id="create login attempt table" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.534790603Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.535611138Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.536594203Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.538695447Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.539638956Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.540413207Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-13T14:27:24.604 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.541242409Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.541679974Z level=info msg="Executing migration" id="create user auth table" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.542066921Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.542925762Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.543765566Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.544616315Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.545495119Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.546357174Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-13T14:27:24.605 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.547240391Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.54822509Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.549092827Z level=info msg="Executing migration" id="create server_lock table" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.549923101Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.550780512Z level=info msg="Executing migration" id="create user auth token table" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.551655388Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.552530798Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.553493545Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.554362314Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-13T14:27:24.606 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.555320446Z level=info msg="Executing migration" id="create cache_data table" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.556161859Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.557144535Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.557840047Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.558828743Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.561612678Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.561992332Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.56485963Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.567774564Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-13T14:27:24.607 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.57057815Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.570978992Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.573626316Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.574266301Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.576434607Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.576986309Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.579210425Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.579653991Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-13T14:27:24.608 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.582517209Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-13T14:27:24.609 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.585321914Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-13T14:27:24.609 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.588099062Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-13T14:27:24.609 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.590572865Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-13T14:27:24.839 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.593481166Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-13T14:27:24.839 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.596204777Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-13T14:27:24.839 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.598986719Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-13T14:27:24.839 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.599594165Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-13T14:27:24.839 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.60230913Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-13T14:27:24.839 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.605015289Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.607541166Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.607961453Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.611528283Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.615004805Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.617898516Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.618319716Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.621104547Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.6238533Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.626472277Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.626922581Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.629660715Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-13T14:27:24.840 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.632413802Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.635074417Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.636054597Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.636889996Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.637841292Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.638537451Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.639365497Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.640272779Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.641076552Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.641938729Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.642751761Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.643724868Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.644387521Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.645326854Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-13T14:27:24.841 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.645776387Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-13T14:27:24.842 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.646585531Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-13T14:27:24.842 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.647485171Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-13T14:27:24.842 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.648343521Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.64921379Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.650059607Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.650925452Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.651878299Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.652318037Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.655257953Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.657954575Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.661003103Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-13T14:27:24.843 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.66156397Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.664469606Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.667323793Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.670152755Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.671077922Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.673937222Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.676734842Z level=info msg="Executing migration" id="create alert_image table" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.679219526Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.679797056Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.682580197Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.685210827Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.685694785Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:27:24.844 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.685777718Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.686066003Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.688859329Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.689372793Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.689495463Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.689615877Z level=info msg="alerts found to migrate" alerts=0 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.691498608Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.694449182Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.694962698Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.69780388Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.7005104Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.703243774Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.703566876Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-13T14:27:24.845 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.706252792Z level=info msg="Executing migration" id="create data_keys table" 2024-09-13T14:27:24.846 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.708908781Z level=info msg="Executing migration" id="create secrets table" 2024-09-13T14:27:24.846 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.711809105Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-13T14:27:24.846 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.716232099Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-13T14:27:24.846 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.717236006Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.718214033Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.722773421Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.727224965Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.728005945Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.728951056Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.729757517Z level=info msg="Executing migration" id="create permission table" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.730326502Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.731207405Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.73221237Z level=info msg="Executing migration" id="create role table" 2024-09-13T14:27:24.847 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.732932203Z level=info msg="Executing migration" id="add column display_name" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.733849272Z level=info msg="Executing migration" id="add column group_name" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.734807181Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.735784047Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.736438484Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.7369434Z level=info msg="Executing migration" id="create team role table" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.737782924Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.738783004Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.739621898Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-13T14:27:24.848 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.74027179Z level=info msg="Executing migration" id="create user role table" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.741130703Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.741996444Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.742839036Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.7433136Z level=info msg="Executing migration" id="create builtin role table" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.744175224Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.745078627Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.745920132Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.746958241Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.747490061Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.748324446Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.748765158Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.749192363Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-13T14:27:24.849 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.749569578Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.750001693Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.750991332Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.751394792Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.7518336Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.752123255Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.752489598Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.752934447Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.753296269Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.753573142Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.753955908Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-13T14:27:24.850 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.754339906Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.754767784Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.755729135Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.756030506Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.756434878Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.756862858Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.757283469Z level=info msg="Executing migration" id="add correlation config column" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.758253507Z level=info msg="Executing migration" id="create entity_events table" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.758678263Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.759088028Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.759166431Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.759432965Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:27:24.851 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.759508913Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.759754368Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.760167426Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.760636899Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.761083889Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.761550477Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.76199258Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.766207223Z level=info msg="Executing migration" id="Drop public config table" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.769268808Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.771850198Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.772305509Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-13T14:27:24.852 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.775137991Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.775604679Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.778727807Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.781386837Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.782409544Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.785327301Z level=info msg="Executing migration" id="add share column" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.788083662Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.790971831Z level=info msg="Executing migration" id="create file table" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.791638745Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.794432683Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.797205749Z level=info msg="Executing migration" id="create file_meta table" 2024-09-13T14:27:24.853 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.797667769Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-13T14:27:24.854 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.798143503Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-13T14:27:24.854 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.800862807Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-13T14:27:24.854 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.801321512Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-13T14:27:24.854 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.803559098Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-13T14:27:24.854 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.804118259Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-13T14:27:24.854 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.80628225Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-13T14:27:24.855 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.806601728Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-13T14:27:24.855 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.809310684Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-13T14:27:24.855 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.811927736Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-13T14:27:24.855 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.81470015Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-13T14:27:24.855 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.81507326Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-13T14:27:24.855 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.81777879Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-13T14:27:24.856 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.820221182Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-13T14:27:24.856 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.823056954Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-13T14:27:24.856 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.833330689Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-13T14:27:24.856 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.836052603Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-13T14:27:24.856 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.838769995Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-13T14:27:24.856 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.842612231Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-13T14:27:25.162 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.84298502Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-13T14:27:25.162 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.843804882Z level=info msg="Executing migration" id="create folder table" 2024-09-13T14:27:25.162 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.844223842Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-13T14:27:25.162 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.844723295Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-13T14:27:25.162 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.845526346Z level=info msg="Executing migration" id="Update folder title length" 2024-09-13T14:27:25.162 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.845851458Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=migrator t=2024-09-13T14:27:24.846703291Z level=info msg="migrations completed" performed=485 skipped=0 duration=682.640288ms 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=sqlstore t=2024-09-13T14:27:24.847118568Z level=info msg="Created default organization" 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=plugin.loader t=2024-09-13T14:27:24.901789763Z level=info msg="Plugin registered" pluginID=input 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=plugin.loader t=2024-09-13T14:27:24.929155873Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=plugin.loader t=2024-09-13T14:27:24.929176124Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=secrets t=2024-09-13T14:27:24.929219136Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=query_data t=2024-09-13T14:27:24.93224303Z level=info msg="Query Service initialization" 2024-09-13T14:27:25.163 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=live.push_http t=2024-09-13T14:27:24.934833785Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:27:25.384 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:25 smithi154 ceph-mon[29001]: Deploying daemon node-exporter.a on smithi154 2024-09-13T14:27:25.384 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:25 smithi154 ceph-mon[39074]: Deploying daemon node-exporter.a on smithi154 2024-09-13T14:27:25.483 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-mon[32794]: Deploying daemon node-exporter.a on smithi154 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=infra.usagestats.collector t=2024-09-13T14:27:25.764934824Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=provisioning.datasources t=2024-09-13T14:27:25.820020092Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=provisioning.datasources t=2024-09-13T14:27:25.852035525Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=provisioning.alerting t=2024-09-13T14:27:25.873790518Z level=info msg="starting to provision alerting" 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=provisioning.alerting t=2024-09-13T14:27:25.8738159Z level=info msg="finished to provision alerting" 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=grafanaStorageLogger t=2024-09-13T14:27:25.873965206Z level=info msg="storage starting" 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=ngalert.state.manager t=2024-09-13T14:27:25.874056866Z level=info msg="Warming state cache for startup" 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=http.server t=2024-09-13T14:27:25.876698351Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=sqlstore.transactions t=2024-09-13T14:27:25.884588918Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=ngalert.state.manager t=2024-09-13T14:27:25.884692007Z level=info msg="State cache has been initialized" states=0 duration=10.634158ms 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=ticker t=2024-09-13T14:27:25.884793255Z level=info msg=starting first_tick=2024-09-13T14:27:30Z 2024-09-13T14:27:26.232 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:27:25.884783016Z level=info msg="starting MultiOrg Alertmanager" 2024-09-13T14:27:26.233 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=sqlstore.transactions t=2024-09-13T14:27:25.918247913Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:27:26.233 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=sqlstore.transactions t=2024-09-13T14:27:25.959337401Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:27:26.233 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:27:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[62030]: logger=sqlstore.transactions t=2024-09-13T14:27:25.970327952Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-13T14:27:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:26 smithi154 ceph-mon[29001]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:26.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:26 smithi154 ceph-mon[39074]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:26.486 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:26 smithi172 ceph-mon[32794]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:27.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:27:27] "GET /metrics HTTP/1.1" 200 28518 "" "Prometheus/2.43.0" 2024-09-13T14:27:28.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:28 smithi154 ceph-mon[29001]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:28.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:28 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:28.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:28 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:28.361 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:28 smithi154 ceph-mon[39074]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:28.361 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:28 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:28.361 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:28 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:28 smithi172 ceph-mon[32794]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:28 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:28 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:30 smithi154 ceph-mon[29001]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:30.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:30 smithi154 ceph-mon[39074]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:30.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:30 smithi172 ceph-mon[32794]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:30.585 INFO:teuthology.orchestra.run.smithi154.stdout:77309411366 2024-09-13T14:27:30.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:27:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:32 smithi154 ceph-mon[29001]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:32.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:32 smithi154 ceph-mon[39074]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:32.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:32 smithi172 ceph-mon[32794]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:33.039 INFO:teuthology.orchestra.run.smithi154.stdout:163208757272 2024-09-13T14:27:33.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:27:34.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:34 smithi154 ceph-mon[39074]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:34.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:34 smithi154 ceph-mon[29001]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:34.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:34 smithi172 ceph-mon[32794]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:35.115 INFO:teuthology.orchestra.run.smithi154.stdout:128849018911 2024-09-13T14:27:35.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:27:35.545 INFO:teuthology.orchestra.run.smithi154.stdout:107374182434 2024-09-13T14:27:35.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:27:36.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:36 smithi154 ceph-mon[29001]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:36.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:36 smithi154 ceph-mon[39074]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:36.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:36 smithi172 ceph-mon[32794]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:36.662 INFO:teuthology.orchestra.run.smithi154.stdout:34359738416 2024-09-13T14:27:36.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:27:37.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:37 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:27:37] "GET /metrics HTTP/1.1" 200 28518 "" "Prometheus/2.43.0" 2024-09-13T14:27:37.848 INFO:teuthology.orchestra.run.smithi154.stdout:55834574892 2024-09-13T14:27:37.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:27:38.259 INFO:teuthology.orchestra.run.smithi154.stdout:197568495636 2024-09-13T14:27:38.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:27:38.286 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:38.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:38 smithi154 ceph-mon[29001]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:38 smithi154 ceph-mon[39074]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:38.471 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:38.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:38 smithi172 ceph-mon[32794]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:39.174 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:39.174 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:40.447 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:40 smithi154 ceph-mon[29001]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:40.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:40 smithi154 ceph-mon[39074]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:40 smithi172 ceph-mon[32794]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:41.720 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:41 smithi154 bash[69574]: Writing manifest to image destination 2024-09-13T14:27:42.407 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:42 smithi154 ceph-mon[29001]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:42.407 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:42 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:42.407 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:42 smithi154 podman[69574]: 2024-09-13 14:27:41.956086458 +0000 UTC m=+4.091888416 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T14:27:42.407 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:42 smithi154 ceph-mon[39074]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:42.408 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:42 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:42 smithi172 ceph-mon[32794]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:42 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:42.935 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:42 smithi154 podman[69574]: 2024-09-13 14:27:42.801626496 +0000 UTC m=+4.937428480 container create 93d3a2d692cab99b3413fe40f66b5f709edae21926d30dded6399960d337ca1b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:27:43.147 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:43.148 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:43.970 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 podman[69574]: 2024-09-13 14:27:43.751583904 +0000 UTC m=+5.887385850 container init 93d3a2d692cab99b3413fe40f66b5f709edae21926d30dded6399960d337ca1b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:27:43.970 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 podman[69574]: 2024-09-13 14:27:43.754804817 +0000 UTC m=+5.890606760 container start 93d3a2d692cab99b3413fe40f66b5f709edae21926d30dded6399960d337ca1b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.758Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.758Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:27:43.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:27:43.972 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:27:43.972 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:27:43.972 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:27:43.972 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:27:43.972 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:27:43.972 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:27:43.973 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:27:43.974 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.759Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:27:43.975 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.760Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:27:43.976 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[69984]: ts=2024-09-13T14:27:43.760Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:27:43.976 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 bash[69574]: 93d3a2d692cab99b3413fe40f66b5f709edae21926d30dded6399960d337ca1b 2024-09-13T14:27:43.976 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:27:43 smithi154 systemd[1]: Started Ceph node-exporter.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:27:44.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[29001]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:44.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[39074]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:44.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:44 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.389 INFO:teuthology.orchestra.run.smithi154.stdout:77309411370 2024-09-13T14:27:44.489 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:44 smithi172 ceph-mon[32794]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:44.489 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:44 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.489 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:44 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.489 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:44 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:44.740 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:44 smithi172 systemd[1]: Starting Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:27:45.205 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:44 smithi172 bash[62287]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-13T14:27:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:45 smithi154 ceph-mon[29001]: Deploying daemon node-exporter.b on smithi172 2024-09-13T14:27:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:45 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2686998861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:27:45.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:45 smithi154 ceph-mon[39074]: Deploying daemon node-exporter.b on smithi172 2024-09-13T14:27:45.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:45 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2686998861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:27:45.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:45 smithi172 ceph-mon[32794]: Deploying daemon node-exporter.b on smithi172 2024-09-13T14:27:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:45 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2686998861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:27:45.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:45 smithi172 bash[62287]: Getting image source signatures 2024-09-13T14:27:45.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:45 smithi172 bash[62287]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-13T14:27:45.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:45 smithi172 bash[62287]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-13T14:27:45.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:45 smithi172 bash[62287]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-13T14:27:46.020 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:46.039 INFO:teuthology.orchestra.run.smithi154.stdout:231928234000 2024-09-13T14:27:46.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:27:46.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:46 smithi154 ceph-mon[29001]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:46.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:46 smithi154 ceph-mon[39074]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:46.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:46 smithi172 ceph-mon[32794]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:47.240 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:46 smithi172 bash[62287]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-13T14:27:47.241 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 bash[62287]: Writing manifest to image destination 2024-09-13T14:27:47.272 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411370 for osd.2 2024-09-13T14:27:47.272 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:47.612 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:47 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:27:47] "GET /metrics HTTP/1.1" 200 28518 "" "Prometheus/2.43.0" 2024-09-13T14:27:47.643 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 podman[62287]: 2024-09-13 14:27:47.311168055 +0000 UTC m=+2.543245225 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-13T14:27:47.643 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 podman[62287]: 2024-09-13 14:27:47.401462576 +0000 UTC m=+2.633539778 container create 6e12b9645ca4e7f9d326570001c6cfacc973103cf6d67f0e0d6175728bc4ac2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:27:47.643 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 podman[62287]: 2024-09-13 14:27:47.63657828 +0000 UTC m=+2.868655440 container init 6e12b9645ca4e7f9d326570001c6cfacc973103cf6d67f0e0d6175728bc4ac2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:27:47.643 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 podman[62287]: 2024-09-13 14:27:47.639354443 +0000 UTC m=+2.871431604 container start 6e12b9645ca4e7f9d326570001c6cfacc973103cf6d67f0e0d6175728bc4ac2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.643Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.643Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.643Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.643Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:27:47.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:27:47.992 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:27:47.993 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:27:47.993 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:27:47.993 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:27:47.993 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:27:47.993 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:27:47.994 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:27:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.644Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.645Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[62360]: ts=2024-09-13T14:27:47.645Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 bash[62287]: 6e12b9645ca4e7f9d326570001c6cfacc973103cf6d67f0e0d6175728bc4ac2e 2024-09-13T14:27:47.996 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:27:47 smithi172 systemd[1]: Started Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:27:48.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[29001]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:48.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:48.255 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[39074]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:48.255 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.255 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.255 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.255 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.255 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.255 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:48 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:48.258 INFO:teuthology.orchestra.run.smithi154.stdout:107374182437 2024-09-13T14:27:48.329 INFO:teuthology.orchestra.run.smithi154.stdout:163208757275 2024-09-13T14:27:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:48 smithi172 ceph-mon[32794]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:48 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:48 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:48 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:48 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:48 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:48 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:27:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:49 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2461354923' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:27:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:49 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1744314295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:27:49.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:49 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2461354923' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:27:49.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:49 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1744314295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:27:49.476 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:49 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2461354923' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:27:49.476 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:49 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1744314295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:27:50.108 INFO:teuthology.orchestra.run.smithi154.stdout:34359738420 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:50 smithi154 ceph-mon[29001]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:50 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2655114201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:50 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/342299094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:50 smithi154 ceph-mon[39074]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:50 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2655114201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:27:50.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:50 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/342299094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:27:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:50 smithi172 ceph-mon[32794]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:50 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2655114201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:27:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:50 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/342299094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:27:50.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757275 for osd.5 2024-09-13T14:27:50.623 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:51.582 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182437 for osd.3 2024-09-13T14:27:51.582 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:51.877 INFO:teuthology.orchestra.run.smithi154.stdout:128849018914 2024-09-13T14:27:52.437 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018911 got 128849018914 for osd.4 2024-09-13T14:27:52.437 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[29001]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[39074]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:52 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:52 smithi172 ceph-mon[32794]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:52 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:52 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:52 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:27:52.766 INFO:teuthology.orchestra.run.smithi154.stdout:197568495639 2024-09-13T14:27:53.287 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:53 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3716557020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:27:53.287 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:53 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3716557020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:27:53.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:53 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3716557020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:27:53.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495639 for osd.6 2024-09-13T14:27:53.837 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:54.187 INFO:teuthology.orchestra.run.smithi154.stdout:55834574896 2024-09-13T14:27:54.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:54 smithi154 ceph-mon[29001]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:54.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:54 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1991234907' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:27:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:54 smithi154 ceph-mon[39074]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:54 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1991234907' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:27:54.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:54 smithi172 ceph-mon[32794]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:54.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:54 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1991234907' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:27:54.494 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:27:54.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738420 for osd.0 2024-09-13T14:27:54.515 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:55.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-09-13T14:27:55.149 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:56 smithi154 ceph-mon[29001]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:56.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:56 smithi154 ceph-mon[39074]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:56.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:56 smithi172 ceph-mon[32794]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:57.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:57.349 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:57.474 INFO:teuthology.orchestra.run.smithi154.stdout:231928234004 2024-09-13T14:27:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:27:57.675 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:27:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:27:57] "GET /metrics HTTP/1.1" 200 28513 "" "Prometheus/2.43.0" 2024-09-13T14:27:58.422 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:58 smithi154 ceph-mon[39074]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:58.422 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:27:58 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2533432189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:27:58.422 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:58 smithi154 ceph-mon[29001]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:58.422 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:27:58 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2533432189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:27:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:58 smithi172 ceph-mon[32794]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:27:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:27:58 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2533432189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:27:58.721 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234000 got 231928234004 for osd.7 2024-09-13T14:27:58.722 DEBUG:teuthology.parallel:result is None 2024-09-13T14:27:58.722 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:27:58.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:27:59.440 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[29001]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[39074]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:00 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:00 smithi172 ceph-mon[32794]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:00 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:00 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:00 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:00 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:00 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:01.618 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:01 smithi154 ceph-mon[29001]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:28:01.618 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:01 smithi154 ceph-mon[29001]: Reconfiguring daemon alertmanager.a on smithi154 2024-09-13T14:28:01.619 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:01 smithi154 ceph-mon[39074]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:28:01.619 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:01 smithi154 ceph-mon[39074]: Reconfiguring daemon alertmanager.a on smithi154 2024-09-13T14:28:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:01 smithi172 ceph-mon[32794]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-13T14:28:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:01 smithi172 ceph-mon[32794]: Reconfiguring daemon alertmanager.a on smithi154 2024-09-13T14:28:01.871 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:28:01.871 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:28:02.448 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:02 smithi154 ceph-mon[29001]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:02.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:02 smithi154 ceph-mon[39074]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:02.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:02 smithi172 ceph-mon[32794]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:03.355 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:03 smithi154 ceph-mon[29001]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:28:03.356 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:03 smithi154 ceph-mon[39074]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:28:03.378 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-13T14:28:01.141120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":227240,"kb_used_data":6984,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749733976,"statfs":{"total":767960285184,"available":767727591424,"internally_reserved":0,"allocated":7151616,"data_stored":4034697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001644"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.497129+0000","last_change":"2024-09-13T14:27:05.656833+0000","last_active":"2024-09-13T14:27:14.497129+0000","last_peered":"2024-09-13T14:27:14.497129+0000","last_clean":"2024-09-13T14:27:14.497129+0000","last_became_active":"2024-09-13T14:27:01.156746+0000","last_became_peered":"2024-09-13T14:27:01.156746+0000","last_unstale":"2024-09-13T14:27:14.497129+0000","last_undegraded":"2024-09-13T14:27:14.497129+0000","last_fullsized":"2024-09-13T14:27:14.497129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:47.883758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031071700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.129303+0000","last_change":"2024-09-13T14:27:05.652142+0000","last_active":"2024-09-13T14:27:14.129303+0000","last_peered":"2024-09-13T14:27:14.129303+0000","last_clean":"2024-09-13T14:27:14.129303+0000","last_became_active":"2024-09-13T14:27:01.356003+0000","last_became_peered":"2024-09-13T14:27:01.356003+0000","last_unstale":"2024-09-13T14:27:14.129303+0000","last_undegraded":"2024-09-13T14:27:14.129303+0000","last_fullsized":"2024-09-13T14:27:14.129303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:10.323111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029468799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.265373+0000","last_change":"2024-09-13T14:27:05.696490+0000","last_active":"2024-09-13T14:27:14.265373+0000","last_peered":"2024-09-13T14:27:14.265373+0000","last_clean":"2024-09-13T14:27:14.265373+0000","last_became_active":"2024-09-13T14:27:01.355887+0000","last_became_peered":"2024-09-13T14:27:01.355887+0000","last_unstale":"2024-09-13T14:27:14.265373+0000","last_undegraded":"2024-09-13T14:27:14.265373+0000","last_fullsized":"2024-09-13T14:27:14.265373+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:38.544216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028694700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.129300+0000","last_change":"2024-09-13T14:27:05.652163+0000","last_active":"2024-09-13T14:27:14.129300+0000","last_peered":"2024-09-13T14:27:14.129300+0000","last_clean":"2024-09-13T14:27:14.129300+0000","last_became_active":"2024-09-13T14:27:01.339091+0000","last_became_peered":"2024-09-13T14:27:01.339091+0000","last_unstale":"2024-09-13T14:27:14.129300+0000","last_undegraded":"2024-09-13T14:27:14.129300+0000","last_fullsized":"2024-09-13T14:27:14.129300+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:42.796882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031603499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.497081+0000","last_change":"2024-09-13T14:27:05.655902+0000","last_active":"2024-09-13T14:27:14.497081+0000","last_peered":"2024-09-13T14:27:14.497081+0000","last_clean":"2024-09-13T14:27:14.497081+0000","last_became_active":"2024-09-13T14:27:01.154210+0000","last_became_peered":"2024-09-13T14:27:01.154210+0000","last_unstale":"2024-09-13T14:27:14.497081+0000","last_undegraded":"2024-09-13T14:27:14.497081+0000","last_fullsized":"2024-09-13T14:27:14.497081+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:12.974096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000960205,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.496692+0000","last_change":"2024-09-13T14:27:05.696202+0000","last_active":"2024-09-13T14:27:14.496692+0000","last_peered":"2024-09-13T14:27:14.496692+0000","last_clean":"2024-09-13T14:27:14.496692+0000","last_became_active":"2024-09-13T14:27:01.339177+0000","last_became_peered":"2024-09-13T14:27:01.339177+0000","last_unstale":"2024-09-13T14:27:14.496692+0000","last_undegraded":"2024-09-13T14:27:14.496692+0000","last_fullsized":"2024-09-13T14:27:14.496692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:01.237560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026063399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.265359+0000","last_change":"2024-09-13T14:27:05.696440+0000","last_active":"2024-09-13T14:27:14.265359+0000","last_peered":"2024-09-13T14:27:14.265359+0000","last_clean":"2024-09-13T14:27:14.265359+0000","last_became_active":"2024-09-13T14:27:01.339069+0000","last_became_peered":"2024-09-13T14:27:01.339069+0000","last_unstale":"2024-09-13T14:27:14.265359+0000","last_undegraded":"2024-09-13T14:27:14.265359+0000","last_fullsized":"2024-09-13T14:27:14.265359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:24.663185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025738200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.497065+0000","last_change":"2024-09-13T14:27:05.655358+0000","last_active":"2024-09-13T14:27:14.497065+0000","last_peered":"2024-09-13T14:27:14.497065+0000","last_clean":"2024-09-13T14:27:14.497065+0000","last_became_active":"2024-09-13T14:27:01.151577+0000","last_became_peered":"2024-09-13T14:27:01.151577+0000","last_unstale":"2024-09-13T14:27:14.497065+0000","last_undegraded":"2024-09-13T14:27:14.497065+0000","last_fullsized":"2024-09-13T14:27:14.497065+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:07.643386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000399641,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:27:05.696184+0000","last_change":"2024-09-13T14:26:30.129218+0000","last_active":"2024-09-13T14:27:05.696184+0000","last_peered":"2024-09-13T14:27:05.696184+0000","last_clean":"2024-09-13T14:27:05.696184+0000","last_became_active":"2024-09-13T14:26:30.128826+0000","last_became_peered":"2024-09-13T14:26:30.128826+0000","last_unstale":"2024-09-13T14:27:05.696184+0000","last_undegraded":"2024-09-13T14:27:05.696184+0000","last_fullsized":"2024-09-13T14:27:05.696184+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:31.240178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716900,"statfs":{"total":95995035648,"available":95966105600,"internally_reserved":0,"allocated":1261568,"data_stored":873130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.65900000000000003}]},{"osd":1,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.70199999999999996}]},{"osd":2,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.71599999999999997}]},{"osd":3,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.57399999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.746},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.63400000000000001}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716900,"statfs":{"total":95995035648,"available":95966105600,"internally_reserved":0,"allocated":1261568,"data_stored":872876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.71499999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.61099999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.68100000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":282839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.49099999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.34599999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.625}]},{"osd":3,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40699999999999997,"15min":0.39600000000000002},"min":{"1min":0.318,"5min":0.308,"15min":0.308},"max":{"1min":0.68400000000000005,"5min":0.87,"15min":0.87},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.41699999999999998,"15min":0.40300000000000002},"min":{"1min":0.35599999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.42199999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38900000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.40899999999999997}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.317,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66600000000000004,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.58499999999999996}]},{"osd":1,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68899999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.5},"min":{"1min":0.33900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60499999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.41999999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68400000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45900000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:27:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39400000000000002,"15min":0.39000000000000001},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.442},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.41899999999999998},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.47399999999999998}]},{"osd":7,"last update":"Fri Sep 13 14:27:30 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.60399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.374,"15min":0.38700000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.51100000000000001,"5min":0.621,"15min":0.621},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39400000000000002,"15min":0.40600000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.28199999999999997}]},{"osd":1,"last update":"Fri Sep 13 14:27:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41299999999999998,"15min":0.42599999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.53600000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.41599999999999998,"15min":0.436},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.52100000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.373}]},{"osd":2,"last update":"Fri Sep 13 14:27:56 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.55800000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.42699999999999999,"15min":0.44500000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56599999999999995,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.26900000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71499999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.505}]},{"osd":5,"last update":"Fri Sep 13 14:27:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.53400000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.73199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31772,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713380,"statfs":{"total":95995035648,"available":95962501120,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36299999999999999,"15min":0.35299999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46800000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36099999999999999,"15min":0.35199999999999998},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.54600000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.59899999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.378,"15min":0.375},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.503,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.38700000000000001,"15min":0.378},"min":{"1min":0.26600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.51000000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.52800000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.434,"15min":0.439},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.52900000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58799999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.56100000000000005}]},{"osd":4,"last update":"Fri Sep 13 14:27:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53300000000000003,"15min":0.53200000000000003},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79500000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.75600000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:27:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.56499999999999995,"15min":0.57799999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66700000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.85399999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.95299999999999996}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.91300000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":675840,"data_stored":283112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.32000000000000001},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.318},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.35199999999999998,"15min":0.318},"min":{"1min":0.27500000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.39300000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.28199999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52800000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.29399999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:26:58 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41299999999999998,"15min":0.41499999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.375},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.47399999999999998,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.42599999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56699999999999995}]},{"osd":6,"last update":"Fri Sep 13 14:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.499},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48999999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:27:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.441},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.33900000000000002,"15min":0.32000000000000001},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.33300000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.376,"15min":0.35699999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51200000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.38400000000000001,"15min":0.375},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.28999999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48799999999999999,"15min":0.49199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.48399999999999999,"15min":0.49199999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.378}]},{"osd":4,"last update":"Fri Sep 13 14:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.624},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.65000000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72199999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.75700000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.55900000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:27:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69399999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.58399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:28:03.380 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:28:03.658 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:03 smithi172 ceph-mon[32794]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:28:04.317 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:04.438 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:04 smithi154 ceph-mon[29001]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:04.439 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:04 smithi154 ceph-mon[39074]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:04.439 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:04 smithi154 systemd[1]: Stopping Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:28:04.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:04 smithi172 ceph-mon[32794]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:05.444 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[67353]: ts=2024-09-13T14:28:05.109Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:28:05.720 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:05 smithi154 podman[71623]: 2024-09-13 14:28:05.45902783 +0000 UTC m=+0.464383005 container died 25176e6d1892e3caccdd5c3296a6926d95320505c0cb1050015746ad4794d7d9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:06.063 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:05 smithi154 podman[71623]: 2024-09-13 14:28:05.845834445 +0000 UTC m=+0.851189666 container cleanup 25176e6d1892e3caccdd5c3296a6926d95320505c0cb1050015746ad4794d7d9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:06.064 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:05 smithi154 bash[71623]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:28:06.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:06 smithi154 ceph-mon[29001]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:06.357 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:06 smithi154 ceph-mon[39074]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:06.358 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:06 smithi154 podman[71636]: 2024-09-13 14:28:06.063568646 +0000 UTC m=+0.929743977 container remove 25176e6d1892e3caccdd5c3296a6926d95320505c0cb1050015746ad4794d7d9 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:06.358 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:06 smithi154 podman[71636]: 2024-09-13 14:28:06.13077276 +0000 UTC m=+0.996948092 volume remove b7a6036671c4d5124f53abd571d0b92409444cfeb2c6505cb66d87d723283c43 2024-09-13T14:28:06.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:06 smithi172 ceph-mon[32794]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:06.744 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:28:06.744 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:28:07.347 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:07 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:28:07.347 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:07 smithi154 systemd[1]: Stopped Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:28:07.347 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:07 smithi154 systemd[1]: Starting Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:28:07.401 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-13T14:28:05.141604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":227240,"kb_used_data":6984,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749733976,"statfs":{"total":767960285184,"available":767727591424,"internally_reserved":0,"allocated":7151616,"data_stored":4034697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.497129+0000","last_change":"2024-09-13T14:27:05.656833+0000","last_active":"2024-09-13T14:27:14.497129+0000","last_peered":"2024-09-13T14:27:14.497129+0000","last_clean":"2024-09-13T14:27:14.497129+0000","last_became_active":"2024-09-13T14:27:01.156746+0000","last_became_peered":"2024-09-13T14:27:01.156746+0000","last_unstale":"2024-09-13T14:27:14.497129+0000","last_undegraded":"2024-09-13T14:27:14.497129+0000","last_fullsized":"2024-09-13T14:27:14.497129+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:47.883758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031071700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.129303+0000","last_change":"2024-09-13T14:27:05.652142+0000","last_active":"2024-09-13T14:27:14.129303+0000","last_peered":"2024-09-13T14:27:14.129303+0000","last_clean":"2024-09-13T14:27:14.129303+0000","last_became_active":"2024-09-13T14:27:01.356003+0000","last_became_peered":"2024-09-13T14:27:01.356003+0000","last_unstale":"2024-09-13T14:27:14.129303+0000","last_undegraded":"2024-09-13T14:27:14.129303+0000","last_fullsized":"2024-09-13T14:27:14.129303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:10.323111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029468799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.265373+0000","last_change":"2024-09-13T14:27:05.696490+0000","last_active":"2024-09-13T14:27:14.265373+0000","last_peered":"2024-09-13T14:27:14.265373+0000","last_clean":"2024-09-13T14:27:14.265373+0000","last_became_active":"2024-09-13T14:27:01.355887+0000","last_became_peered":"2024-09-13T14:27:01.355887+0000","last_unstale":"2024-09-13T14:27:14.265373+0000","last_undegraded":"2024-09-13T14:27:14.265373+0000","last_fullsized":"2024-09-13T14:27:14.265373+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:38.544216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028694700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.129300+0000","last_change":"2024-09-13T14:27:05.652163+0000","last_active":"2024-09-13T14:27:14.129300+0000","last_peered":"2024-09-13T14:27:14.129300+0000","last_clean":"2024-09-13T14:27:14.129300+0000","last_became_active":"2024-09-13T14:27:01.339091+0000","last_became_peered":"2024-09-13T14:27:01.339091+0000","last_unstale":"2024-09-13T14:27:14.129300+0000","last_undegraded":"2024-09-13T14:27:14.129300+0000","last_fullsized":"2024-09-13T14:27:14.129300+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:42.796882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031603499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.497081+0000","last_change":"2024-09-13T14:27:05.655902+0000","last_active":"2024-09-13T14:27:14.497081+0000","last_peered":"2024-09-13T14:27:14.497081+0000","last_clean":"2024-09-13T14:27:14.497081+0000","last_became_active":"2024-09-13T14:27:01.154210+0000","last_became_peered":"2024-09-13T14:27:01.154210+0000","last_unstale":"2024-09-13T14:27:14.497081+0000","last_undegraded":"2024-09-13T14:27:14.497081+0000","last_fullsized":"2024-09-13T14:27:14.497081+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:12.974096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000960205,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.496692+0000","last_change":"2024-09-13T14:27:05.696202+0000","last_active":"2024-09-13T14:27:14.496692+0000","last_peered":"2024-09-13T14:27:14.496692+0000","last_clean":"2024-09-13T14:27:14.496692+0000","last_became_active":"2024-09-13T14:27:01.339177+0000","last_became_peered":"2024-09-13T14:27:01.339177+0000","last_unstale":"2024-09-13T14:27:14.496692+0000","last_undegraded":"2024-09-13T14:27:14.496692+0000","last_fullsized":"2024-09-13T14:27:14.496692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:01.237560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026063399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.265359+0000","last_change":"2024-09-13T14:27:05.696440+0000","last_active":"2024-09-13T14:27:14.265359+0000","last_peered":"2024-09-13T14:27:14.265359+0000","last_clean":"2024-09-13T14:27:14.265359+0000","last_became_active":"2024-09-13T14:27:01.339069+0000","last_became_peered":"2024-09-13T14:27:01.339069+0000","last_unstale":"2024-09-13T14:27:14.265359+0000","last_undegraded":"2024-09-13T14:27:14.265359+0000","last_fullsized":"2024-09-13T14:27:14.265359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:24.663185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025738200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-13T14:27:14.497065+0000","last_change":"2024-09-13T14:27:05.655358+0000","last_active":"2024-09-13T14:27:14.497065+0000","last_peered":"2024-09-13T14:27:14.497065+0000","last_clean":"2024-09-13T14:27:14.497065+0000","last_became_active":"2024-09-13T14:27:01.151577+0000","last_became_peered":"2024-09-13T14:27:01.151577+0000","last_unstale":"2024-09-13T14:27:14.497065+0000","last_undegraded":"2024-09-13T14:27:14.497065+0000","last_fullsized":"2024-09-13T14:27:14.497065+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:07.643386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000399641,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":262,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-13T14:27:05.696184+0000","last_change":"2024-09-13T14:26:30.129218+0000","last_active":"2024-09-13T14:27:05.696184+0000","last_peered":"2024-09-13T14:27:05.696184+0000","last_clean":"2024-09-13T14:27:05.696184+0000","last_became_active":"2024-09-13T14:26:30.128826+0000","last_became_peered":"2024-09-13T14:26:30.128826+0000","last_unstale":"2024-09-13T14:27:05.696184+0000","last_undegraded":"2024-09-13T14:27:05.696184+0000","last_fullsized":"2024-09-13T14:27:05.696184+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:31.240178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716900,"statfs":{"total":95995035648,"available":95966105600,"internally_reserved":0,"allocated":1261568,"data_stored":873130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.66100000000000003}]},{"osd":2,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56999999999999995}]},{"osd":3,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.443},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39600000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.52500000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58599999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716900,"statfs":{"total":95995035648,"available":95966105600,"internally_reserved":0,"allocated":1261568,"data_stored":872876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.67000000000000004}]},{"osd":1,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.438},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.58799999999999997}]},{"osd":4,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.65200000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:27:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.72799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":282839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.49099999999999999,"15min":0.48699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.34599999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.623}]},{"osd":2,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51600000000000001,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.51800000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.38,"5min":0.371,"15min":0.371},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40699999999999997,"15min":0.39600000000000002},"min":{"1min":0.318,"5min":0.308,"15min":0.308},"max":{"1min":0.68400000000000005,"5min":0.87,"15min":0.87},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.41699999999999998,"15min":0.40300000000000002},"min":{"1min":0.35599999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.39000000000000001}]},{"osd":6,"last update":"Fri Sep 13 14:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.497},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.37}]},{"osd":7,"last update":"Fri Sep 13 14:27:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.441},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45600000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.317,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66600000000000004,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.29499999999999998}]},{"osd":1,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54800000000000004,"15min":0.55200000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.68899999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.5},"min":{"1min":0.33900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.371}]},{"osd":2,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.5,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60499999999999998,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.41999999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.437}]},{"osd":3,"last update":"Fri Sep 13 14:27:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68400000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.56799999999999995}]},{"osd":5,"last update":"Fri Sep 13 14:27:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.39400000000000002,"15min":0.39000000000000001},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.41899999999999998},"min":{"1min":0.32300000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Sep 13 14:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Sep 13 14:27:30 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.60199999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.374,"15min":0.38700000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.51100000000000001,"5min":0.621,"15min":0.621},"last":0.216},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39400000000000002,"15min":0.40600000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:27:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41299999999999998,"15min":0.42599999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.53600000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.41599999999999998,"15min":0.436},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.52100000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.24299999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:27:56 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41999999999999998,"15min":0.432},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.55800000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.318},{"interface":"front","average":{"1min":0.377,"5min":0.42699999999999999,"15min":0.44500000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56599999999999995,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.26800000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82699999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71499999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 14:27:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.502},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.68600000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:27:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.52700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31772,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713380,"statfs":{"total":95995035648,"available":95962501120,"internally_reserved":0,"allocated":671744,"data_stored":283093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36299999999999999,"15min":0.35299999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.46800000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36099999999999999,"15min":0.35199999999999998},"min":{"1min":0.26400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.54600000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:27:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.378,"15min":0.375},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.503,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.374},{"interface":"front","average":{"1min":0.378,"5min":0.38700000000000001,"15min":0.378},"min":{"1min":0.26600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.51000000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.42099999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.434,"15min":0.439},"min":{"1min":0.311,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.52900000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58799999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.45000000000000001}]},{"osd":4,"last update":"Fri Sep 13 14:27:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53300000000000003,"15min":0.53200000000000003},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.55300000000000005,"15min":0.55600000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.79500000000000004,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.60299999999999998}]},{"osd":5,"last update":"Fri Sep 13 14:27:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.56499999999999995,"15min":0.57799999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66700000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:27:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.57199999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":675840,"data_stored":283112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34000000000000002,"15min":0.32000000000000001},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57499999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.372},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.35199999999999998,"15min":0.318},"min":{"1min":0.27500000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.25900000000000001}]},{"osd":2,"last update":"Fri Sep 13 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.28199999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52800000000000002,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.314}]},{"osd":3,"last update":"Fri Sep 13 14:28:02 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40899999999999997,"15min":0.41399999999999998},"min":{"1min":0.28000000000000003,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Sep 13 14:27:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56999999999999995,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45500000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60999999999999999,"15min":0.61899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56200000000000006}]},{"osd":6,"last update":"Fri Sep 13 14:27:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57699999999999996}]},{"osd":7,"last update":"Fri Sep 13 14:27:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.496}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":873461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.33900000000000002,"15min":0.32000000000000001},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.33300000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Sep 13 14:27:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.376,"15min":0.35699999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51200000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.38400000000000001,"15min":0.375},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.28999999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.48799999999999999,"15min":0.49199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.48399999999999999,"15min":0.49199999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.378}]},{"osd":4,"last update":"Fri Sep 13 14:27:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.624},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54800000000000004,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.65000000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:27:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72199999999999998,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.75700000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.55900000000000005}]},{"osd":6,"last update":"Fri Sep 13 14:27:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69399999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:27:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.58399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:28:07.403 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:28:07.403 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:28:07.403 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:28:07.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:28:07.650 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:28:07] "GET /metrics HTTP/1.1" 200 28513 "" "Prometheus/2.43.0" 2024-09-13T14:28:07.936 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:07 smithi154 podman[71818]: 2024-09-13 14:28:07.65033772 +0000 UTC m=+0.104890669 volume create 92e3a3475dda0753403871ec1af27331f63b54232be129ead2b20e23ad273c67 2024-09-13T14:28:07.936 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:07 smithi154 podman[71818]: 2024-09-13 14:28:07.565824504 +0000 UTC m=+0.020377459 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-13T14:28:07.936 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:07 smithi154 podman[71818]: 2024-09-13 14:28:07.742443842 +0000 UTC m=+0.196996794 container create d46f4205cce47ed7bf985c4b109664ed2a65cec53a7c32faee5aa6d08101b7ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:08.251 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:08.372 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[29001]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:28:08.372 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[29001]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:08.372 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:08.372 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:08.372 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[39074]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:28:08.372 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[39074]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:08.372 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:08.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 podman[71818]: 2024-09-13 14:28:08.053839982 +0000 UTC m=+0.508392931 container init d46f4205cce47ed7bf985c4b109664ed2a65cec53a7c32faee5aa6d08101b7ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 podman[71818]: 2024-09-13 14:28:08.056627934 +0000 UTC m=+0.511180884 container start d46f4205cce47ed7bf985c4b109664ed2a65cec53a7c32faee5aa6d08101b7ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.068Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.068Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.070Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.154 port=9094 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.071Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.093Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.094Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.096Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:28:08.373 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:08.096Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:28:08.374 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 bash[71818]: d46f4205cce47ed7bf985c4b109664ed2a65cec53a7c32faee5aa6d08101b7ee 2024-09-13T14:28:08.374 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:08 smithi154 systemd[1]: Started Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:28:08.729 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:08 smithi172 ceph-mon[32794]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:28:08.729 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:08 smithi172 ceph-mon[32794]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:08.729 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:08 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:08.729 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:08 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:09 smithi172 ceph-mon[32794]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:28:09.547 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:09 smithi154 ceph-mon[29001]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:28:09.548 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:09 smithi154 ceph-mon[39074]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:28:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:28:09.831 INFO:teuthology.orchestra.run.smithi154.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:28:10.142 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:10 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:10.071Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000294011s 2024-09-13T14:28:10.381 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:10 smithi172 ceph-mon[32794]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:28:10.381 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:10 smithi172 ceph-mon[32794]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:10.381 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:10 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2757714857' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:28:10.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:10 smithi154 ceph-mon[29001]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:28:10.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:10 smithi154 ceph-mon[29001]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:10.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:10 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2757714857' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:28:10.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:10 smithi154 ceph-mon[39074]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:28:10.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:10 smithi154 ceph-mon[39074]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:10.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:10 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2757714857' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:28:10.399 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:28:10.399 INFO:tasks.cephadm:Setup complete, yielding 2024-09-13T14:28:10.399 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-13T14:28:10.411 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T14:28:10.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-09-13T14:28:10.613 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:11.491 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 systemd[1]: Stopping Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:28:11.871 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:28:11.872 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:28:11.872 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:28:11.872 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:28:11.872 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:28:11.872 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:28:11.873 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:28:11.873 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:28:11.873 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:28:11.873 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:28:11.873 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:28:11.873 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[60054]: ts=2024-09-13T14:28:11.604Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:28:11.874 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 podman[63126]: 2024-09-13 14:28:11.614355165 +0000 UTC m=+0.121965502 container died 94f1dc5e39f47951047dff067640ea9a44c1fa003352a0a4d702b6d0ca9a5b48 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:12.240 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 podman[63126]: 2024-09-13 14:28:11.978817282 +0000 UTC m=+0.486427616 container cleanup 94f1dc5e39f47951047dff067640ea9a44c1fa003352a0a4d702b6d0ca9a5b48 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:12.240 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:11 smithi172 bash[63126]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:28:12.605 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-mon[32794]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:12.605 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:12.605 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 podman[63139]: 2024-09-13 14:28:12.254989988 +0000 UTC m=+0.642719300 container remove 94f1dc5e39f47951047dff067640ea9a44c1fa003352a0a4d702b6d0ca9a5b48 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:12.605 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:28:12.605 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 systemd[1]: Stopped Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:28:12.605 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.100s CPU time. 2024-09-13T14:28:12.606 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 systemd[1]: Starting Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:28:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:12 smithi154 ceph-mon[29001]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:12 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:12.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:12 smithi154 ceph-mon[39074]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:12.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:12 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:12.856 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 podman[63229]: 2024-09-13 14:28:12.508192899 +0000 UTC m=+0.016682641 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:28:12.856 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 podman[63229]: 2024-09-13 14:28:12.60575525 +0000 UTC m=+0.114244976 container create 33387ab51e885f6374cc8e6536f44b9c286c92e32af7bed533f9b0ab8dcc6cdc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:12.856 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 podman[63229]: 2024-09-13 14:28:12.799513169 +0000 UTC m=+0.308002906 container init 33387ab51e885f6374cc8e6536f44b9c286c92e32af7bed533f9b0ab8dcc6cdc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 podman[63229]: 2024-09-13 14:28:12.802279889 +0000 UTC m=+0.310769617 container start 33387ab51e885f6374cc8e6536f44b9c286c92e32af7bed533f9b0ab8dcc6cdc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.825Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.825Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.825Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi172 (none))" 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.825Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.826Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.828Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.828Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.829Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.829Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.830Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:28:12.857 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.830Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.083µs 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.830Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.833Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.833Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.833Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.145µs wal_replay_duration=2.738431ms wbl_replay_duration=125ns total_replay_duration=2.794187ms 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.834Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.834Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.834Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.855Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.440879ms db_storage=625ns remote_storage=1.053µs web_handler=420ns query_engine=548ns scrape=385.891µs scrape_sd=89.592µs notify=10.143µs notify_sd=10.982µs rules=20.646455ms tracing=5.498µs 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.855Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:28:12.858 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:28:12.855Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:28:13.218 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:12] ENGINE Bus STOPPING 2024-09-13T14:28:13.218 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:28:13.218 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STOPPED 2024-09-13T14:28:13.218 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STARTING 2024-09-13T14:28:13.219 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Serving on http://:::9283 2024-09-13T14:28:13.240 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 bash[63229]: 33387ab51e885f6374cc8e6536f44b9c286c92e32af7bed533f9b0ab8dcc6cdc 2024-09-13T14:28:13.240 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:28:12 smithi172 systemd[1]: Started Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:28:13.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STARTED 2024-09-13T14:28:13.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STOPPING 2024-09-13T14:28:13.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:28:13.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:28:13.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi154.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:28:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi154.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:13.546 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:28:13.546 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T14:28:13.546 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi154.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:13.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:13 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:13.825 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:28:13.825 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STOPPED 2024-09-13T14:28:13.825 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STARTING 2024-09-13T14:28:13.825 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Serving on http://:::9283 2024-09-13T14:28:13.825 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STARTED 2024-09-13T14:28:14.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:13] ENGINE Bus STOPPING 2024-09-13T14:28:14.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:14.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:28:14.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi154.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:14.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:14 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:14.673 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:14.673 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:28:14.673 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi154.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:14.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi154.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T14:28:14.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:14.676 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:14.676 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[28997]: 2024-09-13T14:28:14.301+0000 7fd44551d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:28:14.676 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:28:14.676 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:14] ENGINE Bus STOPPED 2024-09-13T14:28:14.677 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:14] ENGINE Bus STARTING 2024-09-13T14:28:14.677 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:14] ENGINE Serving on http://:::9283 2024-09-13T14:28:14.677 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:28:14] ENGINE Bus STARTED 2024-09-13T14:28:15.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-13T14:28:15.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: fsmap foo:0 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: Saving service mds.foo spec with placement count:2 2024-09-13T14:28:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:15 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:15.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:28:15.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:28:15.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:15.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:15.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:28:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:28:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:28:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:28:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: fsmap foo:0 2024-09-13T14:28:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: Saving service mds.foo spec with placement count:2 2024-09-13T14:28:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:15.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: fsmap foo:0 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: Saving service mds.foo spec with placement count:2 2024-09-13T14:28:15.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:15 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:16.065 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:16.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[29001]: pgmap v52: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:16.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[29001]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:28:16.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:16.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:16.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[39074]: pgmap v52: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:16.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[39074]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:28:16.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:16.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:16 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:16.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:16 smithi172 ceph-mon[32794]: pgmap v52: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:16.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:16 smithi172 ceph-mon[32794]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T14:28:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:16 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:16 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.449 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:28:17.449 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.449 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.449 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:17.449 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:17.449 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.449 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:17.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:17.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:28:17.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:17 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:17.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:28:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:17 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:18.326 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:28:18 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:28:18.073Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001562332s 2024-09-13T14:28:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:18 smithi154 ceph-mon[29001]: Deploying daemon mds.foo.smithi154.daekzd on smithi154 2024-09-13T14:28:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:18 smithi154 ceph-mon[29001]: pgmap v55: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:18 smithi154 ceph-mon[39074]: Deploying daemon mds.foo.smithi154.daekzd on smithi154 2024-09-13T14:28:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:18 smithi154 ceph-mon[39074]: pgmap v55: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:18 smithi172 ceph-mon[32794]: Deploying daemon mds.foo.smithi154.daekzd on smithi154 2024-09-13T14:28:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:18 smithi172 ceph-mon[32794]: pgmap v55: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:19.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-13T14:28:19.778 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:20.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:20 smithi154 ceph-mon[29001]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:20.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:20 smithi154 ceph-mon[39074]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:20.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:20 smithi172 ceph-mon[32794]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:22.544 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:22 smithi154 ceph-mon[29001]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:22.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:22 smithi154 ceph-mon[39074]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:22.546 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:28:22.558 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:28:22.558 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:28:22.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:28:22.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:22 smithi172 ceph-mon[32794]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:22.875 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: Deploying daemon mds.foo.smithi172.pmmovd on smithi172 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:24.120 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:boot 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: daemon mds.foo.smithi154.daekzd assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: Cluster is now healthy 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: fsmap foo:0 1 up:standby 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: fsmap foo:1 {0=foo.smithi154.daekzd=up:creating} 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[29001]: daemon mds.foo.smithi154.daekzd is now active in filesystem foo as rank 0 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.121 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: Deploying daemon mds.foo.smithi172.pmmovd on smithi172 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:boot 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: daemon mds.foo.smithi154.daekzd assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: Cluster is now healthy 2024-09-13T14:28:24.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: fsmap foo:0 1 up:standby 2024-09-13T14:28:24.123 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: fsmap foo:1 {0=foo.smithi154.daekzd=up:creating} 2024-09-13T14:28:24.123 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:23 smithi154 ceph-mon[39074]: daemon mds.foo.smithi154.daekzd is now active in filesystem foo as rank 0 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: Deploying daemon mds.foo.smithi172.pmmovd on smithi172 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:boot 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: daemon mds.foo.smithi154.daekzd assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: Cluster is now healthy 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: fsmap foo:0 1 up:standby 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:creating} 2024-09-13T14:28:24.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:23 smithi172 ceph-mon[32794]: daemon mds.foo.smithi154.daekzd is now active in filesystem foo as rank 0 2024-09-13T14:28:24.846 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:28:25.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:active 2024-09-13T14:28:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 2024-09-13T14:28:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2708736391' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:28:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:25 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.529 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:active 2024-09-13T14:28:25.529 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 2024-09-13T14:28:25.529 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2708736391' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:28:25.529 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.529 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.529 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.529 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:active 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2708736391' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.530 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:25 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:25.551 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":20,"flags":0,"active_gid":24329,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":302860313},{"type":"v1","addr":"172.21.15.154:6801","nonce":302860313}]},"active_addr":"172.21.15.154:6801/302860313","active_change":"2024-09-13T14:26:57.118185+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24337,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":1379294727}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":3255496958}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":3924873105}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":18164567}]}]} 2024-09-13T14:28:25.555 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:28:25.555 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:28:25.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:28:26.301 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:26.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[29001]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:26.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[29001]: mds.? [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] up:boot 2024-09-13T14:28:26.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[29001]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[29001]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[39074]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[39074]: mds.? [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] up:boot 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[39074]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[39074]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:26.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:26 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:26 smithi172 ceph-mon[32794]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:26 smithi172 ceph-mon[32794]: mds.? [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] up:boot 2024-09-13T14:28:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:26 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:28:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:26 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:26 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:26 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:27.632 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:27 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:27.632 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:27 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:27.632 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:27 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:27.632 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:27 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:27.633 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:28:27] "GET /metrics HTTP/1.1" 200 33676 "" "Prometheus/2.43.0" 2024-09-13T14:28:27.724 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:27 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:27.724 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:27 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:28.060 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:28:28.061 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":71,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T14:28:16.326501+0000","last_up_change":"2024-09-13T14:26:26.512111+0000","last_in_change":"2024-09-13T14:26:09.317846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:28:12.645576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:28:13.527124+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6803","nonce":1775775819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6805","nonce":1775775819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6809","nonce":1775775819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6807","nonce":1775775819}]},"public_addr":"172.21.15.154:6803/1775775819","cluster_addr":"172.21.15.154:6805/1775775819","heartbeat_back_addr":"172.21.15.154:6809/1775775819","heartbeat_front_addr":"172.21.15.154:6807/1775775819","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6811","nonce":3038007336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6813","nonce":3038007336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6817","nonce":3038007336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6815","nonce":3038007336}]},"public_addr":"172.21.15.154:6811/3038007336","cluster_addr":"172.21.15.154:6813/3038007336","heartbeat_back_addr":"172.21.15.154:6817/3038007336","heartbeat_front_addr":"172.21.15.154:6815/3038007336","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6819","nonce":1952703304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6821","nonce":1952703304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6825","nonce":1952703304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6823","nonce":1952703304}]},"public_addr":"172.21.15.154:6819/1952703304","cluster_addr":"172.21.15.154:6821/1952703304","heartbeat_back_addr":"172.21.15.154:6825/1952703304","heartbeat_front_addr":"172.21.15.154:6823/1952703304","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6827","nonce":1896570703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6829","nonce":1896570703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6833","nonce":1896570703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6831","nonce":1896570703}]},"public_addr":"172.21.15.154:6827/1896570703","cluster_addr":"172.21.15.154:6829/1896570703","heartbeat_back_addr":"172.21.15.154:6833/1896570703","heartbeat_front_addr":"172.21.15.154:6831/1896570703","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:28:28.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:28 smithi154 ceph-mon[29001]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:28.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:28 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/630775617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:28:28.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:28 smithi154 ceph-mon[39074]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:28.658 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:28 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/630775617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:28:28.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:28 smithi172 ceph-mon[32794]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:28.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:28 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/630775617' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:28:29.610 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:28:29.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:28:29.632 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[29001]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:active 2024-09-13T14:28:29.632 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[29001]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[29001]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[39074]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:active 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[39074]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:29.633 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:29 smithi154 ceph-mon[39074]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:29.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:29 smithi172 ceph-mon[32794]: mds.? [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] up:active 2024-09-13T14:28:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:29 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:29 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:29 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:29 smithi172 ceph-mon[32794]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:30.045 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: mds.? [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] up:standby 2024-09-13T14:28:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:30.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:30 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:31.157 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: mds.? [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] up:standby 2024-09-13T14:28:31.157 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: mds.? [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] up:standby 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:31.158 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:31.159 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:31.159 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:31.159 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:31.159 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:30 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:28:31.783 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:31 smithi172 ceph-mon[32794]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:31.845 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:28:31.845 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":71,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T14:28:16.326501+0000","last_up_change":"2024-09-13T14:26:26.512111+0000","last_in_change":"2024-09-13T14:26:09.317846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:28:12.645576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:28:13.527124+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6803","nonce":1775775819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6805","nonce":1775775819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6809","nonce":1775775819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1775775819},{"type":"v1","addr":"172.21.15.154:6807","nonce":1775775819}]},"public_addr":"172.21.15.154:6803/1775775819","cluster_addr":"172.21.15.154:6805/1775775819","heartbeat_back_addr":"172.21.15.154:6809/1775775819","heartbeat_front_addr":"172.21.15.154:6807/1775775819","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6811","nonce":3038007336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6813","nonce":3038007336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6817","nonce":3038007336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3038007336},{"type":"v1","addr":"172.21.15.154:6815","nonce":3038007336}]},"public_addr":"172.21.15.154:6811/3038007336","cluster_addr":"172.21.15.154:6813/3038007336","heartbeat_back_addr":"172.21.15.154:6817/3038007336","heartbeat_front_addr":"172.21.15.154:6815/3038007336","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6819","nonce":1952703304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6821","nonce":1952703304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6825","nonce":1952703304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":1952703304},{"type":"v1","addr":"172.21.15.154:6823","nonce":1952703304}]},"public_addr":"172.21.15.154:6819/1952703304","cluster_addr":"172.21.15.154:6821/1952703304","heartbeat_back_addr":"172.21.15.154:6825/1952703304","heartbeat_front_addr":"172.21.15.154:6823/1952703304","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6827","nonce":1896570703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6829","nonce":1896570703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6833","nonce":1896570703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1896570703},{"type":"v1","addr":"172.21.15.154:6831","nonce":1896570703}]},"public_addr":"172.21.15.154:6827/1896570703","cluster_addr":"172.21.15.154:6829/1896570703","heartbeat_back_addr":"172.21.15.154:6833/1896570703","heartbeat_front_addr":"172.21.15.154:6831/1896570703","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:28:32.034 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:31 smithi154 ceph-mon[29001]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:32.035 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:31 smithi154 ceph-mon[39074]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:32 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2553047199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:28:32.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:32 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2553047199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:28:33.028 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:32 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2553047199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T14:28:33.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:28:33.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:28:33.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:28:33.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:28:33.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:28:33.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:28:33.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:28:33.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:28:33.557 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:33.558 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:33.561 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:33.563 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:34.079 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:33 smithi154 ceph-mon[29001]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:34.079 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:33 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:34.080 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:33 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:34.080 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:33 smithi154 ceph-mon[39074]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:34.080 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:33 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:34.080 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:33 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:34.090 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:33 smithi172 ceph-mon[32794]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:33 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:33 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:36.434 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:36 smithi154 ceph-mon[29001]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:36.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:36 smithi154 ceph-mon[39074]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:36.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:36 smithi172 ceph-mon[32794]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:37.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:37 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:28:37] "GET /metrics HTTP/1.1" 200 33676 "" "Prometheus/2.43.0" 2024-09-13T14:28:38.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:38 smithi154 ceph-mon[29001]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:38 smithi154 ceph-mon[39074]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:38.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:38 smithi172 ceph-mon[32794]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:40 smithi154 ceph-mon[29001]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:40.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:40 smithi154 ceph-mon[39074]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:40 smithi172 ceph-mon[32794]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-09-13T14:28:40.594 INFO:teuthology.orchestra.run.smithi154.stdout:34359738431 2024-09-13T14:28:40.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:28:41.687 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:42 smithi154 ceph-mon[29001]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:42 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:42 smithi154 ceph-mon[39074]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:42 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:42 smithi172 ceph-mon[32794]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:42 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:42.560 INFO:teuthology.orchestra.run.smithi154.stdout:55834574906 2024-09-13T14:28:42.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:28:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:44 smithi154 ceph-mon[29001]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:44.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:44 smithi154 ceph-mon[39074]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:44.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:44 smithi172 ceph-mon[32794]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:44.785 INFO:teuthology.orchestra.run.smithi154.stdout:128849018925 2024-09-13T14:28:44.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:28:45.653 INFO:teuthology.orchestra.run.smithi154.stdout:163208757288 2024-09-13T14:28:45.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:28:46.430 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[29001]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[39074]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:46.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:46 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:46 smithi172 ceph-mon[32794]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:46 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:46 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:46 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:28:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:46 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:28:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:46 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:28:46.759 INFO:teuthology.orchestra.run.smithi154.stdout:77309411383 2024-09-13T14:28:46.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:28:47.562 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:47 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:28:47] "GET /metrics HTTP/1.1" 200 33712 "" "Prometheus/2.43.0" 2024-09-13T14:28:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:48 smithi154 ceph-mon[29001]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:48.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:48 smithi154 ceph-mon[39074]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:48.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:48 smithi172 ceph-mon[32794]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:48.860 INFO:teuthology.orchestra.run.smithi154.stdout:107374182450 2024-09-13T14:28:48.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:28:49.496 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:50 smithi154 ceph-mon[29001]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:50 smithi154 ceph-mon[39074]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:50 smithi172 ceph-mon[32794]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:50.908 INFO:teuthology.orchestra.run.smithi154.stdout:231928234015 2024-09-13T14:28:50.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:28:50.928 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:50.928 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:50.934 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:50.934 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:51.718 INFO:teuthology.orchestra.run.smithi154.stdout:197568495652 2024-09-13T14:28:51.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:28:52.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:52 smithi154 ceph-mon[29001]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:52.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:52 smithi154 ceph-mon[39074]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:52 smithi172 ceph-mon[32794]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:52.931 INFO:teuthology.orchestra.run.smithi154.stdout:34359738433 2024-09-13T14:28:53.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:53 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3198573112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:28:53.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:53 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3198573112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:28:53.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:53 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3198573112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T14:28:54.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:54 smithi154 ceph-mon[29001]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:54 smithi154 ceph-mon[39074]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:54.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:54 smithi172 ceph-mon[32794]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:55.879 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738431 got 34359738433 for osd.0 2024-09-13T14:28:55.880 DEBUG:teuthology.parallel:result is None 2024-09-13T14:28:56.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:56 smithi172 ceph-mon[32794]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:56.502 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:56 smithi154 ceph-mon[29001]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:56.502 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:56 smithi154 ceph-mon[39074]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:56.801 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:56.839 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:28:57.349 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:28:57.351 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.351 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:28:57.351 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:57.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:28:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:28:57] "GET /metrics HTTP/1.1" 200 33714 "" "Prometheus/2.43.0" 2024-09-13T14:28:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:57 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:28:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:57 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-09-13T14:28:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:57 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2024-09-13T14:28:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:28:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[39074]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T14:28:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:28:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-13T14:28:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[39074]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:28:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[29001]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T14:28:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:28:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-13T14:28:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:58 smithi154 ceph-mon[29001]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:28:58.538 INFO:teuthology.orchestra.run.smithi154.stdout:55834574909 2024-09-13T14:28:58.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:58 smithi172 ceph-mon[32794]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:28:58.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:58 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-13T14:28:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:58 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-09-13T14:28:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:58 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-13T14:28:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:58 smithi172 ceph-mon[32794]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T14:28:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:59 smithi154 ceph-mon[29001]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:28:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:28:59 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3396092326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:28:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:59 smithi154 ceph-mon[39074]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:28:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:28:59 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3396092326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:28:59.503 INFO:teuthology.orchestra.run.smithi154.stdout:77309411386 2024-09-13T14:28:59.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:59 smithi172 ceph-mon[32794]: osdmap e73: 8 total, 8 up, 8 in 2024-09-13T14:28:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:28:59 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3396092326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T14:29:00.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411383 got 77309411386 for osd.2 2024-09-13T14:29:00.272 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:00 smithi154 ceph-mon[29001]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:00 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3499559864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:29:00.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:00 smithi154 ceph-mon[39074]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:00.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:00 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3499559864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:29:00.633 INFO:teuthology.orchestra.run.smithi154.stdout:107374182453 2024-09-13T14:29:00.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:00 smithi172 ceph-mon[32794]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:00.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:00 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3499559864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T14:29:01.208 INFO:teuthology.orchestra.run.smithi154.stdout:163208757291 2024-09-13T14:29:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:01 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2655993734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:29:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:01 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/530588645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:29:01.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:01 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2655993734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:29:01.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:01 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/530588645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:29:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:01 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2655993734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T14:29:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:01 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/530588645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T14:29:01.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757288 got 163208757291 for osd.5 2024-09-13T14:29:01.877 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:02.443 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:02 smithi154 ceph-mon[29001]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:02.443 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:02 smithi154 ceph-mon[39074]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:02.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:02 smithi172 ceph-mon[32794]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:02.947 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:02.964 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574906 got 55834574909 for osd.1 2024-09-13T14:29:02.964 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:03.424 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182450 got 107374182453 for osd.3 2024-09-13T14:29:03.424 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:04.059 INFO:teuthology.orchestra.run.smithi154.stdout:128849018930 2024-09-13T14:29:04.598 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:04 smithi154 ceph-mon[29001]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:04.598 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:04 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/169801445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:29:04.598 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:04 smithi154 ceph-mon[39074]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:04.598 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:04 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/169801445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:29:04.715 INFO:teuthology.orchestra.run.smithi154.stdout:231928234019 2024-09-13T14:29:04.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:04 smithi172 ceph-mon[32794]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:04.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:04 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/169801445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T14:29:04.990 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018925 got 128849018930 for osd.4 2024-09-13T14:29:04.990 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:05.615 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:05 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3891959427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:29:05.615 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:05 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3891959427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:29:05.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234015 got 231928234019 for osd.7 2024-09-13T14:29:05.635 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:05.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:05 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3891959427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T14:29:06.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:06 smithi154 ceph-mon[29001]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:06.647 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:06 smithi154 ceph-mon[39074]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:06.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:06 smithi172 ceph-mon[32794]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:07.316 INFO:teuthology.orchestra.run.smithi154.stdout:197568495655 2024-09-13T14:29:07.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:29:07] "GET /metrics HTTP/1.1" 200 33714 "" "Prometheus/2.43.0" 2024-09-13T14:29:07.986 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495652 got 197568495655 for osd.6 2024-09-13T14:29:07.986 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:07.986 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:29:07.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:29:08.190 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:08.532 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:08 smithi154 ceph-mon[29001]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:08.533 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:08 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/4127422665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:29:08.533 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:08 smithi154 ceph-mon[39074]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:08.533 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:08 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/4127422665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:29:08.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:08 smithi172 ceph-mon[32794]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:08.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:08 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/4127422665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T14:29:09.932 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:29:09.934 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:29:10.570 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:10 smithi154 ceph-mon[29001]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:10.571 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:10 smithi154 ceph-mon[39074]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:10.592 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-13T14:29:09.154371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232104,"kb_used_data":7752,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729112,"statfs":{"total":767960285184,"available":767722610688,"internally_reserved":0,"allocated":7938048,"data_stored":4445807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.907072"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244216+0000","last_change":"2024-09-13T14:28:14.409388+0000","last_active":"2024-09-13T14:28:57.244216+0000","last_peered":"2024-09-13T14:28:57.244216+0000","last_clean":"2024-09-13T14:28:57.244216+0000","last_became_active":"2024-09-13T14:28:14.409134+0000","last_became_peered":"2024-09-13T14:28:14.409134+0000","last_unstale":"2024-09-13T14:28:57.244216+0000","last_undegraded":"2024-09-13T14:28:57.244216+0000","last_fullsized":"2024-09-13T14:28:57.244216+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:03.857964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426174+0000","last_change":"2024-09-13T14:28:14.300970+0000","last_active":"2024-09-13T14:28:57.426174+0000","last_peered":"2024-09-13T14:28:57.426174+0000","last_clean":"2024-09-13T14:28:57.426174+0000","last_became_active":"2024-09-13T14:28:14.300651+0000","last_became_peered":"2024-09-13T14:28:14.300651+0000","last_unstale":"2024-09-13T14:28:57.426174+0000","last_undegraded":"2024-09-13T14:28:57.426174+0000","last_fullsized":"2024-09-13T14:28:57.426174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:03.739287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245568+0000","last_change":"2024-09-13T14:28:14.411333+0000","last_active":"2024-09-13T14:28:57.245568+0000","last_peered":"2024-09-13T14:28:57.245568+0000","last_clean":"2024-09-13T14:28:57.245568+0000","last_became_active":"2024-09-13T14:28:14.411035+0000","last_became_peered":"2024-09-13T14:28:14.411035+0000","last_unstale":"2024-09-13T14:28:57.245568+0000","last_undegraded":"2024-09-13T14:28:57.245568+0000","last_fullsized":"2024-09-13T14:28:57.245568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:39.991519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250377+0000","last_change":"2024-09-13T14:28:14.294701+0000","last_active":"2024-09-13T14:28:57.250377+0000","last_peered":"2024-09-13T14:28:57.250377+0000","last_clean":"2024-09-13T14:28:57.250377+0000","last_became_active":"2024-09-13T14:28:14.294413+0000","last_became_peered":"2024-09-13T14:28:14.294413+0000","last_unstale":"2024-09-13T14:28:57.250377+0000","last_undegraded":"2024-09-13T14:28:57.250377+0000","last_fullsized":"2024-09-13T14:28:57.250377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:54.638082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250494+0000","last_change":"2024-09-13T14:28:14.410590+0000","last_active":"2024-09-13T14:28:57.250494+0000","last_peered":"2024-09-13T14:28:57.250494+0000","last_clean":"2024-09-13T14:28:57.250494+0000","last_became_active":"2024-09-13T14:28:14.410276+0000","last_became_peered":"2024-09-13T14:28:14.410276+0000","last_unstale":"2024-09-13T14:28:57.250494+0000","last_undegraded":"2024-09-13T14:28:57.250494+0000","last_fullsized":"2024-09-13T14:28:57.250494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:01.051473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426053+0000","last_change":"2024-09-13T14:28:14.292467+0000","last_active":"2024-09-13T14:28:57.426053+0000","last_peered":"2024-09-13T14:28:57.426053+0000","last_clean":"2024-09-13T14:28:57.426053+0000","last_became_active":"2024-09-13T14:28:14.292208+0000","last_became_peered":"2024-09-13T14:28:14.292208+0000","last_unstale":"2024-09-13T14:28:57.426053+0000","last_undegraded":"2024-09-13T14:28:57.426053+0000","last_fullsized":"2024-09-13T14:28:57.426053+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:51.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244606+0000","last_change":"2024-09-13T14:28:14.409497+0000","last_active":"2024-09-13T14:28:57.244606+0000","last_peered":"2024-09-13T14:28:57.244606+0000","last_clean":"2024-09-13T14:28:57.244606+0000","last_became_active":"2024-09-13T14:28:14.409036+0000","last_became_peered":"2024-09-13T14:28:14.409036+0000","last_unstale":"2024-09-13T14:28:57.244606+0000","last_undegraded":"2024-09-13T14:28:57.244606+0000","last_fullsized":"2024-09-13T14:28:57.244606+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:59.234267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.231775+0000","last_change":"2024-09-13T14:28:14.292189+0000","last_active":"2024-09-13T14:28:58.231775+0000","last_peered":"2024-09-13T14:28:58.231775+0000","last_clean":"2024-09-13T14:28:58.231775+0000","last_became_active":"2024-09-13T14:28:14.291937+0000","last_became_peered":"2024-09-13T14:28:14.291937+0000","last_unstale":"2024-09-13T14:28:58.231775+0000","last_undegraded":"2024-09-13T14:28:58.231775+0000","last_fullsized":"2024-09-13T14:28:58.231775+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:09.136718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426526+0000","last_change":"2024-09-13T14:28:14.296105+0000","last_active":"2024-09-13T14:28:57.426526+0000","last_peered":"2024-09-13T14:28:57.426526+0000","last_clean":"2024-09-13T14:28:57.426526+0000","last_became_active":"2024-09-13T14:28:14.295855+0000","last_became_peered":"2024-09-13T14:28:14.295855+0000","last_unstale":"2024-09-13T14:28:57.426526+0000","last_undegraded":"2024-09-13T14:28:57.426526+0000","last_fullsized":"2024-09-13T14:28:57.426526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:31.446251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245100+0000","last_change":"2024-09-13T14:28:14.409515+0000","last_active":"2024-09-13T14:28:57.245100+0000","last_peered":"2024-09-13T14:28:57.245100+0000","last_clean":"2024-09-13T14:28:57.245100+0000","last_became_active":"2024-09-13T14:28:14.409202+0000","last_became_peered":"2024-09-13T14:28:14.409202+0000","last_unstale":"2024-09-13T14:28:57.245100+0000","last_undegraded":"2024-09-13T14:28:57.245100+0000","last_fullsized":"2024-09-13T14:28:57.245100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:24.530144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426473+0000","last_change":"2024-09-13T14:28:14.296094+0000","last_active":"2024-09-13T14:28:57.426473+0000","last_peered":"2024-09-13T14:28:57.426473+0000","last_clean":"2024-09-13T14:28:57.426473+0000","last_became_active":"2024-09-13T14:28:14.295808+0000","last_became_peered":"2024-09-13T14:28:14.295808+0000","last_unstale":"2024-09-13T14:28:57.426473+0000","last_undegraded":"2024-09-13T14:28:57.426473+0000","last_fullsized":"2024-09-13T14:28:57.426473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:12.440358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.254208+0000","last_change":"2024-09-13T14:28:58.254208+0000","last_active":"2024-09-13T14:28:58.254208+0000","last_peered":"2024-09-13T14:28:58.254208+0000","last_clean":"2024-09-13T14:28:58.254208+0000","last_became_active":"2024-09-13T14:28:58.253926+0000","last_became_peered":"2024-09-13T14:28:58.253926+0000","last_unstale":"2024-09-13T14:28:58.254208+0000","last_undegraded":"2024-09-13T14:28:58.254208+0000","last_fullsized":"2024-09-13T14:28:58.254208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:22.344730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426385+0000","last_change":"2024-09-13T14:28:14.289966+0000","last_active":"2024-09-13T14:28:57.426385+0000","last_peered":"2024-09-13T14:28:57.426385+0000","last_clean":"2024-09-13T14:28:57.426385+0000","last_became_active":"2024-09-13T14:28:14.289682+0000","last_became_peered":"2024-09-13T14:28:14.289682+0000","last_unstale":"2024-09-13T14:28:57.426385+0000","last_undegraded":"2024-09-13T14:28:57.426385+0000","last_fullsized":"2024-09-13T14:28:57.426385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:37.096298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.231774+0000","last_change":"2024-09-13T14:28:14.410365+0000","last_active":"2024-09-13T14:28:58.231774+0000","last_peered":"2024-09-13T14:28:58.231774+0000","last_clean":"2024-09-13T14:28:58.231774+0000","last_became_active":"2024-09-13T14:28:14.410093+0000","last_became_peered":"2024-09-13T14:28:14.410093+0000","last_unstale":"2024-09-13T14:28:58.231774+0000","last_undegraded":"2024-09-13T14:28:58.231774+0000","last_fullsized":"2024-09-13T14:28:58.231774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:40.610581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245500+0000","last_change":"2024-09-13T14:28:14.288606+0000","last_active":"2024-09-13T14:28:57.245500+0000","last_peered":"2024-09-13T14:28:57.245500+0000","last_clean":"2024-09-13T14:28:57.245500+0000","last_became_active":"2024-09-13T14:28:14.288328+0000","last_became_peered":"2024-09-13T14:28:14.288328+0000","last_unstale":"2024-09-13T14:28:57.245500+0000","last_undegraded":"2024-09-13T14:28:57.245500+0000","last_fullsized":"2024-09-13T14:28:57.245500+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:05.628581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250438+0000","last_change":"2024-09-13T14:28:14.327415+0000","last_active":"2024-09-13T14:28:57.250438+0000","last_peered":"2024-09-13T14:28:57.250438+0000","last_clean":"2024-09-13T14:28:57.250438+0000","last_became_active":"2024-09-13T14:28:14.327182+0000","last_became_peered":"2024-09-13T14:28:14.327182+0000","last_unstale":"2024-09-13T14:28:57.250438+0000","last_undegraded":"2024-09-13T14:28:57.250438+0000","last_fullsized":"2024-09-13T14:28:57.250438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.594876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243962+0000","last_change":"2024-09-13T14:28:14.304255+0000","last_active":"2024-09-13T14:28:57.243962+0000","last_peered":"2024-09-13T14:28:57.243962+0000","last_clean":"2024-09-13T14:28:57.243962+0000","last_became_active":"2024-09-13T14:28:14.304033+0000","last_became_peered":"2024-09-13T14:28:14.304033+0000","last_unstale":"2024-09-13T14:28:57.243962+0000","last_undegraded":"2024-09-13T14:28:57.243962+0000","last_fullsized":"2024-09-13T14:28:57.243962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:10.401043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242961+0000","last_change":"2024-09-13T14:28:14.324130+0000","last_active":"2024-09-13T14:28:57.242961+0000","last_peered":"2024-09-13T14:28:57.242961+0000","last_clean":"2024-09-13T14:28:57.242961+0000","last_became_active":"2024-09-13T14:28:14.323864+0000","last_became_peered":"2024-09-13T14:28:14.323864+0000","last_unstale":"2024-09-13T14:28:57.242961+0000","last_undegraded":"2024-09-13T14:28:57.242961+0000","last_fullsized":"2024-09-13T14:28:57.242961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:49:53.539380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.231944+0000","last_change":"2024-09-13T14:28:14.292924+0000","last_active":"2024-09-13T14:28:58.231944+0000","last_peered":"2024-09-13T14:28:58.231944+0000","last_clean":"2024-09-13T14:28:58.231944+0000","last_became_active":"2024-09-13T14:28:14.292637+0000","last_became_peered":"2024-09-13T14:28:14.292637+0000","last_unstale":"2024-09-13T14:28:58.231944+0000","last_undegraded":"2024-09-13T14:28:58.231944+0000","last_fullsized":"2024-09-13T14:28:58.231944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:25.763246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243494+0000","last_change":"2024-09-13T14:28:14.328150+0000","last_active":"2024-09-13T14:28:57.243494+0000","last_peered":"2024-09-13T14:28:57.243494+0000","last_clean":"2024-09-13T14:28:57.243494+0000","last_became_active":"2024-09-13T14:28:14.327825+0000","last_became_peered":"2024-09-13T14:28:14.327825+0000","last_unstale":"2024-09-13T14:28:57.243494+0000","last_undegraded":"2024-09-13T14:28:57.243494+0000","last_fullsized":"2024-09-13T14:28:57.243494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:10.215900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243129+0000","last_change":"2024-09-13T14:28:14.301032+0000","last_active":"2024-09-13T14:28:57.243129+0000","last_peered":"2024-09-13T14:28:57.243129+0000","last_clean":"2024-09-13T14:28:57.243129+0000","last_became_active":"2024-09-13T14:28:14.300722+0000","last_became_peered":"2024-09-13T14:28:14.300722+0000","last_unstale":"2024-09-13T14:28:57.243129+0000","last_undegraded":"2024-09-13T14:28:57.243129+0000","last_fullsized":"2024-09-13T14:28:57.243129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:58.411339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243662+0000","last_change":"2024-09-13T14:28:14.327373+0000","last_active":"2024-09-13T14:28:57.243662+0000","last_peered":"2024-09-13T14:28:57.243662+0000","last_clean":"2024-09-13T14:28:57.243662+0000","last_became_active":"2024-09-13T14:28:14.326930+0000","last_became_peered":"2024-09-13T14:28:14.326930+0000","last_unstale":"2024-09-13T14:28:57.243662+0000","last_undegraded":"2024-09-13T14:28:57.243662+0000","last_fullsized":"2024-09-13T14:28:57.243662+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:21.335368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243726+0000","last_change":"2024-09-13T14:28:14.303769+0000","last_active":"2024-09-13T14:28:57.243726+0000","last_peered":"2024-09-13T14:28:57.243726+0000","last_clean":"2024-09-13T14:28:57.243726+0000","last_became_active":"2024-09-13T14:28:14.302350+0000","last_became_peered":"2024-09-13T14:28:14.302350+0000","last_unstale":"2024-09-13T14:28:57.243726+0000","last_undegraded":"2024-09-13T14:28:57.243726+0000","last_fullsized":"2024-09-13T14:28:57.243726+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:57.905748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.247041+0000","last_change":"2024-09-13T14:28:14.330928+0000","last_active":"2024-09-13T14:28:57.247041+0000","last_peered":"2024-09-13T14:28:57.247041+0000","last_clean":"2024-09-13T14:28:57.247041+0000","last_became_active":"2024-09-13T14:28:14.330643+0000","last_became_peered":"2024-09-13T14:28:14.330643+0000","last_unstale":"2024-09-13T14:28:57.247041+0000","last_undegraded":"2024-09-13T14:28:57.247041+0000","last_fullsized":"2024-09-13T14:28:57.247041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:05.343917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":270,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426247+0000","last_change":"2024-09-13T14:26:30.129218+0000","last_active":"2024-09-13T14:28:57.426247+0000","last_peered":"2024-09-13T14:28:57.426247+0000","last_clean":"2024-09-13T14:28:57.426247+0000","last_became_active":"2024-09-13T14:26:30.128826+0000","last_became_peered":"2024-09-13T14:26:30.128826+0000","last_unstale":"2024-09-13T14:28:57.426247+0000","last_undegraded":"2024-09-13T14:28:57.426247+0000","last_fullsized":"2024-09-13T14:28:57.426247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:31.240178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.692335+0000","last_change":"2024-09-13T14:27:05.655358+0000","last_active":"2024-09-13T14:28:58.692335+0000","last_peered":"2024-09-13T14:28:58.692335+0000","last_clean":"2024-09-13T14:28:58.692335+0000","last_became_active":"2024-09-13T14:27:01.151577+0000","last_became_peered":"2024-09-13T14:27:01.151577+0000","last_unstale":"2024-09-13T14:28:58.692335+0000","last_undegraded":"2024-09-13T14:28:58.692335+0000","last_fullsized":"2024-09-13T14:28:58.692335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:07.643386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000399641,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244639+0000","last_change":"2024-09-13T14:28:14.291654+0000","last_active":"2024-09-13T14:28:57.244639+0000","last_peered":"2024-09-13T14:28:57.244639+0000","last_clean":"2024-09-13T14:28:57.244639+0000","last_became_active":"2024-09-13T14:28:14.291364+0000","last_became_peered":"2024-09-13T14:28:14.291364+0000","last_unstale":"2024-09-13T14:28:57.244639+0000","last_undegraded":"2024-09-13T14:28:57.244639+0000","last_fullsized":"2024-09-13T14:28:57.244639+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:04.098245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.232005+0000","last_change":"2024-09-13T14:28:14.409690+0000","last_active":"2024-09-13T14:28:58.232005+0000","last_peered":"2024-09-13T14:28:58.232005+0000","last_clean":"2024-09-13T14:28:58.232005+0000","last_became_active":"2024-09-13T14:28:14.409379+0000","last_became_peered":"2024-09-13T14:28:14.409379+0000","last_unstale":"2024-09-13T14:28:58.232005+0000","last_undegraded":"2024-09-13T14:28:58.232005+0000","last_fullsized":"2024-09-13T14:28:58.232005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:15.731901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:59.126529+0000","last_change":"2024-09-13T14:27:05.696440+0000","last_active":"2024-09-13T14:28:59.126529+0000","last_peered":"2024-09-13T14:28:59.126529+0000","last_clean":"2024-09-13T14:28:59.126529+0000","last_became_active":"2024-09-13T14:27:01.339069+0000","last_became_peered":"2024-09-13T14:27:01.339069+0000","last_unstale":"2024-09-13T14:28:59.126529+0000","last_undegraded":"2024-09-13T14:28:59.126529+0000","last_fullsized":"2024-09-13T14:28:59.126529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:24.663185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025738200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250217+0000","last_change":"2024-09-13T14:28:14.305482+0000","last_active":"2024-09-13T14:28:57.250217+0000","last_peered":"2024-09-13T14:28:57.250217+0000","last_clean":"2024-09-13T14:28:57.250217+0000","last_became_active":"2024-09-13T14:28:14.305219+0000","last_became_peered":"2024-09-13T14:28:14.305219+0000","last_unstale":"2024-09-13T14:28:57.250217+0000","last_undegraded":"2024-09-13T14:28:57.250217+0000","last_fullsized":"2024-09-13T14:28:57.250217+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:04.168870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250441+0000","last_change":"2024-09-13T14:28:14.327196+0000","last_active":"2024-09-13T14:28:57.250441+0000","last_peered":"2024-09-13T14:28:57.250441+0000","last_clean":"2024-09-13T14:28:57.250441+0000","last_became_active":"2024-09-13T14:28:14.326955+0000","last_became_peered":"2024-09-13T14:28:14.326955+0000","last_unstale":"2024-09-13T14:28:57.250441+0000","last_undegraded":"2024-09-13T14:28:57.250441+0000","last_fullsized":"2024-09-13T14:28:57.250441+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:54:54.085094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.250901+0000","last_change":"2024-09-13T14:27:05.696202+0000","last_active":"2024-09-13T14:28:58.250901+0000","last_peered":"2024-09-13T14:28:58.250901+0000","last_clean":"2024-09-13T14:28:58.250901+0000","last_became_active":"2024-09-13T14:27:01.339177+0000","last_became_peered":"2024-09-13T14:27:01.339177+0000","last_unstale":"2024-09-13T14:28:58.250901+0000","last_undegraded":"2024-09-13T14:28:58.250901+0000","last_fullsized":"2024-09-13T14:28:58.250901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:01.237560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026063399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246151+0000","last_change":"2024-09-13T14:28:14.302972+0000","last_active":"2024-09-13T14:28:57.246151+0000","last_peered":"2024-09-13T14:28:57.246151+0000","last_clean":"2024-09-13T14:28:57.246151+0000","last_became_active":"2024-09-13T14:28:14.302708+0000","last_became_peered":"2024-09-13T14:28:14.302708+0000","last_unstale":"2024-09-13T14:28:57.246151+0000","last_undegraded":"2024-09-13T14:28:57.246151+0000","last_fullsized":"2024-09-13T14:28:57.246151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:15.883333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.710958+0000","last_change":"2024-09-13T14:28:58.710958+0000","last_active":"2024-09-13T14:28:58.710958+0000","last_peered":"2024-09-13T14:28:58.710958+0000","last_clean":"2024-09-13T14:28:58.710958+0000","last_became_active":"2024-09-13T14:28:58.710685+0000","last_became_peered":"2024-09-13T14:28:58.710685+0000","last_unstale":"2024-09-13T14:28:58.710958+0000","last_undegraded":"2024-09-13T14:28:58.710958+0000","last_fullsized":"2024-09-13T14:28:58.710958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:14.735996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.692244+0000","last_change":"2024-09-13T14:27:05.655902+0000","last_active":"2024-09-13T14:28:58.692244+0000","last_peered":"2024-09-13T14:28:58.692244+0000","last_clean":"2024-09-13T14:28:58.692244+0000","last_became_active":"2024-09-13T14:27:01.154210+0000","last_became_peered":"2024-09-13T14:27:01.154210+0000","last_unstale":"2024-09-13T14:28:58.692244+0000","last_undegraded":"2024-09-13T14:28:58.692244+0000","last_fullsized":"2024-09-13T14:28:58.692244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:12.974096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000960205,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243146+0000","last_change":"2024-09-13T14:28:14.297316+0000","last_active":"2024-09-13T14:28:57.243146+0000","last_peered":"2024-09-13T14:28:57.243146+0000","last_clean":"2024-09-13T14:28:57.243146+0000","last_became_active":"2024-09-13T14:28:14.296753+0000","last_became_peered":"2024-09-13T14:28:14.296753+0000","last_unstale":"2024-09-13T14:28:57.243146+0000","last_undegraded":"2024-09-13T14:28:57.243146+0000","last_fullsized":"2024-09-13T14:28:57.243146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:47.774278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.247222+0000","last_change":"2024-09-13T14:28:14.331943+0000","last_active":"2024-09-13T14:28:57.247222+0000","last_peered":"2024-09-13T14:28:57.247222+0000","last_clean":"2024-09-13T14:28:57.247222+0000","last_became_active":"2024-09-13T14:28:14.331593+0000","last_became_peered":"2024-09-13T14:28:14.331593+0000","last_unstale":"2024-09-13T14:28:57.247222+0000","last_undegraded":"2024-09-13T14:28:57.247222+0000","last_fullsized":"2024-09-13T14:28:57.247222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:53.600541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245662+0000","last_change":"2024-09-13T14:28:14.302176+0000","last_active":"2024-09-13T14:28:57.245662+0000","last_peered":"2024-09-13T14:28:57.245662+0000","last_clean":"2024-09-13T14:28:57.245662+0000","last_became_active":"2024-09-13T14:28:14.301965+0000","last_became_peered":"2024-09-13T14:28:14.301965+0000","last_unstale":"2024-09-13T14:28:57.245662+0000","last_undegraded":"2024-09-13T14:28:57.245662+0000","last_fullsized":"2024-09-13T14:28:57.245662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:46.166594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.249432+0000","last_change":"2024-09-13T14:27:05.652163+0000","last_active":"2024-09-13T14:28:58.249432+0000","last_peered":"2024-09-13T14:28:58.249432+0000","last_clean":"2024-09-13T14:28:58.249432+0000","last_became_active":"2024-09-13T14:27:01.339091+0000","last_became_peered":"2024-09-13T14:27:01.339091+0000","last_unstale":"2024-09-13T14:28:58.249432+0000","last_undegraded":"2024-09-13T14:28:58.249432+0000","last_fullsized":"2024-09-13T14:28:58.249432+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:42.796882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031603499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246170+0000","last_change":"2024-09-13T14:28:14.326033+0000","last_active":"2024-09-13T14:28:57.246170+0000","last_peered":"2024-09-13T14:28:57.246170+0000","last_clean":"2024-09-13T14:28:57.246170+0000","last_became_active":"2024-09-13T14:28:14.325773+0000","last_became_peered":"2024-09-13T14:28:14.325773+0000","last_unstale":"2024-09-13T14:28:57.246170+0000","last_undegraded":"2024-09-13T14:28:57.246170+0000","last_fullsized":"2024-09-13T14:28:57.246170+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:40.260729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.247308+0000","last_change":"2024-09-13T14:28:14.304078+0000","last_active":"2024-09-13T14:28:57.247308+0000","last_peered":"2024-09-13T14:28:57.247308+0000","last_clean":"2024-09-13T14:28:57.247308+0000","last_became_active":"2024-09-13T14:28:14.303703+0000","last_became_peered":"2024-09-13T14:28:14.303703+0000","last_unstale":"2024-09-13T14:28:57.247308+0000","last_undegraded":"2024-09-13T14:28:57.247308+0000","last_fullsized":"2024-09-13T14:28:57.247308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:17.144867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:59.126737+0000","last_change":"2024-09-13T14:27:05.696490+0000","last_active":"2024-09-13T14:28:59.126737+0000","last_peered":"2024-09-13T14:28:59.126737+0000","last_clean":"2024-09-13T14:28:59.126737+0000","last_became_active":"2024-09-13T14:27:01.355887+0000","last_became_peered":"2024-09-13T14:27:01.355887+0000","last_unstale":"2024-09-13T14:28:59.126737+0000","last_undegraded":"2024-09-13T14:28:59.126737+0000","last_fullsized":"2024-09-13T14:28:59.126737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:38.544216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028694700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250511+0000","last_change":"2024-09-13T14:28:14.409834+0000","last_active":"2024-09-13T14:28:57.250511+0000","last_peered":"2024-09-13T14:28:57.250511+0000","last_clean":"2024-09-13T14:28:57.250511+0000","last_became_active":"2024-09-13T14:28:14.409329+0000","last_became_peered":"2024-09-13T14:28:14.409329+0000","last_unstale":"2024-09-13T14:28:57.250511+0000","last_undegraded":"2024-09-13T14:28:57.250511+0000","last_fullsized":"2024-09-13T14:28:57.250511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:14:35.576410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244694+0000","last_change":"2024-09-13T14:28:14.295388+0000","last_active":"2024-09-13T14:28:57.244694+0000","last_peered":"2024-09-13T14:28:57.244694+0000","last_clean":"2024-09-13T14:28:57.244694+0000","last_became_active":"2024-09-13T14:28:14.295136+0000","last_became_peered":"2024-09-13T14:28:14.295136+0000","last_unstale":"2024-09-13T14:28:57.244694+0000","last_undegraded":"2024-09-13T14:28:57.244694+0000","last_fullsized":"2024-09-13T14:28:57.244694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:37.959212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246510+0000","last_change":"2024-09-13T14:27:05.652142+0000","last_active":"2024-09-13T14:28:57.246510+0000","last_peered":"2024-09-13T14:28:57.246510+0000","last_clean":"2024-09-13T14:28:57.246510+0000","last_became_active":"2024-09-13T14:27:01.356003+0000","last_became_peered":"2024-09-13T14:27:01.356003+0000","last_unstale":"2024-09-13T14:28:57.246510+0000","last_undegraded":"2024-09-13T14:28:57.246510+0000","last_fullsized":"2024-09-13T14:28:57.246510+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:10.323111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029468799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243602+0000","last_change":"2024-09-13T14:28:14.409799+0000","last_active":"2024-09-13T14:28:57.243602+0000","last_peered":"2024-09-13T14:28:57.243602+0000","last_clean":"2024-09-13T14:28:57.243602+0000","last_became_active":"2024-09-13T14:28:14.409351+0000","last_became_peered":"2024-09-13T14:28:14.409351+0000","last_unstale":"2024-09-13T14:28:57.243602+0000","last_undegraded":"2024-09-13T14:28:57.243602+0000","last_fullsized":"2024-09-13T14:28:57.243602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:21.464446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250527+0000","last_change":"2024-09-13T14:28:14.298784+0000","last_active":"2024-09-13T14:28:57.250527+0000","last_peered":"2024-09-13T14:28:57.250527+0000","last_clean":"2024-09-13T14:28:57.250527+0000","last_became_active":"2024-09-13T14:28:14.297847+0000","last_became_peered":"2024-09-13T14:28:14.297847+0000","last_unstale":"2024-09-13T14:28:57.250527+0000","last_undegraded":"2024-09-13T14:28:57.250527+0000","last_fullsized":"2024-09-13T14:28:57.250527+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:49.368487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.232118+0000","last_change":"2024-09-13T14:27:05.656833+0000","last_active":"2024-09-13T14:28:58.232118+0000","last_peered":"2024-09-13T14:28:58.232118+0000","last_clean":"2024-09-13T14:28:58.232118+0000","last_became_active":"2024-09-13T14:27:01.156746+0000","last_became_peered":"2024-09-13T14:27:01.156746+0000","last_unstale":"2024-09-13T14:28:58.232118+0000","last_undegraded":"2024-09-13T14:28:58.232118+0000","last_fullsized":"2024-09-13T14:28:58.232118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:47.883758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031071700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242832+0000","last_change":"2024-09-13T14:28:14.320314+0000","last_active":"2024-09-13T14:28:57.242832+0000","last_peered":"2024-09-13T14:28:57.242832+0000","last_clean":"2024-09-13T14:28:57.242832+0000","last_became_active":"2024-09-13T14:28:14.319934+0000","last_became_peered":"2024-09-13T14:28:14.319934+0000","last_unstale":"2024-09-13T14:28:57.242832+0000","last_undegraded":"2024-09-13T14:28:57.242832+0000","last_fullsized":"2024-09-13T14:28:57.242832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:21.514353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243071+0000","last_change":"2024-09-13T14:28:14.328360+0000","last_active":"2024-09-13T14:28:57.243071+0000","last_peered":"2024-09-13T14:28:57.243071+0000","last_clean":"2024-09-13T14:28:57.243071+0000","last_became_active":"2024-09-13T14:28:14.328069+0000","last_became_peered":"2024-09-13T14:28:14.328069+0000","last_unstale":"2024-09-13T14:28:57.243071+0000","last_undegraded":"2024-09-13T14:28:57.243071+0000","last_fullsized":"2024-09-13T14:28:57.243071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:16.188556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243382+0000","last_change":"2024-09-13T14:28:14.293329+0000","last_active":"2024-09-13T14:28:57.243382+0000","last_peered":"2024-09-13T14:28:57.243382+0000","last_clean":"2024-09-13T14:28:57.243382+0000","last_became_active":"2024-09-13T14:28:14.293075+0000","last_became_peered":"2024-09-13T14:28:14.293075+0000","last_unstale":"2024-09-13T14:28:57.243382+0000","last_undegraded":"2024-09-13T14:28:57.243382+0000","last_fullsized":"2024-09-13T14:28:57.243382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:20.310761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246262+0000","last_change":"2024-09-13T14:28:14.326719+0000","last_active":"2024-09-13T14:28:57.246262+0000","last_peered":"2024-09-13T14:28:57.246262+0000","last_clean":"2024-09-13T14:28:57.246262+0000","last_became_active":"2024-09-13T14:28:14.326235+0000","last_became_peered":"2024-09-13T14:28:14.326235+0000","last_unstale":"2024-09-13T14:28:57.246262+0000","last_undegraded":"2024-09-13T14:28:57.246262+0000","last_fullsized":"2024-09-13T14:28:57.246262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:50.544694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426318+0000","last_change":"2024-09-13T14:28:14.294604+0000","last_active":"2024-09-13T14:28:57.426318+0000","last_peered":"2024-09-13T14:28:57.426318+0000","last_clean":"2024-09-13T14:28:57.426318+0000","last_became_active":"2024-09-13T14:28:14.294325+0000","last_became_peered":"2024-09-13T14:28:14.294325+0000","last_unstale":"2024-09-13T14:28:57.426318+0000","last_undegraded":"2024-09-13T14:28:57.426318+0000","last_fullsized":"2024-09-13T14:28:57.426318+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:00.093345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245601+0000","last_change":"2024-09-13T14:28:14.290143+0000","last_active":"2024-09-13T14:28:57.245601+0000","last_peered":"2024-09-13T14:28:57.245601+0000","last_clean":"2024-09-13T14:28:57.245601+0000","last_became_active":"2024-09-13T14:28:14.289841+0000","last_became_peered":"2024-09-13T14:28:14.289841+0000","last_unstale":"2024-09-13T14:28:57.245601+0000","last_undegraded":"2024-09-13T14:28:57.245601+0000","last_fullsized":"2024-09-13T14:28:57.245601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:02.491519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244624+0000","last_change":"2024-09-13T14:28:14.409499+0000","last_active":"2024-09-13T14:28:57.244624+0000","last_peered":"2024-09-13T14:28:57.244624+0000","last_clean":"2024-09-13T14:28:57.244624+0000","last_became_active":"2024-09-13T14:28:14.409183+0000","last_became_peered":"2024-09-13T14:28:14.409183+0000","last_unstale":"2024-09-13T14:28:57.244624+0000","last_undegraded":"2024-09-13T14:28:57.244624+0000","last_fullsized":"2024-09-13T14:28:57.244624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:14.528158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250849+0000","last_change":"2024-09-13T14:28:14.305415+0000","last_active":"2024-09-13T14:28:57.250849+0000","last_peered":"2024-09-13T14:28:57.250849+0000","last_clean":"2024-09-13T14:28:57.250849+0000","last_became_active":"2024-09-13T14:28:14.305136+0000","last_became_peered":"2024-09-13T14:28:14.305136+0000","last_unstale":"2024-09-13T14:28:57.250849+0000","last_undegraded":"2024-09-13T14:28:57.250849+0000","last_fullsized":"2024-09-13T14:28:57.250849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:57.954514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243602+0000","last_change":"2024-09-13T14:28:14.326743+0000","last_active":"2024-09-13T14:28:57.243602+0000","last_peered":"2024-09-13T14:28:57.243602+0000","last_clean":"2024-09-13T14:28:57.243602+0000","last_became_active":"2024-09-13T14:28:14.326487+0000","last_became_peered":"2024-09-13T14:28:14.326487+0000","last_unstale":"2024-09-13T14:28:57.243602+0000","last_undegraded":"2024-09-13T14:28:57.243602+0000","last_fullsized":"2024-09-13T14:28:57.243602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:51.055640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.232112+0000","last_change":"2024-09-13T14:28:14.326719+0000","last_active":"2024-09-13T14:28:58.232112+0000","last_peered":"2024-09-13T14:28:58.232112+0000","last_clean":"2024-09-13T14:28:58.232112+0000","last_became_active":"2024-09-13T14:28:14.325781+0000","last_became_peered":"2024-09-13T14:28:14.325781+0000","last_unstale":"2024-09-13T14:28:58.232112+0000","last_undegraded":"2024-09-13T14:28:58.232112+0000","last_fullsized":"2024-09-13T14:28:58.232112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:55.145639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243879+0000","last_change":"2024-09-13T14:28:14.296674+0000","last_active":"2024-09-13T14:28:57.243879+0000","last_peered":"2024-09-13T14:28:57.243879+0000","last_clean":"2024-09-13T14:28:57.243879+0000","last_became_active":"2024-09-13T14:28:14.294834+0000","last_became_peered":"2024-09-13T14:28:14.294834+0000","last_unstale":"2024-09-13T14:28:57.243879+0000","last_undegraded":"2024-09-13T14:28:57.243879+0000","last_fullsized":"2024-09-13T14:28:57.243879+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:48.737423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250369+0000","last_change":"2024-09-13T14:28:14.410514+0000","last_active":"2024-09-13T14:28:57.250369+0000","last_peered":"2024-09-13T14:28:57.250369+0000","last_clean":"2024-09-13T14:28:57.250369+0000","last_became_active":"2024-09-13T14:28:14.410229+0000","last_became_peered":"2024-09-13T14:28:14.410229+0000","last_unstale":"2024-09-13T14:28:57.250369+0000","last_undegraded":"2024-09-13T14:28:57.250369+0000","last_fullsized":"2024-09-13T14:28:57.250369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:19.969842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246875+0000","last_change":"2024-09-13T14:28:14.299703+0000","last_active":"2024-09-13T14:28:57.246875+0000","last_peered":"2024-09-13T14:28:57.246875+0000","last_clean":"2024-09-13T14:28:57.246875+0000","last_became_active":"2024-09-13T14:28:14.299438+0000","last_became_peered":"2024-09-13T14:28:14.299438+0000","last_unstale":"2024-09-13T14:28:57.246875+0000","last_undegraded":"2024-09-13T14:28:57.246875+0000","last_fullsized":"2024-09-13T14:28:57.246875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:35.365026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244428+0000","last_change":"2024-09-13T14:28:14.327313+0000","last_active":"2024-09-13T14:28:57.244428+0000","last_peered":"2024-09-13T14:28:57.244428+0000","last_clean":"2024-09-13T14:28:57.244428+0000","last_became_active":"2024-09-13T14:28:14.327070+0000","last_became_peered":"2024-09-13T14:28:14.327070+0000","last_unstale":"2024-09-13T14:28:57.244428+0000","last_undegraded":"2024-09-13T14:28:57.244428+0000","last_fullsized":"2024-09-13T14:28:57.244428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:51.182604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242970+0000","last_change":"2024-09-13T14:28:14.300953+0000","last_active":"2024-09-13T14:28:57.242970+0000","last_peered":"2024-09-13T14:28:57.242970+0000","last_clean":"2024-09-13T14:28:57.242970+0000","last_became_active":"2024-09-13T14:28:14.300680+0000","last_became_peered":"2024-09-13T14:28:14.300680+0000","last_unstale":"2024-09-13T14:28:57.242970+0000","last_undegraded":"2024-09-13T14:28:57.242970+0000","last_fullsized":"2024-09-13T14:28:57.242970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:50.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245428+0000","last_change":"2024-09-13T14:28:15.784014+0000","last_active":"2024-09-13T14:28:57.245428+0000","last_peered":"2024-09-13T14:28:57.245428+0000","last_clean":"2024-09-13T14:28:57.245428+0000","last_became_active":"2024-09-13T14:28:15.783699+0000","last_became_peered":"2024-09-13T14:28:15.783699+0000","last_unstale":"2024-09-13T14:28:57.245428+0000","last_undegraded":"2024-09-13T14:28:57.245428+0000","last_fullsized":"2024-09-13T14:28:57.245428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:35.219154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250603+0000","last_change":"2024-09-13T14:28:14.300429+0000","last_active":"2024-09-13T14:28:57.250603+0000","last_peered":"2024-09-13T14:28:57.250603+0000","last_clean":"2024-09-13T14:28:57.250603+0000","last_became_active":"2024-09-13T14:28:14.300170+0000","last_became_peered":"2024-09-13T14:28:14.300170+0000","last_unstale":"2024-09-13T14:28:57.250603+0000","last_undegraded":"2024-09-13T14:28:57.250603+0000","last_fullsized":"2024-09-13T14:28:57.250603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:23.449026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.693277+0000","last_change":"2024-09-13T14:28:58.693277+0000","last_active":"2024-09-13T14:28:58.693277+0000","last_peered":"2024-09-13T14:28:58.693277+0000","last_clean":"2024-09-13T14:28:58.693277+0000","last_became_active":"2024-09-13T14:28:58.692922+0000","last_became_peered":"2024-09-13T14:28:58.692922+0000","last_unstale":"2024-09-13T14:28:58.693277+0000","last_undegraded":"2024-09-13T14:28:58.693277+0000","last_fullsized":"2024-09-13T14:28:58.693277+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:29.745606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245464+0000","last_change":"2024-09-13T14:28:14.297221+0000","last_active":"2024-09-13T14:28:57.245464+0000","last_peered":"2024-09-13T14:28:57.245464+0000","last_clean":"2024-09-13T14:28:57.245464+0000","last_became_active":"2024-09-13T14:28:14.296948+0000","last_became_peered":"2024-09-13T14:28:14.296948+0000","last_unstale":"2024-09-13T14:28:57.245464+0000","last_undegraded":"2024-09-13T14:28:57.245464+0000","last_fullsized":"2024-09-13T14:28:57.245464+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:09.514319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242909+0000","last_change":"2024-09-13T14:28:14.326771+0000","last_active":"2024-09-13T14:28:57.242909+0000","last_peered":"2024-09-13T14:28:57.242909+0000","last_clean":"2024-09-13T14:28:57.242909+0000","last_became_active":"2024-09-13T14:28:14.326385+0000","last_became_peered":"2024-09-13T14:28:14.326385+0000","last_unstale":"2024-09-13T14:28:57.242909+0000","last_undegraded":"2024-09-13T14:28:57.242909+0000","last_fullsized":"2024-09-13T14:28:57.242909+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:31.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245617+0000","last_change":"2024-09-13T14:28:14.297761+0000","last_active":"2024-09-13T14:28:57.245617+0000","last_peered":"2024-09-13T14:28:57.245617+0000","last_clean":"2024-09-13T14:28:57.245617+0000","last_became_active":"2024-09-13T14:28:14.297478+0000","last_became_peered":"2024-09-13T14:28:14.297478+0000","last_unstale":"2024-09-13T14:28:57.245617+0000","last_undegraded":"2024-09-13T14:28:57.245617+0000","last_fullsized":"2024-09-13T14:28:57.245617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:48.450673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244697+0000","last_change":"2024-09-13T14:28:14.321376+0000","last_active":"2024-09-13T14:28:57.244697+0000","last_peered":"2024-09-13T14:28:57.244697+0000","last_clean":"2024-09-13T14:28:57.244697+0000","last_became_active":"2024-09-13T14:28:14.321111+0000","last_became_peered":"2024-09-13T14:28:14.321111+0000","last_unstale":"2024-09-13T14:28:57.244697+0000","last_undegraded":"2024-09-13T14:28:57.244697+0000","last_fullsized":"2024-09-13T14:28:57.244697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:05.841741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244469+0000","last_change":"2024-09-13T14:28:14.302559+0000","last_active":"2024-09-13T14:28:57.244469+0000","last_peered":"2024-09-13T14:28:57.244469+0000","last_clean":"2024-09-13T14:28:57.244469+0000","last_became_active":"2024-09-13T14:28:14.302316+0000","last_became_peered":"2024-09-13T14:28:14.302316+0000","last_unstale":"2024-09-13T14:28:57.244469+0000","last_undegraded":"2024-09-13T14:28:57.244469+0000","last_fullsized":"2024-09-13T14:28:57.244469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:13.022170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242743+0000","last_change":"2024-09-13T14:28:14.326173+0000","last_active":"2024-09-13T14:28:57.242743+0000","last_peered":"2024-09-13T14:28:57.242743+0000","last_clean":"2024-09-13T14:28:57.242743+0000","last_became_active":"2024-09-13T14:28:14.325858+0000","last_became_peered":"2024-09-13T14:28:14.325858+0000","last_unstale":"2024-09-13T14:28:57.242743+0000","last_undegraded":"2024-09-13T14:28:57.242743+0000","last_fullsized":"2024-09-13T14:28:57.242743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:54.148777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250827+0000","last_change":"2024-09-13T14:28:14.298881+0000","last_active":"2024-09-13T14:28:57.250827+0000","last_peered":"2024-09-13T14:28:57.250827+0000","last_clean":"2024-09-13T14:28:57.250827+0000","last_became_active":"2024-09-13T14:28:14.298584+0000","last_became_peered":"2024-09-13T14:28:14.298584+0000","last_unstale":"2024-09-13T14:28:57.250827+0000","last_undegraded":"2024-09-13T14:28:57.250827+0000","last_fullsized":"2024-09-13T14:28:57.250827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:57.287159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234020,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1376256,"data_stored":925413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":0.72199999999999998,"15min":0.59399999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":18.359999999999999,"5min":18.359999999999999,"15min":18.359999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76200000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.68400000000000005,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":12.74,"5min":12.74,"15min":12.74},"last":0.58199999999999996}]},{"osd":2,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56799999999999995}]},{"osd":3,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.498},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.71399999999999997,"15min":0.58399999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":18.314,"5min":18.314,"15min":18.314},"last":0.626}]},{"osd":4,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.436},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.433}]},{"osd":5,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66000000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69799999999999995,"5min":0.755,"15min":0.755},"last":0.55000000000000004}]},{"osd":6,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64300000000000002,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47099999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1363968,"data_stored":925013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.49199999999999999,"15min":0.48099999999999998},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.629}]},{"osd":1,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69399999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.41699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66900000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71299999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66300000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51900000000000002,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67700000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.68400000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.74399999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.40899999999999997},"min":{"1min":0.32400000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.44,"15min":0.42699999999999999},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70199999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74199999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.72899999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717384,"statfs":{"total":95995035648,"available":95966601216,"internally_reserved":0,"allocated":765952,"data_stored":333994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.38400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69999999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.505,"15min":0.501},"min":{"1min":0.377,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64700000000000002,"5min":0.747,"15min":0.747},"last":0.51100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.38100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65700000000000003,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.51500000000000001,"15min":0.51000000000000001},"min":{"1min":0.40400000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56200000000000006}]},{"osd":2,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50800000000000001,"15min":0.503},"min":{"1min":0.38500000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.63700000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.39000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.625,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66400000000000003,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.68500000000000005}]},{"osd":4,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.39800000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54300000000000004,"5min":0.87,"15min":0.87},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41799999999999998,"15min":0.40300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68200000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.433}]},{"osd":6,"last update":"Fri Sep 13 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42999999999999999,"15min":0.42099999999999999},"min":{"1min":0.316,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.499},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.29899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.70599999999999996,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.62}]},{"osd":7,"last update":"Fri Sep 13 14:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69499999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.58899999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46800000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":782336,"data_stored":335306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66200000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.503,"15min":0.48999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63200000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66700000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.501,"15min":0.499},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.61299999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.501,"15min":0.499},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.64700000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.748,"5min":0.56899999999999995,"15min":0.53100000000000003},"min":{"1min":0.433,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":4.7560000000000002},"last":0.61799999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69799999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.627,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.50800000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40799999999999997,"15min":0.39400000000000002},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65300000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66500000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.374}]},{"osd":6,"last update":"Fri Sep 13 14:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67300000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39300000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.497,"15min":0.495},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.499,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41799999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27760,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":761856,"data_stored":333686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.36399999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":0.49299999999999999,"5min":0.621,"15min":0.621},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.40300000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48999999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.45100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39800000000000002,"15min":0.42099999999999999},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.51000000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.39800000000000002,"15min":0.42999999999999999},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Sep 13 14:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40999999999999998,"15min":0.42799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52100000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.441},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.59299999999999997,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.71099999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39700000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89200000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.40999999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.77800000000000002,"5min":0.997,"15min":0.997},"last":0.80900000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.748}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31860,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":765952,"data_stored":334212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:40 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.39900000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:28:40 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.378,"15min":0.375},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39000000000000001,"15min":0.379},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.30399999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41199999999999998,"15min":0.432},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58199999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41999999999999998,"15min":0.44},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56100000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.34000000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.54500000000000004,"15min":0.53600000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.51500000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.40100000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.41399999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:28:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.456,"5min":0.73299999999999998,"15min":0.61299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":17.195,"5min":17.195,"15min":17.195},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.42799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.50800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31860,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":761856,"data_stored":333615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.32300000000000001},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58499999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.36799999999999999,"15min":0.32400000000000001},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.54800000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.45900000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.50900000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.38,"15min":0.38100000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.496,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39800000000000002,"15min":0.40999999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.497,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.497},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46899999999999997},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.51700000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79700000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69399999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Fri Sep 13 14:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.753,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78500000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Sep 13 14:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.438,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.73899999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.75800000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.626}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1359872,"data_stored":924568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:28:17 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.33700000000000002,"15min":0.31900000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.48899999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.33600000000000002},"min":{"1min":0.25600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.378,"15min":0.35799999999999998},"min":{"1min":0.27200000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.52200000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38100000000000001,"15min":0.374},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.39300000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.44500000000000001,"15min":0.47799999999999998},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.48099999999999998,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.44,"15min":0.47799999999999998},"min":{"1min":0.309,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.46400000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.312}]},{"osd":4,"last update":"Fri Sep 13 14:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76200000000000001,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.46500000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60099999999999998,"15min":0.61099999999999999},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:28:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.62},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78700000000000003,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71699999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:29:10.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:29:10.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:10 smithi172 ceph-mon[32794]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:10.797 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:11 smithi154 ceph-mon[29001]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:11.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:11 smithi154 ceph-mon[39074]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:11.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:11 smithi172 ceph-mon[32794]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:12 smithi154 ceph-mon[29001]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:12 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:12.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:12 smithi154 ceph-mon[39074]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:12.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:12 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:12.472 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:29:12.473 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:29:12.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:12 smithi172 ceph-mon[32794]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:12.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:12 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:13.169 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-13T14:29:11.154820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592749,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":232104,"kb_used_data":7752,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":749729112,"statfs":{"total":767960285184,"available":767722610688,"internally_reserved":0,"allocated":7938048,"data_stored":4445807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229625401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002327"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244216+0000","last_change":"2024-09-13T14:28:14.409388+0000","last_active":"2024-09-13T14:28:57.244216+0000","last_peered":"2024-09-13T14:28:57.244216+0000","last_clean":"2024-09-13T14:28:57.244216+0000","last_became_active":"2024-09-13T14:28:14.409134+0000","last_became_peered":"2024-09-13T14:28:14.409134+0000","last_unstale":"2024-09-13T14:28:57.244216+0000","last_undegraded":"2024-09-13T14:28:57.244216+0000","last_fullsized":"2024-09-13T14:28:57.244216+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:03.857964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426174+0000","last_change":"2024-09-13T14:28:14.300970+0000","last_active":"2024-09-13T14:28:57.426174+0000","last_peered":"2024-09-13T14:28:57.426174+0000","last_clean":"2024-09-13T14:28:57.426174+0000","last_became_active":"2024-09-13T14:28:14.300651+0000","last_became_peered":"2024-09-13T14:28:14.300651+0000","last_unstale":"2024-09-13T14:28:57.426174+0000","last_undegraded":"2024-09-13T14:28:57.426174+0000","last_fullsized":"2024-09-13T14:28:57.426174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:03.739287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245568+0000","last_change":"2024-09-13T14:28:14.411333+0000","last_active":"2024-09-13T14:28:57.245568+0000","last_peered":"2024-09-13T14:28:57.245568+0000","last_clean":"2024-09-13T14:28:57.245568+0000","last_became_active":"2024-09-13T14:28:14.411035+0000","last_became_peered":"2024-09-13T14:28:14.411035+0000","last_unstale":"2024-09-13T14:28:57.245568+0000","last_undegraded":"2024-09-13T14:28:57.245568+0000","last_fullsized":"2024-09-13T14:28:57.245568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:39.991519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250377+0000","last_change":"2024-09-13T14:28:14.294701+0000","last_active":"2024-09-13T14:28:57.250377+0000","last_peered":"2024-09-13T14:28:57.250377+0000","last_clean":"2024-09-13T14:28:57.250377+0000","last_became_active":"2024-09-13T14:28:14.294413+0000","last_became_peered":"2024-09-13T14:28:14.294413+0000","last_unstale":"2024-09-13T14:28:57.250377+0000","last_undegraded":"2024-09-13T14:28:57.250377+0000","last_fullsized":"2024-09-13T14:28:57.250377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:54.638082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250494+0000","last_change":"2024-09-13T14:28:14.410590+0000","last_active":"2024-09-13T14:28:57.250494+0000","last_peered":"2024-09-13T14:28:57.250494+0000","last_clean":"2024-09-13T14:28:57.250494+0000","last_became_active":"2024-09-13T14:28:14.410276+0000","last_became_peered":"2024-09-13T14:28:14.410276+0000","last_unstale":"2024-09-13T14:28:57.250494+0000","last_undegraded":"2024-09-13T14:28:57.250494+0000","last_fullsized":"2024-09-13T14:28:57.250494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:01.051473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426053+0000","last_change":"2024-09-13T14:28:14.292467+0000","last_active":"2024-09-13T14:28:57.426053+0000","last_peered":"2024-09-13T14:28:57.426053+0000","last_clean":"2024-09-13T14:28:57.426053+0000","last_became_active":"2024-09-13T14:28:14.292208+0000","last_became_peered":"2024-09-13T14:28:14.292208+0000","last_unstale":"2024-09-13T14:28:57.426053+0000","last_undegraded":"2024-09-13T14:28:57.426053+0000","last_fullsized":"2024-09-13T14:28:57.426053+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:51.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244606+0000","last_change":"2024-09-13T14:28:14.409497+0000","last_active":"2024-09-13T14:28:57.244606+0000","last_peered":"2024-09-13T14:28:57.244606+0000","last_clean":"2024-09-13T14:28:57.244606+0000","last_became_active":"2024-09-13T14:28:14.409036+0000","last_became_peered":"2024-09-13T14:28:14.409036+0000","last_unstale":"2024-09-13T14:28:57.244606+0000","last_undegraded":"2024-09-13T14:28:57.244606+0000","last_fullsized":"2024-09-13T14:28:57.244606+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:47:59.234267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.231775+0000","last_change":"2024-09-13T14:28:14.292189+0000","last_active":"2024-09-13T14:28:58.231775+0000","last_peered":"2024-09-13T14:28:58.231775+0000","last_clean":"2024-09-13T14:28:58.231775+0000","last_became_active":"2024-09-13T14:28:14.291937+0000","last_became_peered":"2024-09-13T14:28:14.291937+0000","last_unstale":"2024-09-13T14:28:58.231775+0000","last_undegraded":"2024-09-13T14:28:58.231775+0000","last_fullsized":"2024-09-13T14:28:58.231775+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:09.136718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426526+0000","last_change":"2024-09-13T14:28:14.296105+0000","last_active":"2024-09-13T14:28:57.426526+0000","last_peered":"2024-09-13T14:28:57.426526+0000","last_clean":"2024-09-13T14:28:57.426526+0000","last_became_active":"2024-09-13T14:28:14.295855+0000","last_became_peered":"2024-09-13T14:28:14.295855+0000","last_unstale":"2024-09-13T14:28:57.426526+0000","last_undegraded":"2024-09-13T14:28:57.426526+0000","last_fullsized":"2024-09-13T14:28:57.426526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:52:31.446251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245100+0000","last_change":"2024-09-13T14:28:14.409515+0000","last_active":"2024-09-13T14:28:57.245100+0000","last_peered":"2024-09-13T14:28:57.245100+0000","last_clean":"2024-09-13T14:28:57.245100+0000","last_became_active":"2024-09-13T14:28:14.409202+0000","last_became_peered":"2024-09-13T14:28:14.409202+0000","last_unstale":"2024-09-13T14:28:57.245100+0000","last_undegraded":"2024-09-13T14:28:57.245100+0000","last_fullsized":"2024-09-13T14:28:57.245100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:24.530144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426473+0000","last_change":"2024-09-13T14:28:14.296094+0000","last_active":"2024-09-13T14:28:57.426473+0000","last_peered":"2024-09-13T14:28:57.426473+0000","last_clean":"2024-09-13T14:28:57.426473+0000","last_became_active":"2024-09-13T14:28:14.295808+0000","last_became_peered":"2024-09-13T14:28:14.295808+0000","last_unstale":"2024-09-13T14:28:57.426473+0000","last_undegraded":"2024-09-13T14:28:57.426473+0000","last_fullsized":"2024-09-13T14:28:57.426473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:12.440358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.254208+0000","last_change":"2024-09-13T14:28:58.254208+0000","last_active":"2024-09-13T14:28:58.254208+0000","last_peered":"2024-09-13T14:28:58.254208+0000","last_clean":"2024-09-13T14:28:58.254208+0000","last_became_active":"2024-09-13T14:28:58.253926+0000","last_became_peered":"2024-09-13T14:28:58.253926+0000","last_unstale":"2024-09-13T14:28:58.254208+0000","last_undegraded":"2024-09-13T14:28:58.254208+0000","last_fullsized":"2024-09-13T14:28:58.254208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:22.344730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426385+0000","last_change":"2024-09-13T14:28:14.289966+0000","last_active":"2024-09-13T14:28:57.426385+0000","last_peered":"2024-09-13T14:28:57.426385+0000","last_clean":"2024-09-13T14:28:57.426385+0000","last_became_active":"2024-09-13T14:28:14.289682+0000","last_became_peered":"2024-09-13T14:28:14.289682+0000","last_unstale":"2024-09-13T14:28:57.426385+0000","last_undegraded":"2024-09-13T14:28:57.426385+0000","last_fullsized":"2024-09-13T14:28:57.426385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:37.096298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.231774+0000","last_change":"2024-09-13T14:28:14.410365+0000","last_active":"2024-09-13T14:28:58.231774+0000","last_peered":"2024-09-13T14:28:58.231774+0000","last_clean":"2024-09-13T14:28:58.231774+0000","last_became_active":"2024-09-13T14:28:14.410093+0000","last_became_peered":"2024-09-13T14:28:14.410093+0000","last_unstale":"2024-09-13T14:28:58.231774+0000","last_undegraded":"2024-09-13T14:28:58.231774+0000","last_fullsized":"2024-09-13T14:28:58.231774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:40.610581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245500+0000","last_change":"2024-09-13T14:28:14.288606+0000","last_active":"2024-09-13T14:28:57.245500+0000","last_peered":"2024-09-13T14:28:57.245500+0000","last_clean":"2024-09-13T14:28:57.245500+0000","last_became_active":"2024-09-13T14:28:14.288328+0000","last_became_peered":"2024-09-13T14:28:14.288328+0000","last_unstale":"2024-09-13T14:28:57.245500+0000","last_undegraded":"2024-09-13T14:28:57.245500+0000","last_fullsized":"2024-09-13T14:28:57.245500+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:05.628581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250438+0000","last_change":"2024-09-13T14:28:14.327415+0000","last_active":"2024-09-13T14:28:57.250438+0000","last_peered":"2024-09-13T14:28:57.250438+0000","last_clean":"2024-09-13T14:28:57.250438+0000","last_became_active":"2024-09-13T14:28:14.327182+0000","last_became_peered":"2024-09-13T14:28:14.327182+0000","last_unstale":"2024-09-13T14:28:57.250438+0000","last_undegraded":"2024-09-13T14:28:57.250438+0000","last_fullsized":"2024-09-13T14:28:57.250438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:17.594876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243962+0000","last_change":"2024-09-13T14:28:14.304255+0000","last_active":"2024-09-13T14:28:57.243962+0000","last_peered":"2024-09-13T14:28:57.243962+0000","last_clean":"2024-09-13T14:28:57.243962+0000","last_became_active":"2024-09-13T14:28:14.304033+0000","last_became_peered":"2024-09-13T14:28:14.304033+0000","last_unstale":"2024-09-13T14:28:57.243962+0000","last_undegraded":"2024-09-13T14:28:57.243962+0000","last_fullsized":"2024-09-13T14:28:57.243962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:10.401043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242961+0000","last_change":"2024-09-13T14:28:14.324130+0000","last_active":"2024-09-13T14:28:57.242961+0000","last_peered":"2024-09-13T14:28:57.242961+0000","last_clean":"2024-09-13T14:28:57.242961+0000","last_became_active":"2024-09-13T14:28:14.323864+0000","last_became_peered":"2024-09-13T14:28:14.323864+0000","last_unstale":"2024-09-13T14:28:57.242961+0000","last_undegraded":"2024-09-13T14:28:57.242961+0000","last_fullsized":"2024-09-13T14:28:57.242961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:49:53.539380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.231944+0000","last_change":"2024-09-13T14:28:14.292924+0000","last_active":"2024-09-13T14:28:58.231944+0000","last_peered":"2024-09-13T14:28:58.231944+0000","last_clean":"2024-09-13T14:28:58.231944+0000","last_became_active":"2024-09-13T14:28:14.292637+0000","last_became_peered":"2024-09-13T14:28:14.292637+0000","last_unstale":"2024-09-13T14:28:58.231944+0000","last_undegraded":"2024-09-13T14:28:58.231944+0000","last_fullsized":"2024-09-13T14:28:58.231944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:25.763246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243494+0000","last_change":"2024-09-13T14:28:14.328150+0000","last_active":"2024-09-13T14:28:57.243494+0000","last_peered":"2024-09-13T14:28:57.243494+0000","last_clean":"2024-09-13T14:28:57.243494+0000","last_became_active":"2024-09-13T14:28:14.327825+0000","last_became_peered":"2024-09-13T14:28:14.327825+0000","last_unstale":"2024-09-13T14:28:57.243494+0000","last_undegraded":"2024-09-13T14:28:57.243494+0000","last_fullsized":"2024-09-13T14:28:57.243494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:10:10.215900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243129+0000","last_change":"2024-09-13T14:28:14.301032+0000","last_active":"2024-09-13T14:28:57.243129+0000","last_peered":"2024-09-13T14:28:57.243129+0000","last_clean":"2024-09-13T14:28:57.243129+0000","last_became_active":"2024-09-13T14:28:14.300722+0000","last_became_peered":"2024-09-13T14:28:14.300722+0000","last_unstale":"2024-09-13T14:28:57.243129+0000","last_undegraded":"2024-09-13T14:28:57.243129+0000","last_fullsized":"2024-09-13T14:28:57.243129+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:58.411339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243662+0000","last_change":"2024-09-13T14:28:14.327373+0000","last_active":"2024-09-13T14:28:57.243662+0000","last_peered":"2024-09-13T14:28:57.243662+0000","last_clean":"2024-09-13T14:28:57.243662+0000","last_became_active":"2024-09-13T14:28:14.326930+0000","last_became_peered":"2024-09-13T14:28:14.326930+0000","last_unstale":"2024-09-13T14:28:57.243662+0000","last_undegraded":"2024-09-13T14:28:57.243662+0000","last_fullsized":"2024-09-13T14:28:57.243662+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:21.335368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243726+0000","last_change":"2024-09-13T14:28:14.303769+0000","last_active":"2024-09-13T14:28:57.243726+0000","last_peered":"2024-09-13T14:28:57.243726+0000","last_clean":"2024-09-13T14:28:57.243726+0000","last_became_active":"2024-09-13T14:28:14.302350+0000","last_became_peered":"2024-09-13T14:28:14.302350+0000","last_unstale":"2024-09-13T14:28:57.243726+0000","last_undegraded":"2024-09-13T14:28:57.243726+0000","last_fullsized":"2024-09-13T14:28:57.243726+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:57.905748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.247041+0000","last_change":"2024-09-13T14:28:14.330928+0000","last_active":"2024-09-13T14:28:57.247041+0000","last_peered":"2024-09-13T14:28:57.247041+0000","last_clean":"2024-09-13T14:28:57.247041+0000","last_became_active":"2024-09-13T14:28:14.330643+0000","last_became_peered":"2024-09-13T14:28:14.330643+0000","last_unstale":"2024-09-13T14:28:57.247041+0000","last_undegraded":"2024-09-13T14:28:57.247041+0000","last_fullsized":"2024-09-13T14:28:57.247041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:05.343917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":270,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426247+0000","last_change":"2024-09-13T14:26:30.129218+0000","last_active":"2024-09-13T14:28:57.426247+0000","last_peered":"2024-09-13T14:28:57.426247+0000","last_clean":"2024-09-13T14:28:57.426247+0000","last_became_active":"2024-09-13T14:26:30.128826+0000","last_became_peered":"2024-09-13T14:26:30.128826+0000","last_unstale":"2024-09-13T14:28:57.426247+0000","last_undegraded":"2024-09-13T14:28:57.426247+0000","last_fullsized":"2024-09-13T14:28:57.426247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:31.240178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.692335+0000","last_change":"2024-09-13T14:27:05.655358+0000","last_active":"2024-09-13T14:28:58.692335+0000","last_peered":"2024-09-13T14:28:58.692335+0000","last_clean":"2024-09-13T14:28:58.692335+0000","last_became_active":"2024-09-13T14:27:01.151577+0000","last_became_peered":"2024-09-13T14:27:01.151577+0000","last_unstale":"2024-09-13T14:28:58.692335+0000","last_undegraded":"2024-09-13T14:28:58.692335+0000","last_fullsized":"2024-09-13T14:28:58.692335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:07.643386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000399641,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244639+0000","last_change":"2024-09-13T14:28:14.291654+0000","last_active":"2024-09-13T14:28:57.244639+0000","last_peered":"2024-09-13T14:28:57.244639+0000","last_clean":"2024-09-13T14:28:57.244639+0000","last_became_active":"2024-09-13T14:28:14.291364+0000","last_became_peered":"2024-09-13T14:28:14.291364+0000","last_unstale":"2024-09-13T14:28:57.244639+0000","last_undegraded":"2024-09-13T14:28:57.244639+0000","last_fullsized":"2024-09-13T14:28:57.244639+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:04.098245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.232005+0000","last_change":"2024-09-13T14:28:14.409690+0000","last_active":"2024-09-13T14:28:58.232005+0000","last_peered":"2024-09-13T14:28:58.232005+0000","last_clean":"2024-09-13T14:28:58.232005+0000","last_became_active":"2024-09-13T14:28:14.409379+0000","last_became_peered":"2024-09-13T14:28:14.409379+0000","last_unstale":"2024-09-13T14:28:58.232005+0000","last_undegraded":"2024-09-13T14:28:58.232005+0000","last_fullsized":"2024-09-13T14:28:58.232005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:15.731901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:59.126529+0000","last_change":"2024-09-13T14:27:05.696440+0000","last_active":"2024-09-13T14:28:59.126529+0000","last_peered":"2024-09-13T14:28:59.126529+0000","last_clean":"2024-09-13T14:28:59.126529+0000","last_became_active":"2024-09-13T14:27:01.339069+0000","last_became_peered":"2024-09-13T14:27:01.339069+0000","last_unstale":"2024-09-13T14:28:59.126529+0000","last_undegraded":"2024-09-13T14:28:59.126529+0000","last_fullsized":"2024-09-13T14:28:59.126529+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:24.663185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025738200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250217+0000","last_change":"2024-09-13T14:28:14.305482+0000","last_active":"2024-09-13T14:28:57.250217+0000","last_peered":"2024-09-13T14:28:57.250217+0000","last_clean":"2024-09-13T14:28:57.250217+0000","last_became_active":"2024-09-13T14:28:14.305219+0000","last_became_peered":"2024-09-13T14:28:14.305219+0000","last_unstale":"2024-09-13T14:28:57.250217+0000","last_undegraded":"2024-09-13T14:28:57.250217+0000","last_fullsized":"2024-09-13T14:28:57.250217+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:03:04.168870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250441+0000","last_change":"2024-09-13T14:28:14.327196+0000","last_active":"2024-09-13T14:28:57.250441+0000","last_peered":"2024-09-13T14:28:57.250441+0000","last_clean":"2024-09-13T14:28:57.250441+0000","last_became_active":"2024-09-13T14:28:14.326955+0000","last_became_peered":"2024-09-13T14:28:14.326955+0000","last_unstale":"2024-09-13T14:28:57.250441+0000","last_undegraded":"2024-09-13T14:28:57.250441+0000","last_fullsized":"2024-09-13T14:28:57.250441+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:54:54.085094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.250901+0000","last_change":"2024-09-13T14:27:05.696202+0000","last_active":"2024-09-13T14:28:58.250901+0000","last_peered":"2024-09-13T14:28:58.250901+0000","last_clean":"2024-09-13T14:28:58.250901+0000","last_became_active":"2024-09-13T14:27:01.339177+0000","last_became_peered":"2024-09-13T14:27:01.339177+0000","last_unstale":"2024-09-13T14:28:58.250901+0000","last_undegraded":"2024-09-13T14:28:58.250901+0000","last_fullsized":"2024-09-13T14:28:58.250901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:01.237560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026063399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246151+0000","last_change":"2024-09-13T14:28:14.302972+0000","last_active":"2024-09-13T14:28:57.246151+0000","last_peered":"2024-09-13T14:28:57.246151+0000","last_clean":"2024-09-13T14:28:57.246151+0000","last_became_active":"2024-09-13T14:28:14.302708+0000","last_became_peered":"2024-09-13T14:28:14.302708+0000","last_unstale":"2024-09-13T14:28:57.246151+0000","last_undegraded":"2024-09-13T14:28:57.246151+0000","last_fullsized":"2024-09-13T14:28:57.246151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:15.883333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":11,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.710958+0000","last_change":"2024-09-13T14:28:58.710958+0000","last_active":"2024-09-13T14:28:58.710958+0000","last_peered":"2024-09-13T14:28:58.710958+0000","last_clean":"2024-09-13T14:28:58.710958+0000","last_became_active":"2024-09-13T14:28:58.710685+0000","last_became_peered":"2024-09-13T14:28:58.710685+0000","last_unstale":"2024-09-13T14:28:58.710958+0000","last_undegraded":"2024-09-13T14:28:58.710958+0000","last_fullsized":"2024-09-13T14:28:58.710958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:14.735996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":29,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.692244+0000","last_change":"2024-09-13T14:27:05.655902+0000","last_active":"2024-09-13T14:28:58.692244+0000","last_peered":"2024-09-13T14:28:58.692244+0000","last_clean":"2024-09-13T14:28:58.692244+0000","last_became_active":"2024-09-13T14:27:01.154210+0000","last_became_peered":"2024-09-13T14:27:01.154210+0000","last_unstale":"2024-09-13T14:28:58.692244+0000","last_undegraded":"2024-09-13T14:28:58.692244+0000","last_fullsized":"2024-09-13T14:28:58.692244+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:12.974096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000960205,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243146+0000","last_change":"2024-09-13T14:28:14.297316+0000","last_active":"2024-09-13T14:28:57.243146+0000","last_peered":"2024-09-13T14:28:57.243146+0000","last_clean":"2024-09-13T14:28:57.243146+0000","last_became_active":"2024-09-13T14:28:14.296753+0000","last_became_peered":"2024-09-13T14:28:14.296753+0000","last_unstale":"2024-09-13T14:28:57.243146+0000","last_undegraded":"2024-09-13T14:28:57.243146+0000","last_fullsized":"2024-09-13T14:28:57.243146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:47.774278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.247222+0000","last_change":"2024-09-13T14:28:14.331943+0000","last_active":"2024-09-13T14:28:57.247222+0000","last_peered":"2024-09-13T14:28:57.247222+0000","last_clean":"2024-09-13T14:28:57.247222+0000","last_became_active":"2024-09-13T14:28:14.331593+0000","last_became_peered":"2024-09-13T14:28:14.331593+0000","last_unstale":"2024-09-13T14:28:57.247222+0000","last_undegraded":"2024-09-13T14:28:57.247222+0000","last_fullsized":"2024-09-13T14:28:57.247222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:53.600541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245662+0000","last_change":"2024-09-13T14:28:14.302176+0000","last_active":"2024-09-13T14:28:57.245662+0000","last_peered":"2024-09-13T14:28:57.245662+0000","last_clean":"2024-09-13T14:28:57.245662+0000","last_became_active":"2024-09-13T14:28:14.301965+0000","last_became_peered":"2024-09-13T14:28:14.301965+0000","last_unstale":"2024-09-13T14:28:57.245662+0000","last_undegraded":"2024-09-13T14:28:57.245662+0000","last_fullsized":"2024-09-13T14:28:57.245662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:46.166594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.249432+0000","last_change":"2024-09-13T14:27:05.652163+0000","last_active":"2024-09-13T14:28:58.249432+0000","last_peered":"2024-09-13T14:28:58.249432+0000","last_clean":"2024-09-13T14:28:58.249432+0000","last_became_active":"2024-09-13T14:27:01.339091+0000","last_became_peered":"2024-09-13T14:27:01.339091+0000","last_unstale":"2024-09-13T14:28:58.249432+0000","last_undegraded":"2024-09-13T14:28:58.249432+0000","last_fullsized":"2024-09-13T14:28:58.249432+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:42.796882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031603499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246170+0000","last_change":"2024-09-13T14:28:14.326033+0000","last_active":"2024-09-13T14:28:57.246170+0000","last_peered":"2024-09-13T14:28:57.246170+0000","last_clean":"2024-09-13T14:28:57.246170+0000","last_became_active":"2024-09-13T14:28:14.325773+0000","last_became_peered":"2024-09-13T14:28:14.325773+0000","last_unstale":"2024-09-13T14:28:57.246170+0000","last_undegraded":"2024-09-13T14:28:57.246170+0000","last_fullsized":"2024-09-13T14:28:57.246170+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:40.260729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.247308+0000","last_change":"2024-09-13T14:28:14.304078+0000","last_active":"2024-09-13T14:28:57.247308+0000","last_peered":"2024-09-13T14:28:57.247308+0000","last_clean":"2024-09-13T14:28:57.247308+0000","last_became_active":"2024-09-13T14:28:14.303703+0000","last_became_peered":"2024-09-13T14:28:14.303703+0000","last_unstale":"2024-09-13T14:28:57.247308+0000","last_undegraded":"2024-09-13T14:28:57.247308+0000","last_fullsized":"2024-09-13T14:28:57.247308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:49:17.144867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:59.126737+0000","last_change":"2024-09-13T14:27:05.696490+0000","last_active":"2024-09-13T14:28:59.126737+0000","last_peered":"2024-09-13T14:28:59.126737+0000","last_clean":"2024-09-13T14:28:59.126737+0000","last_became_active":"2024-09-13T14:27:01.355887+0000","last_became_peered":"2024-09-13T14:27:01.355887+0000","last_unstale":"2024-09-13T14:28:59.126737+0000","last_undegraded":"2024-09-13T14:28:59.126737+0000","last_fullsized":"2024-09-13T14:28:59.126737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:38.544216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028694700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250511+0000","last_change":"2024-09-13T14:28:14.409834+0000","last_active":"2024-09-13T14:28:57.250511+0000","last_peered":"2024-09-13T14:28:57.250511+0000","last_clean":"2024-09-13T14:28:57.250511+0000","last_became_active":"2024-09-13T14:28:14.409329+0000","last_became_peered":"2024-09-13T14:28:14.409329+0000","last_unstale":"2024-09-13T14:28:57.250511+0000","last_undegraded":"2024-09-13T14:28:57.250511+0000","last_fullsized":"2024-09-13T14:28:57.250511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:14:35.576410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244694+0000","last_change":"2024-09-13T14:28:14.295388+0000","last_active":"2024-09-13T14:28:57.244694+0000","last_peered":"2024-09-13T14:28:57.244694+0000","last_clean":"2024-09-13T14:28:57.244694+0000","last_became_active":"2024-09-13T14:28:14.295136+0000","last_became_peered":"2024-09-13T14:28:14.295136+0000","last_unstale":"2024-09-13T14:28:57.244694+0000","last_undegraded":"2024-09-13T14:28:57.244694+0000","last_fullsized":"2024-09-13T14:28:57.244694+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:37.959212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246510+0000","last_change":"2024-09-13T14:27:05.652142+0000","last_active":"2024-09-13T14:28:57.246510+0000","last_peered":"2024-09-13T14:28:57.246510+0000","last_clean":"2024-09-13T14:28:57.246510+0000","last_became_active":"2024-09-13T14:27:01.356003+0000","last_became_peered":"2024-09-13T14:27:01.356003+0000","last_unstale":"2024-09-13T14:28:57.246510+0000","last_undegraded":"2024-09-13T14:28:57.246510+0000","last_fullsized":"2024-09-13T14:28:57.246510+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:17:10.323111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029468799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243602+0000","last_change":"2024-09-13T14:28:14.409799+0000","last_active":"2024-09-13T14:28:57.243602+0000","last_peered":"2024-09-13T14:28:57.243602+0000","last_clean":"2024-09-13T14:28:57.243602+0000","last_became_active":"2024-09-13T14:28:14.409351+0000","last_became_peered":"2024-09-13T14:28:14.409351+0000","last_unstale":"2024-09-13T14:28:57.243602+0000","last_undegraded":"2024-09-13T14:28:57.243602+0000","last_fullsized":"2024-09-13T14:28:57.243602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:21.464446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250527+0000","last_change":"2024-09-13T14:28:14.298784+0000","last_active":"2024-09-13T14:28:57.250527+0000","last_peered":"2024-09-13T14:28:57.250527+0000","last_clean":"2024-09-13T14:28:57.250527+0000","last_became_active":"2024-09-13T14:28:14.297847+0000","last_became_peered":"2024-09-13T14:28:14.297847+0000","last_unstale":"2024-09-13T14:28:57.250527+0000","last_undegraded":"2024-09-13T14:28:57.250527+0000","last_fullsized":"2024-09-13T14:28:57.250527+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:49.368487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.232118+0000","last_change":"2024-09-13T14:27:05.656833+0000","last_active":"2024-09-13T14:28:58.232118+0000","last_peered":"2024-09-13T14:28:58.232118+0000","last_clean":"2024-09-13T14:28:58.232118+0000","last_became_active":"2024-09-13T14:27:01.156746+0000","last_became_peered":"2024-09-13T14:27:01.156746+0000","last_unstale":"2024-09-13T14:28:58.232118+0000","last_undegraded":"2024-09-13T14:28:58.232118+0000","last_fullsized":"2024-09-13T14:28:58.232118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:47.883758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031071700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242832+0000","last_change":"2024-09-13T14:28:14.320314+0000","last_active":"2024-09-13T14:28:57.242832+0000","last_peered":"2024-09-13T14:28:57.242832+0000","last_clean":"2024-09-13T14:28:57.242832+0000","last_became_active":"2024-09-13T14:28:14.319934+0000","last_became_peered":"2024-09-13T14:28:14.319934+0000","last_unstale":"2024-09-13T14:28:57.242832+0000","last_undegraded":"2024-09-13T14:28:57.242832+0000","last_fullsized":"2024-09-13T14:28:57.242832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:21.514353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243071+0000","last_change":"2024-09-13T14:28:14.328360+0000","last_active":"2024-09-13T14:28:57.243071+0000","last_peered":"2024-09-13T14:28:57.243071+0000","last_clean":"2024-09-13T14:28:57.243071+0000","last_became_active":"2024-09-13T14:28:14.328069+0000","last_became_peered":"2024-09-13T14:28:14.328069+0000","last_unstale":"2024-09-13T14:28:57.243071+0000","last_undegraded":"2024-09-13T14:28:57.243071+0000","last_fullsized":"2024-09-13T14:28:57.243071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:16.188556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243382+0000","last_change":"2024-09-13T14:28:14.293329+0000","last_active":"2024-09-13T14:28:57.243382+0000","last_peered":"2024-09-13T14:28:57.243382+0000","last_clean":"2024-09-13T14:28:57.243382+0000","last_became_active":"2024-09-13T14:28:14.293075+0000","last_became_peered":"2024-09-13T14:28:14.293075+0000","last_unstale":"2024-09-13T14:28:57.243382+0000","last_undegraded":"2024-09-13T14:28:57.243382+0000","last_fullsized":"2024-09-13T14:28:57.243382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:20.310761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246262+0000","last_change":"2024-09-13T14:28:14.326719+0000","last_active":"2024-09-13T14:28:57.246262+0000","last_peered":"2024-09-13T14:28:57.246262+0000","last_clean":"2024-09-13T14:28:57.246262+0000","last_became_active":"2024-09-13T14:28:14.326235+0000","last_became_peered":"2024-09-13T14:28:14.326235+0000","last_unstale":"2024-09-13T14:28:57.246262+0000","last_undegraded":"2024-09-13T14:28:57.246262+0000","last_fullsized":"2024-09-13T14:28:57.246262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:50.544694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.426318+0000","last_change":"2024-09-13T14:28:14.294604+0000","last_active":"2024-09-13T14:28:57.426318+0000","last_peered":"2024-09-13T14:28:57.426318+0000","last_clean":"2024-09-13T14:28:57.426318+0000","last_became_active":"2024-09-13T14:28:14.294325+0000","last_became_peered":"2024-09-13T14:28:14.294325+0000","last_unstale":"2024-09-13T14:28:57.426318+0000","last_undegraded":"2024-09-13T14:28:57.426318+0000","last_fullsized":"2024-09-13T14:28:57.426318+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:00.093345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245601+0000","last_change":"2024-09-13T14:28:14.290143+0000","last_active":"2024-09-13T14:28:57.245601+0000","last_peered":"2024-09-13T14:28:57.245601+0000","last_clean":"2024-09-13T14:28:57.245601+0000","last_became_active":"2024-09-13T14:28:14.289841+0000","last_became_peered":"2024-09-13T14:28:14.289841+0000","last_unstale":"2024-09-13T14:28:57.245601+0000","last_undegraded":"2024-09-13T14:28:57.245601+0000","last_fullsized":"2024-09-13T14:28:57.245601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:02.491519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244624+0000","last_change":"2024-09-13T14:28:14.409499+0000","last_active":"2024-09-13T14:28:57.244624+0000","last_peered":"2024-09-13T14:28:57.244624+0000","last_clean":"2024-09-13T14:28:57.244624+0000","last_became_active":"2024-09-13T14:28:14.409183+0000","last_became_peered":"2024-09-13T14:28:14.409183+0000","last_unstale":"2024-09-13T14:28:57.244624+0000","last_undegraded":"2024-09-13T14:28:57.244624+0000","last_fullsized":"2024-09-13T14:28:57.244624+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:14.528158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250849+0000","last_change":"2024-09-13T14:28:14.305415+0000","last_active":"2024-09-13T14:28:57.250849+0000","last_peered":"2024-09-13T14:28:57.250849+0000","last_clean":"2024-09-13T14:28:57.250849+0000","last_became_active":"2024-09-13T14:28:14.305136+0000","last_became_peered":"2024-09-13T14:28:14.305136+0000","last_unstale":"2024-09-13T14:28:57.250849+0000","last_undegraded":"2024-09-13T14:28:57.250849+0000","last_fullsized":"2024-09-13T14:28:57.250849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:57.954514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243602+0000","last_change":"2024-09-13T14:28:14.326743+0000","last_active":"2024-09-13T14:28:57.243602+0000","last_peered":"2024-09-13T14:28:57.243602+0000","last_clean":"2024-09-13T14:28:57.243602+0000","last_became_active":"2024-09-13T14:28:14.326487+0000","last_became_peered":"2024-09-13T14:28:14.326487+0000","last_unstale":"2024-09-13T14:28:57.243602+0000","last_undegraded":"2024-09-13T14:28:57.243602+0000","last_fullsized":"2024-09-13T14:28:57.243602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:51.055640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.232112+0000","last_change":"2024-09-13T14:28:14.326719+0000","last_active":"2024-09-13T14:28:58.232112+0000","last_peered":"2024-09-13T14:28:58.232112+0000","last_clean":"2024-09-13T14:28:58.232112+0000","last_became_active":"2024-09-13T14:28:14.325781+0000","last_became_peered":"2024-09-13T14:28:14.325781+0000","last_unstale":"2024-09-13T14:28:58.232112+0000","last_undegraded":"2024-09-13T14:28:58.232112+0000","last_fullsized":"2024-09-13T14:28:58.232112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:55.145639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.243879+0000","last_change":"2024-09-13T14:28:14.296674+0000","last_active":"2024-09-13T14:28:57.243879+0000","last_peered":"2024-09-13T14:28:57.243879+0000","last_clean":"2024-09-13T14:28:57.243879+0000","last_became_active":"2024-09-13T14:28:14.294834+0000","last_became_peered":"2024-09-13T14:28:14.294834+0000","last_unstale":"2024-09-13T14:28:57.243879+0000","last_undegraded":"2024-09-13T14:28:57.243879+0000","last_fullsized":"2024-09-13T14:28:57.243879+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:48.737423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250369+0000","last_change":"2024-09-13T14:28:14.410514+0000","last_active":"2024-09-13T14:28:57.250369+0000","last_peered":"2024-09-13T14:28:57.250369+0000","last_clean":"2024-09-13T14:28:57.250369+0000","last_became_active":"2024-09-13T14:28:14.410229+0000","last_became_peered":"2024-09-13T14:28:14.410229+0000","last_unstale":"2024-09-13T14:28:57.250369+0000","last_undegraded":"2024-09-13T14:28:57.250369+0000","last_fullsized":"2024-09-13T14:28:57.250369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:19.969842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.246875+0000","last_change":"2024-09-13T14:28:14.299703+0000","last_active":"2024-09-13T14:28:57.246875+0000","last_peered":"2024-09-13T14:28:57.246875+0000","last_clean":"2024-09-13T14:28:57.246875+0000","last_became_active":"2024-09-13T14:28:14.299438+0000","last_became_peered":"2024-09-13T14:28:14.299438+0000","last_unstale":"2024-09-13T14:28:57.246875+0000","last_undegraded":"2024-09-13T14:28:57.246875+0000","last_fullsized":"2024-09-13T14:28:57.246875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:35.365026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244428+0000","last_change":"2024-09-13T14:28:14.327313+0000","last_active":"2024-09-13T14:28:57.244428+0000","last_peered":"2024-09-13T14:28:57.244428+0000","last_clean":"2024-09-13T14:28:57.244428+0000","last_became_active":"2024-09-13T14:28:14.327070+0000","last_became_peered":"2024-09-13T14:28:14.327070+0000","last_unstale":"2024-09-13T14:28:57.244428+0000","last_undegraded":"2024-09-13T14:28:57.244428+0000","last_fullsized":"2024-09-13T14:28:57.244428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:51.182604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242970+0000","last_change":"2024-09-13T14:28:14.300953+0000","last_active":"2024-09-13T14:28:57.242970+0000","last_peered":"2024-09-13T14:28:57.242970+0000","last_clean":"2024-09-13T14:28:57.242970+0000","last_became_active":"2024-09-13T14:28:14.300680+0000","last_became_peered":"2024-09-13T14:28:14.300680+0000","last_unstale":"2024-09-13T14:28:57.242970+0000","last_undegraded":"2024-09-13T14:28:57.242970+0000","last_fullsized":"2024-09-13T14:28:57.242970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:50.630085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245428+0000","last_change":"2024-09-13T14:28:15.784014+0000","last_active":"2024-09-13T14:28:57.245428+0000","last_peered":"2024-09-13T14:28:57.245428+0000","last_clean":"2024-09-13T14:28:57.245428+0000","last_became_active":"2024-09-13T14:28:15.783699+0000","last_became_peered":"2024-09-13T14:28:15.783699+0000","last_unstale":"2024-09-13T14:28:57.245428+0000","last_undegraded":"2024-09-13T14:28:57.245428+0000","last_fullsized":"2024-09-13T14:28:57.245428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:35.219154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250603+0000","last_change":"2024-09-13T14:28:14.300429+0000","last_active":"2024-09-13T14:28:57.250603+0000","last_peered":"2024-09-13T14:28:57.250603+0000","last_clean":"2024-09-13T14:28:57.250603+0000","last_became_active":"2024-09-13T14:28:14.300170+0000","last_became_peered":"2024-09-13T14:28:14.300170+0000","last_unstale":"2024-09-13T14:28:57.250603+0000","last_undegraded":"2024-09-13T14:28:57.250603+0000","last_fullsized":"2024-09-13T14:28:57.250603+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:23.449026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-13T14:28:58.693277+0000","last_change":"2024-09-13T14:28:58.693277+0000","last_active":"2024-09-13T14:28:58.693277+0000","last_peered":"2024-09-13T14:28:58.693277+0000","last_clean":"2024-09-13T14:28:58.693277+0000","last_became_active":"2024-09-13T14:28:58.692922+0000","last_became_peered":"2024-09-13T14:28:58.692922+0000","last_unstale":"2024-09-13T14:28:58.693277+0000","last_undegraded":"2024-09-13T14:28:58.693277+0000","last_fullsized":"2024-09-13T14:28:58.693277+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:29.745606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245464+0000","last_change":"2024-09-13T14:28:14.297221+0000","last_active":"2024-09-13T14:28:57.245464+0000","last_peered":"2024-09-13T14:28:57.245464+0000","last_clean":"2024-09-13T14:28:57.245464+0000","last_became_active":"2024-09-13T14:28:14.296948+0000","last_became_peered":"2024-09-13T14:28:14.296948+0000","last_unstale":"2024-09-13T14:28:57.245464+0000","last_undegraded":"2024-09-13T14:28:57.245464+0000","last_fullsized":"2024-09-13T14:28:57.245464+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:09.514319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242909+0000","last_change":"2024-09-13T14:28:14.326771+0000","last_active":"2024-09-13T14:28:57.242909+0000","last_peered":"2024-09-13T14:28:57.242909+0000","last_clean":"2024-09-13T14:28:57.242909+0000","last_became_active":"2024-09-13T14:28:14.326385+0000","last_became_peered":"2024-09-13T14:28:14.326385+0000","last_unstale":"2024-09-13T14:28:57.242909+0000","last_undegraded":"2024-09-13T14:28:57.242909+0000","last_fullsized":"2024-09-13T14:28:57.242909+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:31.430331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.245617+0000","last_change":"2024-09-13T14:28:14.297761+0000","last_active":"2024-09-13T14:28:57.245617+0000","last_peered":"2024-09-13T14:28:57.245617+0000","last_clean":"2024-09-13T14:28:57.245617+0000","last_became_active":"2024-09-13T14:28:14.297478+0000","last_became_peered":"2024-09-13T14:28:14.297478+0000","last_unstale":"2024-09-13T14:28:57.245617+0000","last_undegraded":"2024-09-13T14:28:57.245617+0000","last_fullsized":"2024-09-13T14:28:57.245617+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:48.450673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244697+0000","last_change":"2024-09-13T14:28:14.321376+0000","last_active":"2024-09-13T14:28:57.244697+0000","last_peered":"2024-09-13T14:28:57.244697+0000","last_clean":"2024-09-13T14:28:57.244697+0000","last_became_active":"2024-09-13T14:28:14.321111+0000","last_became_peered":"2024-09-13T14:28:14.321111+0000","last_unstale":"2024-09-13T14:28:57.244697+0000","last_undegraded":"2024-09-13T14:28:57.244697+0000","last_fullsized":"2024-09-13T14:28:57.244697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:37:05.841741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.244469+0000","last_change":"2024-09-13T14:28:14.302559+0000","last_active":"2024-09-13T14:28:57.244469+0000","last_peered":"2024-09-13T14:28:57.244469+0000","last_clean":"2024-09-13T14:28:57.244469+0000","last_became_active":"2024-09-13T14:28:14.302316+0000","last_became_peered":"2024-09-13T14:28:14.302316+0000","last_unstale":"2024-09-13T14:28:57.244469+0000","last_undegraded":"2024-09-13T14:28:57.244469+0000","last_fullsized":"2024-09-13T14:28:57.244469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:13.022170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.242743+0000","last_change":"2024-09-13T14:28:14.326173+0000","last_active":"2024-09-13T14:28:57.242743+0000","last_peered":"2024-09-13T14:28:57.242743+0000","last_clean":"2024-09-13T14:28:57.242743+0000","last_became_active":"2024-09-13T14:28:14.325858+0000","last_became_peered":"2024-09-13T14:28:14.325858+0000","last_unstale":"2024-09-13T14:28:57.242743+0000","last_undegraded":"2024-09-13T14:28:57.242743+0000","last_fullsized":"2024-09-13T14:28:57.242743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:54.148777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T14:28:57.250827+0000","last_change":"2024-09-13T14:28:14.298881+0000","last_active":"2024-09-13T14:28:57.250827+0000","last_peered":"2024-09-13T14:28:57.250827+0000","last_clean":"2024-09-13T14:28:57.250827+0000","last_became_active":"2024-09-13T14:28:14.298584+0000","last_became_peered":"2024-09-13T14:28:14.298584+0000","last_unstale":"2024-09-13T14:28:57.250827+0000","last_undegraded":"2024-09-13T14:28:57.250827+0000","last_fullsized":"2024-09-13T14:28:57.250827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:57.287159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234020,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1376256,"data_stored":925413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":0.72199999999999998,"15min":0.59399999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":18.359999999999999,"5min":18.359999999999999,"15min":18.359999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.41399999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.51100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76200000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.196,"5min":0.68400000000000005,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":12.74,"5min":12.74,"15min":12.74},"last":0.58199999999999996}]},{"osd":2,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56799999999999995}]},{"osd":3,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.498},{"interface":"front","average":{"1min":1.4930000000000001,"5min":0.71399999999999997,"15min":0.58399999999999996},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":18.314,"5min":18.314,"15min":18.314},"last":0.626}]},{"osd":4,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.42099999999999999,"15min":0.41699999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.436},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.433}]},{"osd":5,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.66000000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.69799999999999995,"5min":0.755,"15min":0.755},"last":0.55000000000000004}]},{"osd":6,"last update":"Fri Sep 13 14:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64300000000000002,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47099999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28356,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716796,"statfs":{"total":95995035648,"available":95965999104,"internally_reserved":0,"allocated":1363968,"data_stored":925013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.49199999999999999,"15min":0.48099999999999998},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.629}]},{"osd":1,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69399999999999995,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.41699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66900000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71299999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66300000000000003}]},{"osd":3,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51900000000000002,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67700000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.68400000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.74399999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.40899999999999997},"min":{"1min":0.32400000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.45700000000000002,"15min":0.44500000000000001},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Sep 13 14:28:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.44,"15min":0.42699999999999999},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70199999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45300000000000001,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53100000000000003,"15min":0.53000000000000003},"min":{"1min":0.35599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74199999999999999,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.72899999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717384,"statfs":{"total":95995035648,"available":95966601216,"internally_reserved":0,"allocated":765952,"data_stored":333994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.38400000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69999999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.505,"15min":0.501},"min":{"1min":0.377,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64700000000000002,"5min":0.747,"15min":0.747},"last":0.51100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.38100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65700000000000003,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.51500000000000001,"15min":0.51000000000000001},"min":{"1min":0.40400000000000003,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65900000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56200000000000006}]},{"osd":2,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.50800000000000001,"15min":0.503},"min":{"1min":0.38500000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.63700000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.39000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.625,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.52500000000000002,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66400000000000003,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.68500000000000005}]},{"osd":4,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.39800000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54300000000000004,"5min":0.87,"15min":0.87},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41799999999999998,"15min":0.40300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68200000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.433}]},{"osd":6,"last update":"Fri Sep 13 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42999999999999999,"15min":0.42099999999999999},"min":{"1min":0.316,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.499},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.46600000000000003,"15min":0.46400000000000002},"min":{"1min":0.29899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.70599999999999996,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.62}]},{"osd":7,"last update":"Fri Sep 13 14:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.495,"15min":0.49299999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69499999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.51200000000000001,"15min":0.52000000000000002},"min":{"1min":0.32800000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.58899999999999997,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46800000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":782336,"data_stored":335306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50800000000000001,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66200000000000003,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.503,"15min":0.48999999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.63200000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66700000000000004,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.501,"15min":0.499},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.61299999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.501,"15min":0.499},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.64700000000000002,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.748,"5min":0.56899999999999995,"15min":0.53100000000000003},"min":{"1min":0.433,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.7560000000000002,"5min":4.7560000000000002,"15min":4.7560000000000002},"last":0.61799999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51700000000000002,"15min":0.51100000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69799999999999995,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.51900000000000002},"min":{"1min":0.377,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.627,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.50800000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:28:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.40799999999999997,"15min":0.39400000000000002},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.65300000000000002,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.44500000000000001,"15min":0.42499999999999999},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66500000000000004,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.374}]},{"osd":6,"last update":"Fri Sep 13 14:29:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47799999999999998,"15min":0.46000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67300000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45100000000000001,"15min":0.441},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.39300000000000002}]},{"osd":7,"last update":"Fri Sep 13 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.497,"15min":0.495},"min":{"1min":0.33800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.499,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41799999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27760,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":761856,"data_stored":333686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.36399999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.19,"15min":0.19},"max":{"1min":0.49299999999999999,"5min":0.621,"15min":0.621},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.38600000000000001,"15min":0.40300000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48999999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.45100000000000001}]},{"osd":1,"last update":"Fri Sep 13 14:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.39800000000000002,"15min":0.42099999999999999},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.51000000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.39800000000000002,"15min":0.42999999999999999},"min":{"1min":0.24299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Sep 13 14:28:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40999999999999998,"15min":0.42799999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52100000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.441},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.59299999999999997,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Sep 13 14:28:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.54700000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.71099999999999997}]},{"osd":5,"last update":"Fri Sep 13 14:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59399999999999997,"15min":0.60699999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71599999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39700000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.89200000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Sep 13 14:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.40999999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.77800000000000002,"5min":0.997,"15min":0.997},"last":0.80900000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.79400000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.748}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31860,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":765952,"data_stored":334212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:40 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"min":{"1min":0.23300000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.39900000000000002}]},{"osd":1,"last update":"Fri Sep 13 14:28:40 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.378,"15min":0.375},"min":{"1min":0.26300000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39000000000000001,"15min":0.379},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.30399999999999999}]},{"osd":3,"last update":"Fri Sep 13 14:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.41199999999999998,"15min":0.432},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58199999999999996,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.41999999999999998,"15min":0.44},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56100000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.34000000000000002}]},{"osd":4,"last update":"Fri Sep 13 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.54500000000000004,"15min":0.53600000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.51500000000000001}]},{"osd":5,"last update":"Fri Sep 13 14:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.56599999999999995,"15min":0.57799999999999996},"min":{"1min":0.40100000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.41399999999999998}]},{"osd":6,"last update":"Fri Sep 13 14:28:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.456,"5min":0.73299999999999998,"15min":0.61299999999999999},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":17.195,"5min":17.195,"15min":17.195},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Sep 13 14:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54400000000000004,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.42799999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.50800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31860,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713292,"statfs":{"total":95995035648,"available":95962411008,"internally_reserved":0,"allocated":761856,"data_stored":333615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 14:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.32300000000000001},"min":{"1min":0.247,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.58499999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.36799999999999999,"15min":0.32400000000000001},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.54800000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.45900000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41299999999999998,"15min":0.40899999999999997},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.50900000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.38,"15min":0.38100000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.496,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Sep 13 14:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.39800000000000002,"15min":0.40999999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.497,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.497},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45100000000000001,"15min":0.46899999999999997},"min":{"1min":0.309,"5min":0.307,"15min":0.307},"max":{"1min":0.51700000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Sep 13 14:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79700000000000004,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69399999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Fri Sep 13 14:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59999999999999998,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.753,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78500000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Sep 13 14:29:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.438,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.73899999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55200000000000005}]},{"osd":7,"last update":"Fri Sep 13 14:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.39900000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.75800000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.626}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1359872,"data_stored":924568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 14:28:17 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.33700000000000002,"15min":0.31900000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.48899999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.33600000000000002},"min":{"1min":0.25600000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Fri Sep 13 14:28:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.378,"15min":0.35799999999999998},"min":{"1min":0.27200000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.52200000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38100000000000001,"15min":0.374},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51800000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.39300000000000002}]},{"osd":3,"last update":"Fri Sep 13 14:29:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.44500000000000001,"15min":0.47799999999999998},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.48099999999999998,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.44,"15min":0.47799999999999998},"min":{"1min":0.309,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.46400000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.312}]},{"osd":4,"last update":"Fri Sep 13 14:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55100000000000005,"15min":0.54100000000000004},"min":{"1min":0.38300000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76200000000000001,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.46500000000000002}]},{"osd":5,"last update":"Fri Sep 13 14:28:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60099999999999998,"15min":0.61099999999999999},"min":{"1min":0.39800000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Sep 13 14:28:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.62},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59499999999999997,"15min":0.59399999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.78700000000000003,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Sep 13 14:28:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.61599999999999999,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71699999999999997,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:29:13.173 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:29:13.173 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:29:13.173 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:29:13.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:29:13.375 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:14.435 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:14 smithi154 ceph-mon[39074]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:14.435 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:14 smithi154 ceph-mon[39074]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:14.435 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:14 smithi154 ceph-mon[29001]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:14.435 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:14 smithi154 ceph-mon[29001]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:14.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:14 smithi172 ceph-mon[32794]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:29:14.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:14 smithi172 ceph-mon[32794]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:15.108 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:29:15.109 INFO:teuthology.orchestra.run.smithi154.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:29:15.383 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:15 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2861213345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:29:15.384 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:15 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2861213345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:29:15.665 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:29:15.665 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:29:15.675 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-13T14:29:15.675 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:29:15.683 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:29:15.683 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:29:15.684 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-13T14:29:15.689 INFO:tasks.workunit:Pulling workunits from ref reef 2024-09-13T14:29:15.690 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T14:29:15.690 DEBUG:teuthology.orchestra.run.smithi154:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:29:15.691 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:29:15.691 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-13T14:29:15.691 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T14:29:15.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-13T14:29:15.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:29:15.708 INFO:teuthology.orchestra.run.smithi154.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T14:29:15.709 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:29:15.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:15 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2861213345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T14:29:15.763 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T14:29:15.763 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T14:29:15.818 INFO:tasks.workunit:timeout=3h 2024-09-13T14:29:15.818 INFO:tasks.workunit:cleanup=True 2024-09-13T14:29:15.819 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-13T14:29:15.877 INFO:tasks.workunit.client.0.smithi154.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T14:29:15.898 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:16 smithi154 ceph-mon[29001]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:16.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:16 smithi154 ceph-mon[39074]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:16.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:16 smithi172 ceph-mon[32794]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:17.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:29:17] "GET /metrics HTTP/1.1" 200 33723 "" "Prometheus/2.43.0" 2024-09-13T14:29:18.109 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:29:17] ENGINE Bus STOPPING 2024-09-13T14:29:18.109 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:29:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:29:18.109 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:29:17] ENGINE Bus STOPPED 2024-09-13T14:29:18.109 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:29:17] ENGINE Bus STARTING 2024-09-13T14:29:18.109 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:29:17] ENGINE Serving on http://:::9283 2024-09-13T14:29:18.109 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:29:17] ENGINE Bus STARTED 2024-09-13T14:29:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[29001]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[29001]: from='client.? ' entity='client.admin' 2024-09-13T14:29:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:29:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:29:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:29:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:29:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[39074]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[39074]: from='client.? ' entity='client.admin' 2024-09-13T14:29:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:29:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:29:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:29:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:18 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:29:18.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-13T14:29:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:18 smithi172 ceph-mon[32794]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:18 smithi172 ceph-mon[32794]: from='client.? ' entity='client.admin' 2024-09-13T14:29:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:18 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:29:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:18 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:29:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:18 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:29:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:18 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:29:18.791 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:20.584 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:20 smithi154 ceph-mon[29001]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:20.584 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:20 smithi154 ceph-mon[39074]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:20.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:20 smithi172 ceph-mon[32794]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:20.889 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T14:29:20.949 INFO:tasks.workunit.client.0.smithi154.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T14:29:20.951 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T14:29:20.952 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T14:29:22.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for mgr daemons to upgrade' 2024-09-13T14:29:22.431 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:22.466 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T14:29:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:22 smithi154 ceph-mon[29001]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:22.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:22 smithi154 ceph-mon[39074]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:22.588 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T14:29:22.628 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T14:29:22.629 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T14:29:22.630 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T14:29:22.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:22 smithi172 ceph-mon[32794]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:29:22.784 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T14:29:22.787 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T14:29:22.787 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T14:29:22.803 INFO:tasks.workunit:Running workunits matching rbd/import_export.sh on client.0... 2024-09-13T14:29:22.804 INFO:tasks.workunit:Running workunit rbd/import_export.sh... 2024-09-13T14:29:22.804 DEBUG:teuthology.orchestra.run.smithi154:workunit test rbd/import_export.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/import_export.sh 2024-09-13T14:29:22.950 INFO:tasks.workunit.client.0.smithi154.stderr:+ export RBD_FORCE_ALLOW_V1=1 2024-09-13T14:29:22.950 INFO:tasks.workunit.client.0.smithi154.stderr:+ RBD_FORCE_ALLOW_V1=1 2024-09-13T14:29:22.950 INFO:tasks.workunit.client.0.smithi154.stderr:+ TMPDIR=/tmp/rbd_import_export_78348 2024-09-13T14:29:22.950 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm -rf /tmp/rbd_import_export_78348 2024-09-13T14:29:22.951 INFO:tasks.workunit.client.0.smithi154.stderr:+ mkdir /tmp/rbd_import_export_78348 2024-09-13T14:29:22.952 INFO:tasks.workunit.client.0.smithi154.stderr:+ trap 'rm -rf /tmp/rbd_import_export_78348' INT TERM EXIT 2024-09-13T14:29:22.952 INFO:tasks.workunit.client.0.smithi154.stderr:+ mkdir foo.78348 2024-09-13T14:29:22.953 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import foo.78348 foo.dir 2024-09-13T14:29:23.018 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: cannot import a directory 2024-09-13T14:29:23.018 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 0% complete...failed. 2024-09-13T14:29:23.018 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: import failed: (21) Is a directory 2024-09-13T14:29:23.037 INFO:tasks.workunit.client.0.smithi154.stderr:+ true 2024-09-13T14:29:23.037 INFO:tasks.workunit.client.0.smithi154.stderr:+ rmdir foo.78348 2024-09-13T14:29:23.062 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/bin/sh of=/tmp/rbd_import_export_78348/img bs=1k count=1 seek=10 2024-09-13T14:29:23.063 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records in 2024-09-13T14:29:23.063 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records out 2024-09-13T14:29:23.063 INFO:tasks.workunit.client.0.smithi154.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 3.9903e-05 s, 25.7 MB/s 2024-09-13T14:29:23.063 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_78348/img bs=1k count=10 seek=100 2024-09-13T14:29:23.063 INFO:tasks.workunit.client.0.smithi154.stderr:10+0 records in 2024-09-13T14:29:23.064 INFO:tasks.workunit.client.0.smithi154.stderr:10+0 records out 2024-09-13T14:29:23.064 INFO:tasks.workunit.client.0.smithi154.stderr:10240 bytes (10 kB, 10 KiB) copied, 6.8733e-05 s, 149 MB/s 2024-09-13T14:29:23.064 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/bin/rm of=/tmp/rbd_import_export_78348/img bs=1k count=100 seek=1000 2024-09-13T14:29:23.065 INFO:tasks.workunit.client.0.smithi154.stderr:60+1 records in 2024-09-13T14:29:23.065 INFO:tasks.workunit.client.0.smithi154.stderr:60+1 records out 2024-09-13T14:29:23.065 INFO:tasks.workunit.client.0.smithi154.stderr:61640 bytes (62 kB, 60 KiB) copied, 0.000150031 s, 411 MB/s 2024-09-13T14:29:23.065 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/bin/ls of=/tmp/rbd_import_export_78348/img bs=1k seek=10000 2024-09-13T14:29:23.066 INFO:tasks.workunit.client.0.smithi154.stderr:137+1 records in 2024-09-13T14:29:23.066 INFO:tasks.workunit.client.0.smithi154.stderr:137+1 records out 2024-09-13T14:29:23.066 INFO:tasks.workunit.client.0.smithi154.stderr:140928 bytes (141 kB, 138 KiB) copied, 0.000492897 s, 286 MB/s 2024-09-13T14:29:23.066 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/bin/ln of=/tmp/rbd_import_export_78348/img bs=1k seek=100000 2024-09-13T14:29:23.067 INFO:tasks.workunit.client.0.smithi154.stderr:60+1 records in 2024-09-13T14:29:23.067 INFO:tasks.workunit.client.0.smithi154.stderr:60+1 records out 2024-09-13T14:29:23.067 INFO:tasks.workunit.client.0.smithi154.stderr:61592 bytes (62 kB, 60 KiB) copied, 0.000149707 s, 411 MB/s 2024-09-13T14:29:23.067 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/bin/grep of=/tmp/rbd_import_export_78348/img bs=1k seek=1000000 2024-09-13T14:29:23.068 INFO:tasks.workunit.client.0.smithi154.stderr:154+1 records in 2024-09-13T14:29:23.068 INFO:tasks.workunit.client.0.smithi154.stderr:154+1 records out 2024-09-13T14:29:23.068 INFO:tasks.workunit.client.0.smithi154.stderr:158176 bytes (158 kB, 154 KiB) copied, 0.000336241 s, 470 MB/s 2024-09-13T14:29:23.068 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm testimg 2024-09-13T14:29:23.108 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 0% complete...failed. 2024-09-13T14:29:23.108 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: delete error: (2) No such file or directory 2024-09-13T14:29:23.111 INFO:tasks.workunit.client.0.smithi154.stderr:+ true 2024-09-13T14:29:23.111 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format /tmp/rbd_import_export_78348/img testimg 2024-09-13T14:29:23.639 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 1% complete... Importing image: 2% complete... Importing image: 3% complete... Importing image: 4% complete... Importing image: 5% complete... Importing image: 6% complete... Importing image: 7% complete... Importing image: 8% complete... Importing image: 9% complete... Importing image: 10% complete... Importing image: 11% complete... Importing image: 12% complete... Importing image: 13% complete... Importing image: 14% complete... Importing image: 15% complete... Importing image: 16% complete... Importing image: 17% complete... Importing image: 18% complete... Importing image: 19% complete... Importing image: 20% complete... Importing image: 21% complete... Importing image: 22% complete... Importing image: 23% complete... Importing image: 24% complete... Importing image: 25% complete... Importing image: 26% complete... Importing image: 27% complete... Importing image: 28% complete... Importing image: 29% complete... Importing image: 30% complete... Importing image: 31% complete... Importing image: 32% complete... Importing image: 33% complete... Importing image: 34% complete... Importing image: 35% complete... Importing image: 36% complete... Importing image: 37% complete... Importing image: 38% complete... Importing image: 39% complete... Importing image: 40% complete... Importing image: 41% complete... Importing image: 42% complete... Importing image: 43% complete... Importing image: 44% complete... Importing image: 45% complete... Importing image: 46% complete... Importing image: 47% complete... Importing image: 48% complete... Importing image: 49% complete... Importing image: 50% complete... Importing image: 51% complete... Importing image: 52% complete... Importing image: 53% complete... Importing image: 54% complete... Importing image: 55% complete... Importing image: 56% complete... Importing image: 57% complete... Importing image: 58% complete... Importing image: 59% complete... Importing image: 60% complete... Importing image: 61% complete... Importing image: 62% complete... Importing image: 63% complete... Importing image: 64% complete... Importing image: 65% complete... Importing image: 66% complete... Importing image: 67% complete... Importing image: 68% complete... Importing image: 69% complete... Importing image: 70% complete... Importing image: 71% complete... Importing image: 72% complete... Importing image: 73% complete... Importing image: 74% complete... Importing image: 75% complete... Importing image: 76% complete... Importing image: 77% complete... Importing image: 78% complete... Importing image: 79% complete... Importing image: 80% complete... Importing image: 81% complete... Importing image: 82% complete... Importing image: 83% complete... Importing image: 84% complete... Importing image: 85% complete... Importing image: 86% complete... Importing image: 87% complete... Importing image: 88% complete... Importing image: 89% complete... Importing image: 90% complete... Importing image: 91% complete... Importing image: 92% complete... Importing image: 93% complete... Importing image: 94% complete... Importing image: 95% complete... Importing image: 96% complete... Importing image: 97% complete... Importing image: 98% complete... Importing image: 99% complete... Importing image: 100% complete...done. 2024-09-13T14:29:23.676 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:23.676 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export testimg /tmp/rbd_import_export_78348/img2 2024-09-13T14:29:23.842 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:23.846 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export testimg - 2024-09-13T14:29:24.479 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:24.483 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm testimg 2024-09-13T14:29:24.565 INFO:teuthology.orchestra.run.smithi154.stdout:wait for mgr daemons to upgrade 2024-09-13T14:29:24.565 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:24 smithi154 ceph-mon[29001]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-13T14:29:24.566 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:24 smithi154 ceph-mon[39074]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-13T14:29:24.576 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T14:29:24.579 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp /tmp/rbd_import_export_78348/img /tmp/rbd_import_export_78348/img2 2024-09-13T14:29:24.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:24 smithi172 ceph-mon[32794]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-13T14:29:25.129 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp /tmp/rbd_import_export_78348/img /tmp/rbd_import_export_78348/img3 2024-09-13T14:29:25.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-09-13T14:29:25.419 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:29:25.555 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img2 /tmp/rbd_import_export_78348/img3 2024-09-13T14:29:25.721 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format - testimg 2024-09-13T14:29:26.102 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:26.102 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:26.106 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export testimg /tmp/rbd_import_export_78348/img2 2024-09-13T14:29:26.283 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:26.288 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export testimg - 2024-09-13T14:29:26.563 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:26 smithi154 ceph-mon[29001]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2024-09-13T14:29:26.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:26 smithi154 ceph-mon[39074]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2024-09-13T14:29:26.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:26 smithi172 ceph-mon[32794]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2024-09-13T14:29:26.921 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:26.926 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm testimg 2024-09-13T14:29:27.018 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-13T14:29:27.021 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp /tmp/rbd_import_export_78348/img /tmp/rbd_import_export_78348/img2 2024-09-13T14:29:27.596 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp /tmp/rbd_import_export_78348/img /tmp/rbd_import_export_78348/img3 2024-09-13T14:29:27.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:27 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:27.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:29:27] "GET /metrics HTTP/1.1" 200 33727 "" "Prometheus/2.43.0" 2024-09-13T14:29:27.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:27 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:27.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:27 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:28.013 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img /tmp/rbd_import_export_78348/img2 /tmp/rbd_import_export_78348/img3 2024-09-13T14:29:28.252 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd help export 2024-09-13T14:29:28.252 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep -q export-format 2024-09-13T14:29:28.275 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_78348/img bs=1k count=10 seek=100 2024-09-13T14:29:28.276 INFO:tasks.workunit.client.0.smithi154.stderr:10+0 records in 2024-09-13T14:29:28.276 INFO:tasks.workunit.client.0.smithi154.stderr:10+0 records out 2024-09-13T14:29:28.276 INFO:tasks.workunit.client.0.smithi154.stderr:10240 bytes (10 kB, 10 KiB) copied, 6.5752e-05 s, 156 MB/s 2024-09-13T14:29:28.277 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format /tmp/rbd_import_export_78348/img testimg 2024-09-13T14:29:28.337 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:28.337 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:28.340 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap create testimg@snap 2024-09-13T14:29:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:28 smithi154 ceph-mon[29001]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2024-09-13T14:29:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:28 smithi154 ceph-mon[29001]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:28 smithi154 ceph-mon[39074]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2024-09-13T14:29:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:28 smithi154 ceph-mon[39074]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:28.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:28 smithi172 ceph-mon[32794]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 85 B/s wr, 0 op/s 2024-09-13T14:29:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:28 smithi172 ceph-mon[32794]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:29:29.334 INFO:tasks.workunit.client.0.smithi154.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T14:29:29.341 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd image-meta set testimg key1 value1 2024-09-13T14:29:29.398 INFO:tasks.workunit.client.0.smithi154.stderr:++ rbd image-meta list testimg 2024-09-13T14:29:29.443 INFO:tasks.workunit.client.0.smithi154.stderr:+ IMAGEMETA_BEFORE='There is 1 metadatum on this image: 2024-09-13T14:29:29.443 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-13T14:29:29.443 INFO:tasks.workunit.client.0.smithi154.stderr:Key Value 2024-09-13T14:29:29.443 INFO:tasks.workunit.client.0.smithi154.stderr:key1 value1' 2024-09-13T14:29:29.443 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:29.488 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:29.491 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_78348/img_v2 testimg_import 2024-09-13T14:29:30.346 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 50% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:29:30.350 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd info testimg_import 2024-09-13T14:29:30.392 INFO:tasks.workunit.client.0.smithi154.stdout:rbd image 'testimg_import': 2024-09-13T14:29:30.392 INFO:tasks.workunit.client.0.smithi154.stdout: size 110 KiB in 1 objects 2024-09-13T14:29:30.392 INFO:tasks.workunit.client.0.smithi154.stdout: order 22 (4 MiB objects) 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: snapshot_count: 1 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: id: 3a3560fa417c 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: block_name_prefix: rbd_data.3a3560fa417c 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: format: 2 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: op_features: 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: flags: 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: create_timestamp: Fri Sep 13 14:29:29 2024 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: access_timestamp: Fri Sep 13 14:29:29 2024 2024-09-13T14:29:30.393 INFO:tasks.workunit.client.0.smithi154.stdout: modify_timestamp: Fri Sep 13 14:29:29 2024 2024-09-13T14:29:30.397 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd info testimg_import@snap 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout:rbd image 'testimg_import': 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: size 110 KiB in 1 objects 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: order 22 (4 MiB objects) 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: snapshot_count: 1 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: id: 3a3560fa417c 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: block_name_prefix: rbd_data.3a3560fa417c 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: format: 2 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: op_features: 2024-09-13T14:29:30.438 INFO:tasks.workunit.client.0.smithi154.stdout: flags: 2024-09-13T14:29:30.439 INFO:tasks.workunit.client.0.smithi154.stdout: create_timestamp: Fri Sep 13 14:29:29 2024 2024-09-13T14:29:30.439 INFO:tasks.workunit.client.0.smithi154.stdout: access_timestamp: Fri Sep 13 14:29:29 2024 2024-09-13T14:29:30.439 INFO:tasks.workunit.client.0.smithi154.stdout: modify_timestamp: Fri Sep 13 14:29:29 2024 2024-09-13T14:29:30.439 INFO:tasks.workunit.client.0.smithi154.stdout: protected: False 2024-09-13T14:29:30.441 INFO:tasks.workunit.client.0.smithi154.stderr:++ rbd image-meta list testimg_import 2024-09-13T14:29:30.486 INFO:tasks.workunit.client.0.smithi154.stderr:+ IMAGEMETA_AFTER='There is 1 metadatum on this image: 2024-09-13T14:29:30.486 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-13T14:29:30.486 INFO:tasks.workunit.client.0.smithi154.stderr:Key Value 2024-09-13T14:29:30.486 INFO:tasks.workunit.client.0.smithi154.stderr:key1 value1' 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 'There is 1 metadatum on this image: 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr:Key Value 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr:key1 value1' = 'There is 1 metadatum on this image: 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr:Key Value 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr:key1 value1' ']' 2024-09-13T14:29:30.487 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export testimg_import /tmp/rbd_import_export_78348/img_import 2024-09-13T14:29:30.525 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete...done. 2024-09-13T14:29:30.529 INFO:tasks.workunit.client.0.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_78348/img /tmp/rbd_import_export_78348/img_import 2024-09-13T14:29:30.529 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_78348/img /tmp/rbd_import_export_78348/img_import 2024-09-13T14:29:30.530 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/img --format %b 2024-09-13T14:29:30.531 INFO:tasks.workunit.client.0.smithi154.stderr:+ origsize=24 2024-09-13T14:29:30.531 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/img_import --format %b 2024-09-13T14:29:30.532 INFO:tasks.workunit.client.0.smithi154.stderr:+ exportsize=224 2024-09-13T14:29:30.532 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=200 2024-09-13T14:29:30.532 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=200 2024-09-13T14:29:30.532 INFO:tasks.workunit.client.0.smithi154.stderr:+ test 200 -ge 0 -a 200 -lt 4096 2024-09-13T14:29:30.532 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export testimg@snap /tmp/rbd_import_export_78348/img_snap 2024-09-13T14:29:30.573 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete...done. 2024-09-13T14:29:30.576 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export testimg_import@snap /tmp/rbd_import_export_78348/img_snap_import 2024-09-13T14:29:30.618 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete...done. 2024-09-13T14:29:30.621 INFO:tasks.workunit.client.0.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_78348/img_snap /tmp/rbd_import_export_78348/img_snap_import 2024-09-13T14:29:30.621 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_78348/img_snap /tmp/rbd_import_export_78348/img_snap_import 2024-09-13T14:29:30.622 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/img_snap --format %b 2024-09-13T14:29:30.623 INFO:tasks.workunit.client.0.smithi154.stderr:+ origsize=224 2024-09-13T14:29:30.623 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/img_snap_import --format %b 2024-09-13T14:29:30.624 INFO:tasks.workunit.client.0.smithi154.stderr:+ exportsize=224 2024-09-13T14:29:30.624 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:30.624 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:30.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:29:30.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:30.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img_import 2024-09-13T14:29:30.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img_snap 2024-09-13T14:29:30.626 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img_snap_import 2024-09-13T14:29:30.627 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap rm testimg_import@snap 2024-09-13T14:29:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:30 smithi154 ceph-mon[29001]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 78 KiB/s wr, 67 op/s 2024-09-13T14:29:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:30 smithi154 ceph-mon[29001]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:29:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:30 smithi154 ceph-mon[39074]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 78 KiB/s wr, 67 op/s 2024-09-13T14:29:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:30 smithi154 ceph-mon[39074]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:29:30.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:30 smithi172 ceph-mon[32794]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 78 KiB/s wr, 67 op/s 2024-09-13T14:29:30.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:30 smithi172 ceph-mon[32794]: osdmap e74: 8 total, 8 up, 8 in 2024-09-13T14:29:31.348 INFO:tasks.workunit.client.0.smithi154.stderr: Removing snap: 100% complete...done. 2024-09-13T14:29:31.356 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg_import 2024-09-13T14:29:31.451 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:31.454 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap rm testimg@snap 2024-09-13T14:29:31.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:31 smithi154 ceph-mon[29001]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:29:31.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:31 smithi154 ceph-mon[39074]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:29:31.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:31 smithi172 ceph-mon[32794]: osdmap e75: 8 total, 8 up, 8 in 2024-09-13T14:29:32.346 INFO:tasks.workunit.client.0.smithi154.stderr: Removing snap: 100% complete...done. 2024-09-13T14:29:32.354 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm testimg 2024-09-13T14:29:32.445 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:32.448 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --order 20 /tmp/rbd_import_export_78348/img testimg 2024-09-13T14:29:32.510 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:32.510 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:29:32.513 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:32.553 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:32.556 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_78348/img_v2 testimg_import 2024-09-13T14:29:32.622 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:29:32.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd info testimg_import 2024-09-13T14:29:32.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep order 2024-09-13T14:29:32.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ awk '{print $2}' 2024-09-13T14:29:32.625 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep 20 2024-09-13T14:29:32.641 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:32 smithi172 ceph-mon[32794]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 290 KiB/s rd, 117 KiB/s wr, 100 op/s 2024-09-13T14:29:32.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:32 smithi172 ceph-mon[32794]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:29:32.666 INFO:tasks.workunit.client.0.smithi154.stdout:20 2024-09-13T14:29:32.666 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:32.666 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg_import 2024-09-13T14:29:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:32 smithi154 ceph-mon[29001]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 290 KiB/s rd, 117 KiB/s wr, 100 op/s 2024-09-13T14:29:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:32 smithi154 ceph-mon[29001]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:29:32.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:32 smithi154 ceph-mon[39074]: pgmap v96: 73 pgs: 73 active+clean; 579 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 290 KiB/s rd, 117 KiB/s wr, 100 op/s 2024-09-13T14:29:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:32 smithi154 ceph-mon[39074]: osdmap e76: 8 total, 8 up, 8 in 2024-09-13T14:29:32.754 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:32.757 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg 2024-09-13T14:29:32.842 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:32.845 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --image-feature layering /tmp/rbd_import_export_78348/img testimg 2024-09-13T14:29:32.899 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:32.902 INFO:tasks.workunit.client.0.smithi154.stderr:++ rbd info testimg 2024-09-13T14:29:32.903 INFO:tasks.workunit.client.0.smithi154.stderr:++ grep features 2024-09-13T14:29:32.942 INFO:tasks.workunit.client.0.smithi154.stderr:+ FEATURES_BEFORE=' features: layering 2024-09-13T14:29:32.942 INFO:tasks.workunit.client.0.smithi154.stderr: op_features: ' 2024-09-13T14:29:32.942 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:32.987 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:32.990 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_78348/img_v2 testimg_import 2024-09-13T14:29:33.047 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:29:33.051 INFO:tasks.workunit.client.0.smithi154.stderr:++ rbd info testimg_import 2024-09-13T14:29:33.051 INFO:tasks.workunit.client.0.smithi154.stderr:++ grep features 2024-09-13T14:29:33.095 INFO:tasks.workunit.client.0.smithi154.stderr:+ FEATURES_AFTER=' features: layering 2024-09-13T14:29:33.095 INFO:tasks.workunit.client.0.smithi154.stderr: op_features: ' 2024-09-13T14:29:33.095 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' ' features: layering 2024-09-13T14:29:33.095 INFO:tasks.workunit.client.0.smithi154.stderr: op_features: ' '!=' ' features: layering 2024-09-13T14:29:33.095 INFO:tasks.workunit.client.0.smithi154.stderr: op_features: ' ']' 2024-09-13T14:29:33.095 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:33.096 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg_import 2024-09-13T14:29:33.167 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:33.170 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg 2024-09-13T14:29:33.241 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:33.244 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --stripe-count 1000 --stripe-unit 4096 /tmp/rbd_import_export_78348/img testimg 2024-09-13T14:29:33.309 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:33.312 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:33.357 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:33.360 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_78348/img_v2 testimg_import 2024-09-13T14:29:33.424 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:29:33.428 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd info testimg_import 2024-09-13T14:29:33.428 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep 'stripe unit' 2024-09-13T14:29:33.428 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep -Ei '(4 KiB|4096)' 2024-09-13T14:29:33.469 INFO:tasks.workunit.client.0.smithi154.stdout: stripe unit: 4 KiB 2024-09-13T14:29:33.470 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd info testimg_import 2024-09-13T14:29:33.470 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep 'stripe count' 2024-09-13T14:29:33.470 INFO:tasks.workunit.client.0.smithi154.stderr:+ awk '{print $3}' 2024-09-13T14:29:33.470 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep 1000 2024-09-13T14:29:33.510 INFO:tasks.workunit.client.0.smithi154.stdout:1000 2024-09-13T14:29:33.511 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/img_v2 2024-09-13T14:29:33.511 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg_import 2024-09-13T14:29:33.599 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T14:29:33.602 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg 2024-09-13T14:29:33.693 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-13T14:29:33.697 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --image-format=2 /tmp/rbd_import_export_78348/img testimg 2024-09-13T14:29:33.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:33 smithi154 ceph-mon[29001]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:29:33.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:33 smithi154 ceph-mon[39074]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:29:33.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:33 smithi172 ceph-mon[32794]: osdmap e77: 8 total, 8 up, 8 in 2024-09-13T14:29:33.752 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:33.755 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap create testimg@snap1 2024-09-13T14:29:34.365 INFO:tasks.workunit.client.0.smithi154.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T14:29:34.371 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap create testimg@snap2 2024-09-13T14:29:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:34 smithi154 ceph-mon[29001]: pgmap v99: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 233 KiB/s rd, 25 KiB/s wr, 290 op/s 2024-09-13T14:29:34.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:34 smithi154 ceph-mon[39074]: pgmap v99: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 233 KiB/s rd, 25 KiB/s wr, 290 op/s 2024-09-13T14:29:34.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:34 smithi172 ceph-mon[32794]: pgmap v99: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 233 KiB/s rd, 25 KiB/s wr, 290 op/s 2024-09-13T14:29:35.373 INFO:tasks.workunit.client.0.smithi154.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-13T14:29:35.380 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap protect testimg@snap2 2024-09-13T14:29:35.426 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_78348/snap_protect 2024-09-13T14:29:35.470 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 33% complete... Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:35.473 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_78348/snap_protect testimg_import 2024-09-13T14:29:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:35 smithi154 ceph-mon[29001]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:29:35.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:35 smithi154 ceph-mon[39074]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:29:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:35 smithi172 ceph-mon[32794]: osdmap e78: 8 total, 8 up, 8 in 2024-09-13T14:29:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:36 smithi154 ceph-mon[29001]: pgmap v101: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 30 KiB/s wr, 370 op/s 2024-09-13T14:29:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:36 smithi154 ceph-mon[29001]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:29:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:36 smithi154 ceph-mon[39074]: pgmap v101: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 30 KiB/s wr, 370 op/s 2024-09-13T14:29:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:36 smithi154 ceph-mon[39074]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:29:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:36 smithi172 ceph-mon[32794]: pgmap v101: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 30 KiB/s wr, 370 op/s 2024-09-13T14:29:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:36 smithi172 ceph-mon[32794]: osdmap e79: 8 total, 8 up, 8 in 2024-09-13T14:29:37.396 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 33% complete... Importing image: 66% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-09-13T14:29:37.400 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd info testimg_import@snap1 2024-09-13T14:29:37.400 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep 'protected: False' 2024-09-13T14:29:37.440 INFO:tasks.workunit.client.0.smithi154.stdout: protected: False 2024-09-13T14:29:37.440 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd info testimg_import@snap2 2024-09-13T14:29:37.441 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep 'protected: True' 2024-09-13T14:29:37.485 INFO:tasks.workunit.client.0.smithi154.stdout: protected: True 2024-09-13T14:29:37.485 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/snap_protect 2024-09-13T14:29:37.485 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap unprotect testimg@snap2 2024-09-13T14:29:37.533 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap unprotect testimg_import@snap2 2024-09-13T14:29:37.581 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap purge testimg 2024-09-13T14:29:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:37 smithi154 ceph-mon[29001]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:29:37.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:37 smithi154 ceph-mon[39074]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:29:37.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:37 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:29:37] "GET /metrics HTTP/1.1" 200 33727 "" "Prometheus/2.43.0" 2024-09-13T14:29:37.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:37 smithi172 ceph-mon[32794]: osdmap e80: 8 total, 8 up, 8 in 2024-09-13T14:29:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:38 smithi154 ceph-mon[29001]: pgmap v104: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 30 KiB/s wr, 371 op/s 2024-09-13T14:29:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:38 smithi154 ceph-mon[29001]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:29:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:38 smithi154 ceph-mon[39074]: pgmap v104: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 30 KiB/s wr, 371 op/s 2024-09-13T14:29:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:38 smithi154 ceph-mon[39074]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:29:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:38 smithi172 ceph-mon[32794]: pgmap v104: 73 pgs: 73 active+clean; 799 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 287 KiB/s rd, 30 KiB/s wr, 371 op/s 2024-09-13T14:29:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:38 smithi172 ceph-mon[32794]: osdmap e81: 8 total, 8 up, 8 in 2024-09-13T14:29:39.162 INFO:tasks.workunit.client.0.smithi154.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T14:29:39.171 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd snap purge testimg_import 2024-09-13T14:29:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:39 smithi154 ceph-mon[29001]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:29:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:39 smithi154 ceph-mon[29001]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:29:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:39 smithi154 ceph-mon[29001]: pgmap v108: 73 pgs: 73 active+clean; 799 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 290 KiB/s rd, 32 KiB/s wr, 392 op/s 2024-09-13T14:29:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:39 smithi154 ceph-mon[39074]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:29:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:39 smithi154 ceph-mon[39074]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:29:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:39 smithi154 ceph-mon[39074]: pgmap v108: 73 pgs: 73 active+clean; 799 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 290 KiB/s rd, 32 KiB/s wr, 392 op/s 2024-09-13T14:29:39.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:39 smithi172 ceph-mon[32794]: osdmap e82: 8 total, 8 up, 8 in 2024-09-13T14:29:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:39 smithi172 ceph-mon[32794]: osdmap e83: 8 total, 8 up, 8 in 2024-09-13T14:29:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:39 smithi172 ceph-mon[32794]: pgmap v108: 73 pgs: 73 active+clean; 799 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 290 KiB/s rd, 32 KiB/s wr, 392 op/s 2024-09-13T14:29:41.168 INFO:tasks.workunit.client.0.smithi154.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-13T14:29:41.177 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg 2024-09-13T14:29:41.277 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:41.280 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd remove testimg_import 2024-09-13T14:29:41.370 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:41.374 INFO:tasks.workunit.client.0.smithi154.stderr:+ tiered=0 2024-09-13T14:29:41.374 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph osd dump 2024-09-13T14:29:41.374 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep '^pool' 2024-09-13T14:29:41.374 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep ''\''rbd'\''' 2024-09-13T14:29:41.374 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep tier 2024-09-13T14:29:41.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:41 smithi154 ceph-mon[29001]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:29:41.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:41 smithi154 ceph-mon[39074]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:29:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:41 smithi172 ceph-mon[32794]: osdmap e84: 8 total, 8 up, 8 in 2024-09-13T14:29:41.766 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/dev/urandom bs=1M seek=1 count=1 of=/tmp/rbd_import_export_78348/sparse1 2024-09-13T14:29:41.771 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records in 2024-09-13T14:29:41.771 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records out 2024-09-13T14:29:41.771 INFO:tasks.workunit.client.0.smithi154.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00339201 s, 309 MB/s 2024-09-13T14:29:41.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/dev/urandom bs=1M count=1 of=/tmp/rbd_import_export_78348/sparse2 2024-09-13T14:29:41.775 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records in 2024-09-13T14:29:41.775 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records out 2024-09-13T14:29:41.775 INFO:tasks.workunit.client.0.smithi154.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352642 s, 297 MB/s 2024-09-13T14:29:41.775 INFO:tasks.workunit.client.0.smithi154.stderr:+ truncate /tmp/rbd_import_export_78348/sparse2 -s 2M 2024-09-13T14:29:41.789 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse1 2024-09-13T14:29:41.831 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 0% complete...failed. 2024-09-13T14:29:41.832 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: delete error: (2) No such file or directory 2024-09-13T14:29:41.834 INFO:tasks.workunit.client.0.smithi154.stderr:+ true 2024-09-13T14:29:41.834 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_78348/sparse1 2024-09-13T14:29:41.901 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-09-13T14:29:41.902 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:41.902 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:29:41.905 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd ls -l 2024-09-13T14:29:41.905 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep sparse1 2024-09-13T14:29:41.905 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-09-13T14:29:41.944 INFO:tasks.workunit.client.0.smithi154.stdout:sparse1 2 MiB 2 2024-09-13T14:29:41.945 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects sparse1 2024-09-13T14:29:41.945 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse1 2024-09-13T14:29:41.945 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse1 2024-09-13T14:29:41.945 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep block_name_prefix 2024-09-13T14:29:41.945 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk '{print $NF;}' 2024-09-13T14:29:41.989 INFO:tasks.workunit.client.0.smithi154.stderr:++ prefix=rbd_data.3b4fcece7736 2024-09-13T14:29:41.992 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep rbd_data.3b4fcece7736 2024-09-13T14:29:41.992 INFO:tasks.workunit.client.0.smithi154.stderr:++++ get_image_data_pool sparse1 2024-09-13T14:29:41.992 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sed -e 's/rbd_data.3b4fcece7736\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:29:41.992 INFO:tasks.workunit.client.0.smithi154.stderr:++++ image=sparse1 2024-09-13T14:29:41.992 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sort -u 2024-09-13T14:29:41.993 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ rbd info sparse1 2024-09-13T14:29:41.993 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:29:41.993 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:29:42.032 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool= 2024-09-13T14:29:42.032 INFO:tasks.workunit.client.0.smithi154.stderr:++++ '[' -z ']' 2024-09-13T14:29:42.032 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool=rbd 2024-09-13T14:29:42.032 INFO:tasks.workunit.client.0.smithi154.stderr:++++ echo rbd 2024-09-13T14:29:42.032 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rados ls -p rbd 2024-09-13T14:29:42.070 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects=1 2024-09-13T14:29:42.071 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo 1 2024-09-13T14:29:42.071 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-13T14:29:42.071 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export sparse1 /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.114 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:42.117 INFO:tasks.workunit.client.0.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_78348/sparse1 /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.117 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_78348/sparse1 /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.119 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse1 --format %b 2024-09-13T14:29:42.120 INFO:tasks.workunit.client.0.smithi154.stderr:+ origsize=2048 2024-09-13T14:29:42.120 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse1.out --format %b 2024-09-13T14:29:42.121 INFO:tasks.workunit.client.0.smithi154.stderr:+ exportsize=2048 2024-09-13T14:29:42.121 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:42.121 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:42.121 INFO:tasks.workunit.client.0.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:29:42.121 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.122 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse1 2024-09-13T14:29:42.211 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-09-13T14:29:42.214 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse2 2024-09-13T14:29:42.254 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 0% complete...failed. 2024-09-13T14:29:42.254 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: delete error: (2) No such file or directory 2024-09-13T14:29:42.258 INFO:tasks.workunit.client.0.smithi154.stderr:+ true 2024-09-13T14:29:42.258 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_78348/sparse2 2024-09-13T14:29:42.325 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-09-13T14:29:42.325 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:42.325 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:29:42.329 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd ls -l 2024-09-13T14:29:42.329 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep sparse2 2024-09-13T14:29:42.329 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-09-13T14:29:42.371 INFO:tasks.workunit.client.0.smithi154.stdout:sparse2 2 MiB 2 2024-09-13T14:29:42.371 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects sparse2 2024-09-13T14:29:42.372 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse2 2024-09-13T14:29:42.372 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse2 2024-09-13T14:29:42.372 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep block_name_prefix 2024-09-13T14:29:42.372 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk '{print $NF;}' 2024-09-13T14:29:42.415 INFO:tasks.workunit.client.0.smithi154.stderr:++ prefix=rbd_data.622427d1c347 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep rbd_data.622427d1c347 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sed -e 's/rbd_data.622427d1c347\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:++++ get_image_data_pool sparse2 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sort -u 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:++++ image=sparse2 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ rbd info sparse2 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:29:42.417 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:29:42.456 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool= 2024-09-13T14:29:42.457 INFO:tasks.workunit.client.0.smithi154.stderr:++++ '[' -z ']' 2024-09-13T14:29:42.457 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool=rbd 2024-09-13T14:29:42.457 INFO:tasks.workunit.client.0.smithi154.stderr:++++ echo rbd 2024-09-13T14:29:42.457 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rados ls -p rbd 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[29001]: pgmap v110: 73 pgs: 73 active+clean; 799 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 275 KiB/s rd, 31 KiB/s wr, 372 op/s 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[29001]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/4041565453' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/4041565453' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[39074]: pgmap v110: 73 pgs: 73 active+clean; 799 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 275 KiB/s rd, 31 KiB/s wr, 372 op/s 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[39074]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/4041565453' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:29:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:42 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/4041565453' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:29:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:42 smithi172 ceph-mon[32794]: pgmap v110: 73 pgs: 73 active+clean; 799 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 275 KiB/s rd, 31 KiB/s wr, 372 op/s 2024-09-13T14:29:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:42 smithi172 ceph-mon[32794]: osdmap e85: 8 total, 8 up, 8 in 2024-09-13T14:29:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:42 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/4041565453' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:29:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:42 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/4041565453' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-13T14:29:42.498 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects=0 2024-09-13T14:29:42.498 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo 0 2024-09-13T14:29:42.498 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-09-13T14:29:42.498 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export sparse2 /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:42.538 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:42.541 INFO:tasks.workunit.client.0.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_78348/sparse2 /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:42.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_78348/sparse2 /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:42.543 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse2 --format %b 2024-09-13T14:29:42.544 INFO:tasks.workunit.client.0.smithi154.stderr:+ origsize=2048 2024-09-13T14:29:42.545 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse2.out --format %b 2024-09-13T14:29:42.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ exportsize=2048 2024-09-13T14:29:42.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:42.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:42.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:29:42.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:42.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse2 2024-09-13T14:29:42.634 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-09-13T14:29:42.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ truncate /tmp/rbd_import_export_78348/sparse1 -s 10M 2024-09-13T14:29:42.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format --order 20 - sparse1 2024-09-13T14:29:42.725 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:42.726 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:42.726 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:29:42.728 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd ls -l 2024-09-13T14:29:42.729 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep sparse1 2024-09-13T14:29:42.729 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep -Ei '(10 MiB|10240k)' 2024-09-13T14:29:42.768 INFO:tasks.workunit.client.0.smithi154.stdout:sparse1 10 MiB 2 2024-09-13T14:29:42.769 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects sparse1 2024-09-13T14:29:42.769 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse1 2024-09-13T14:29:42.770 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse1 2024-09-13T14:29:42.770 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep block_name_prefix 2024-09-13T14:29:42.770 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk '{print $NF;}' 2024-09-13T14:29:42.811 INFO:tasks.workunit.client.0.smithi154.stderr:++ prefix=rbd_data.6251a344d9e1 2024-09-13T14:29:42.812 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep rbd_data.6251a344d9e1 2024-09-13T14:29:42.812 INFO:tasks.workunit.client.0.smithi154.stderr:++++ get_image_data_pool sparse1 2024-09-13T14:29:42.812 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sed -e 's/rbd_data.6251a344d9e1\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:29:42.812 INFO:tasks.workunit.client.0.smithi154.stderr:++++ image=sparse1 2024-09-13T14:29:42.813 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sort -u 2024-09-13T14:29:42.813 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ rbd info sparse1 2024-09-13T14:29:42.813 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:29:42.813 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:29:42.852 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool= 2024-09-13T14:29:42.852 INFO:tasks.workunit.client.0.smithi154.stderr:++++ '[' -z ']' 2024-09-13T14:29:42.852 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool=rbd 2024-09-13T14:29:42.852 INFO:tasks.workunit.client.0.smithi154.stderr:++++ echo rbd 2024-09-13T14:29:42.852 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rados ls -p rbd 2024-09-13T14:29:42.890 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects=1 2024-09-13T14:29:42.890 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo 1 2024-09-13T14:29:42.890 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-13T14:29:42.890 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export sparse1 /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.932 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 10% complete... Exporting image: 20% complete... Exporting image: 30% complete... Exporting image: 40% complete... Exporting image: 50% complete... Exporting image: 60% complete... Exporting image: 70% complete... Exporting image: 80% complete... Exporting image: 90% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:42.936 INFO:tasks.workunit.client.0.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_78348/sparse1 /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.936 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_78348/sparse1 /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.942 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse1 --format %b 2024-09-13T14:29:42.943 INFO:tasks.workunit.client.0.smithi154.stderr:+ origsize=2048 2024-09-13T14:29:42.944 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse1.out --format %b 2024-09-13T14:29:42.945 INFO:tasks.workunit.client.0.smithi154.stderr:+ exportsize=2048 2024-09-13T14:29:42.945 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:42.945 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:42.945 INFO:tasks.workunit.client.0.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:29:42.945 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/sparse1.out 2024-09-13T14:29:42.946 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse1 2024-09-13T14:29:43.034 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-09-13T14:29:43.038 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/dev/urandom bs=2M count=1 of=/tmp/rbd_import_export_78348/sparse2 oflag=append conv=notrunc 2024-09-13T14:29:43.045 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records in 2024-09-13T14:29:43.045 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records out 2024-09-13T14:29:43.045 INFO:tasks.workunit.client.0.smithi154.stderr:2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00635523 s, 330 MB/s 2024-09-13T14:29:43.045 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format --order 20 - sparse2 2024-09-13T14:29:43.128 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:43.129 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:43.129 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:29:43.132 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd ls -l 2024-09-13T14:29:43.132 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep sparse2 2024-09-13T14:29:43.132 INFO:tasks.workunit.client.0.smithi154.stderr:+ grep -Ei '(4 MiB|4096k)' 2024-09-13T14:29:43.174 INFO:tasks.workunit.client.0.smithi154.stdout:sparse2 4 MiB 2 2024-09-13T14:29:43.175 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects sparse2 2024-09-13T14:29:43.175 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse2 2024-09-13T14:29:43.176 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse2 2024-09-13T14:29:43.176 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep block_name_prefix 2024-09-13T14:29:43.176 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk '{print $NF;}' 2024-09-13T14:29:43.220 INFO:tasks.workunit.client.0.smithi154.stderr:++ prefix=rbd_data.627e2c7bc358 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep rbd_data.627e2c7bc358 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:++++ get_image_data_pool sparse2 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sed -e 's/rbd_data.627e2c7bc358\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:++++ image=sparse2 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sort -u 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ rbd info sparse2 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:29:43.221 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:29:43.262 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool= 2024-09-13T14:29:43.262 INFO:tasks.workunit.client.0.smithi154.stderr:++++ '[' -z ']' 2024-09-13T14:29:43.262 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool=rbd 2024-09-13T14:29:43.262 INFO:tasks.workunit.client.0.smithi154.stderr:++++ echo rbd 2024-09-13T14:29:43.262 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rados ls -p rbd 2024-09-13T14:29:43.301 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects='0 2024-09-13T14:29:43.301 INFO:tasks.workunit.client.0.smithi154.stderr:2 2024-09-13T14:29:43.301 INFO:tasks.workunit.client.0.smithi154.stderr:3' 2024-09-13T14:29:43.301 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo 0 2 3 2024-09-13T14:29:43.301 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 -eq 1 -o '0 2 3' = '0 2 3' ']' 2024-09-13T14:29:43.301 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export sparse2 /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:43.341 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 25% complete... Exporting image: 50% complete... Exporting image: 75% complete... Exporting image: 100% complete...done. 2024-09-13T14:29:43.345 INFO:tasks.workunit.client.0.smithi154.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_78348/sparse2 /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:43.345 INFO:tasks.workunit.client.0.smithi154.stderr:+ cmp -l /tmp/rbd_import_export_78348/sparse2 /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:43.348 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse2 --format %b 2024-09-13T14:29:43.349 INFO:tasks.workunit.client.0.smithi154.stderr:+ origsize=6144 2024-09-13T14:29:43.349 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse2.out --format %b 2024-09-13T14:29:43.351 INFO:tasks.workunit.client.0.smithi154.stderr:+ exportsize=6144 2024-09-13T14:29:43.351 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:43.351 INFO:tasks.workunit.client.0.smithi154.stderr:+ difference=0 2024-09-13T14:29:43.351 INFO:tasks.workunit.client.0.smithi154.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-09-13T14:29:43.351 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/sparse2.out 2024-09-13T14:29:43.352 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse2 2024-09-13T14:29:43.443 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 25% complete... Removing image: 50% complete... Removing image: 75% complete... Removing image: 100% complete...done. 2024-09-13T14:29:43.446 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'partially-sparse file imports to partially-sparse image' 2024-09-13T14:29:43.447 INFO:tasks.workunit.client.0.smithi154.stdout:partially-sparse file imports to partially-sparse image 2024-09-13T14:29:43.447 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_78348/sparse1 sparse 2024-09-13T14:29:43.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:43 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:43.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:43 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:43.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:43 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:43.517 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 10% complete... Importing image: 20% complete... Importing image: 30% complete... Importing image: 40% complete... Importing image: 50% complete... Importing image: 60% complete... Importing image: 70% complete... Importing image: 80% complete... Importing image: 90% complete... Importing image: 100% complete...done. 2024-09-13T14:29:43.518 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:43.518 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --order is deprecated, use --object-size 2024-09-13T14:29:43.521 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects sparse 2024-09-13T14:29:43.521 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse 2024-09-13T14:29:43.522 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse 2024-09-13T14:29:43.522 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep block_name_prefix 2024-09-13T14:29:43.522 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk '{print $NF;}' 2024-09-13T14:29:43.562 INFO:tasks.workunit.client.0.smithi154.stderr:++ prefix=rbd_data.3c0996be87fc 2024-09-13T14:29:43.563 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep rbd_data.3c0996be87fc 2024-09-13T14:29:43.563 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sed -e 's/rbd_data.3c0996be87fc\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:29:43.563 INFO:tasks.workunit.client.0.smithi154.stderr:++++ get_image_data_pool sparse 2024-09-13T14:29:43.563 INFO:tasks.workunit.client.0.smithi154.stderr:++++ image=sparse 2024-09-13T14:29:43.564 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sort -u 2024-09-13T14:29:43.564 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ rbd info sparse 2024-09-13T14:29:43.564 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:29:43.564 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:29:43.607 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool= 2024-09-13T14:29:43.608 INFO:tasks.workunit.client.0.smithi154.stderr:++++ '[' -z ']' 2024-09-13T14:29:43.608 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool=rbd 2024-09-13T14:29:43.608 INFO:tasks.workunit.client.0.smithi154.stderr:++++ echo rbd 2024-09-13T14:29:43.608 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rados ls -p rbd 2024-09-13T14:29:43.644 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects=1 2024-09-13T14:29:43.645 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo 1 2024-09-13T14:29:43.645 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-09-13T14:29:43.645 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse 2024-09-13T14:29:43.735 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-09-13T14:29:43.738 INFO:tasks.workunit.client.0.smithi154.stdout:zeros import through stdin to sparse image 2024-09-13T14:29:43.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'zeros import through stdin to sparse image' 2024-09-13T14:29:43.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/dev/zero bs=1M count=4 2024-09-13T14:29:43.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd import --new-format - sparse 2024-09-13T14:29:43.793 INFO:tasks.workunit.client.0.smithi154.stderr:4+0 records in 2024-09-13T14:29:43.793 INFO:tasks.workunit.client.0.smithi154.stderr:4+0 records out 2024-09-13T14:29:43.794 INFO:tasks.workunit.client.0.smithi154.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0545623 s, 76.9 MB/s 2024-09-13T14:29:43.804 INFO:tasks.workunit.client.0.smithi154.stderr: Importing image: 100% complete...done. 2024-09-13T14:29:43.804 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:43.808 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects sparse 2024-09-13T14:29:43.808 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse 2024-09-13T14:29:43.808 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse 2024-09-13T14:29:43.808 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep block_name_prefix 2024-09-13T14:29:43.808 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk '{print $NF;}' 2024-09-13T14:29:43.846 INFO:tasks.workunit.client.0.smithi154.stderr:++ prefix=rbd_data.62ccbe5d5d99 2024-09-13T14:29:43.847 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep rbd_data.62ccbe5d5d99 2024-09-13T14:29:43.847 INFO:tasks.workunit.client.0.smithi154.stderr:++++ get_image_data_pool sparse 2024-09-13T14:29:43.847 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sed -e 's/rbd_data.62ccbe5d5d99\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:29:43.847 INFO:tasks.workunit.client.0.smithi154.stderr:++++ image=sparse 2024-09-13T14:29:43.847 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sort -u 2024-09-13T14:29:43.848 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ rbd info sparse 2024-09-13T14:29:43.848 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:29:43.848 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:29:43.890 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool= 2024-09-13T14:29:43.890 INFO:tasks.workunit.client.0.smithi154.stderr:++++ '[' -z ']' 2024-09-13T14:29:43.890 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool=rbd 2024-09-13T14:29:43.890 INFO:tasks.workunit.client.0.smithi154.stderr:++++ echo rbd 2024-09-13T14:29:43.890 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rados ls -p rbd 2024-09-13T14:29:43.927 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects= 2024-09-13T14:29:43.927 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo 2024-09-13T14:29:43.927 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 -eq 1 -o '' = '' ']' 2024-09-13T14:29:43.927 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse 2024-09-13T14:29:44.016 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:44.018 INFO:tasks.workunit.client.0.smithi154.stdout:zeros export to sparse file 2024-09-13T14:29:44.019 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'zeros export to sparse file' 2024-09-13T14:29:44.019 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd create --new-format sparse --size 4 2024-09-13T14:29:44.061 INFO:tasks.workunit.client.0.smithi154.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-09-13T14:29:44.064 INFO:tasks.workunit.client.0.smithi154.stderr:++ rbd info sparse 2024-09-13T14:29:44.065 INFO:tasks.workunit.client.0.smithi154.stderr:++ grep block_name_prefix 2024-09-13T14:29:44.065 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $NF;}' 2024-09-13T14:29:44.107 INFO:tasks.workunit.client.0.smithi154.stderr:+ prefix=rbd_data.3c4b8013f407 2024-09-13T14:29:44.107 INFO:tasks.workunit.client.0.smithi154.stderr:+ dd if=/dev/zero bs=4M count=1 2024-09-13T14:29:44.108 INFO:tasks.workunit.client.0.smithi154.stderr:++ get_image_data_pool sparse 2024-09-13T14:29:44.108 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse 2024-09-13T14:29:44.108 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse 2024-09-13T14:29:44.108 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep 'data_pool: ' 2024-09-13T14:29:44.108 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk -F: '{ print $NF }' 2024-09-13T14:29:44.149 INFO:tasks.workunit.client.0.smithi154.stderr:++ data_pool= 2024-09-13T14:29:44.149 INFO:tasks.workunit.client.0.smithi154.stderr:++ '[' -z ']' 2024-09-13T14:29:44.149 INFO:tasks.workunit.client.0.smithi154.stderr:++ data_pool=rbd 2024-09-13T14:29:44.149 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo rbd 2024-09-13T14:29:44.149 INFO:tasks.workunit.client.0.smithi154.stderr:+ rados -p rbd put rbd_data.3c4b8013f407.000000000000 - 2024-09-13T14:29:44.189 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records in 2024-09-13T14:29:44.189 INFO:tasks.workunit.client.0.smithi154.stderr:1+0 records out 2024-09-13T14:29:44.189 INFO:tasks.workunit.client.0.smithi154.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0815529 s, 51.4 MB/s 2024-09-13T14:29:44.210 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects sparse 2024-09-13T14:29:44.210 INFO:tasks.workunit.client.0.smithi154.stderr:++ image=sparse 2024-09-13T14:29:44.210 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rbd info sparse 2024-09-13T14:29:44.210 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep block_name_prefix 2024-09-13T14:29:44.210 INFO:tasks.workunit.client.0.smithi154.stderr:+++ awk '{print $NF;}' 2024-09-13T14:29:44.250 INFO:tasks.workunit.client.0.smithi154.stderr:++ prefix=rbd_data.3c4b8013f407 2024-09-13T14:29:44.251 INFO:tasks.workunit.client.0.smithi154.stderr:+++ grep rbd_data.3c4b8013f407 2024-09-13T14:29:44.251 INFO:tasks.workunit.client.0.smithi154.stderr:++++ get_image_data_pool sparse 2024-09-13T14:29:44.251 INFO:tasks.workunit.client.0.smithi154.stderr:++++ image=sparse 2024-09-13T14:29:44.251 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sed -e 's/rbd_data.3c4b8013f407\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-09-13T14:29:44.251 INFO:tasks.workunit.client.0.smithi154.stderr:+++ sort -u 2024-09-13T14:29:44.252 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ rbd info sparse 2024-09-13T14:29:44.252 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ grep 'data_pool: ' 2024-09-13T14:29:44.252 INFO:tasks.workunit.client.0.smithi154.stderr:+++++ awk -F: '{ print $NF }' 2024-09-13T14:29:44.296 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool= 2024-09-13T14:29:44.296 INFO:tasks.workunit.client.0.smithi154.stderr:++++ '[' -z ']' 2024-09-13T14:29:44.296 INFO:tasks.workunit.client.0.smithi154.stderr:++++ data_pool=rbd 2024-09-13T14:29:44.296 INFO:tasks.workunit.client.0.smithi154.stderr:++++ echo rbd 2024-09-13T14:29:44.296 INFO:tasks.workunit.client.0.smithi154.stderr:+++ rados ls -p rbd 2024-09-13T14:29:44.333 INFO:tasks.workunit.client.0.smithi154.stderr:++ objects=0 2024-09-13T14:29:44.333 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo 0 2024-09-13T14:29:44.333 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-09-13T14:29:44.333 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/sparse 2024-09-13T14:29:44.334 INFO:tasks.workunit.client.0.smithi154.stderr:rm: cannot remove '/tmp/rbd_import_export_78348/sparse': No such file or directory 2024-09-13T14:29:44.334 INFO:tasks.workunit.client.0.smithi154.stderr:+ true 2024-09-13T14:29:44.334 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd export sparse /tmp/rbd_import_export_78348/sparse 2024-09-13T14:29:44.376 INFO:tasks.workunit.client.0.smithi154.stderr: Exporting image: 100% complete...done. 2024-09-13T14:29:44.379 INFO:tasks.workunit.client.0.smithi154.stderr:++ stat /tmp/rbd_import_export_78348/sparse --format=%b 2024-09-13T14:29:44.381 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 0 = 0 ']' 2024-09-13T14:29:44.381 INFO:tasks.workunit.client.0.smithi154.stderr:+ rbd rm sparse 2024-09-13T14:29:44.465 INFO:tasks.workunit.client.0.smithi154.stderr: Removing image: 100% complete...done. 2024-09-13T14:29:44.468 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm /tmp/rbd_import_export_78348/sparse /tmp/rbd_import_export_78348/sparse1 /tmp/rbd_import_export_78348/sparse2 /tmp/rbd_import_export_78348/sparse3 2024-09-13T14:29:44.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:44 smithi154 ceph-mon[29001]: pgmap v112: 73 pgs: 73 active+clean; 3.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 890 KiB/s rd, 1.3 MiB/s wr, 331 op/s 2024-09-13T14:29:44.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:44 smithi154 ceph-mon[39074]: pgmap v112: 73 pgs: 73 active+clean; 3.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 890 KiB/s rd, 1.3 MiB/s wr, 331 op/s 2024-09-13T14:29:44.470 INFO:tasks.workunit.client.0.smithi154.stderr:rm: cannot remove '/tmp/rbd_import_export_78348/sparse3': No such file or directory 2024-09-13T14:29:44.470 INFO:tasks.workunit.client.0.smithi154.stderr:+ true 2024-09-13T14:29:44.471 INFO:tasks.workunit.client.0.smithi154.stdout:OK 2024-09-13T14:29:44.472 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo OK 2024-09-13T14:29:44.472 INFO:tasks.workunit.client.0.smithi154.stderr:+ rm -rf /tmp/rbd_import_export_78348 2024-09-13T14:29:44.472 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T14:29:44.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T14:29:44.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:44 smithi172 ceph-mon[32794]: pgmap v112: 73 pgs: 73 active+clean; 3.6 MiB data, 248 MiB used, 715 GiB / 715 GiB avail; 890 KiB/s rd, 1.3 MiB/s wr, 331 op/s 2024-09-13T14:29:44.500 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.0... 2024-09-13T14:29:44.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T14:29:44.748 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:44.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T14:29:44.774 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T14:29:44.774 DEBUG:teuthology.orchestra.run.smithi154:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T14:29:44.830 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T14:29:44.830 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T14:29:44.830 INFO:teuthology.task.print:**** done rbd/import_export.sh 5-workload 2024-09-13T14:29:44.830 DEBUG:teuthology.parallel:result is None 2024-09-13T14:29:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:46 smithi154 ceph-mon[29001]: pgmap v113: 73 pgs: 73 active+clean; 3.6 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 719 KiB/s rd, 1.0 MiB/s wr, 280 op/s 2024-09-13T14:29:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:46 smithi154 ceph-mon[39074]: pgmap v113: 73 pgs: 73 active+clean; 3.6 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 719 KiB/s rd, 1.0 MiB/s wr, 280 op/s 2024-09-13T14:29:46.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:46 smithi172 ceph-mon[32794]: pgmap v113: 73 pgs: 73 active+clean; 3.6 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 719 KiB/s rd, 1.0 MiB/s wr, 280 op/s 2024-09-13T14:29:47.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:47 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:29:47] "GET /metrics HTTP/1.1" 200 33747 "" "Prometheus/2.43.0" 2024-09-13T14:29:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:48 smithi154 ceph-mon[29001]: pgmap v114: 73 pgs: 73 active+clean; 3.6 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 540 KiB/s rd, 779 KiB/s wr, 210 op/s 2024-09-13T14:29:48.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:48 smithi154 ceph-mon[39074]: pgmap v114: 73 pgs: 73 active+clean; 3.6 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 540 KiB/s rd, 779 KiB/s wr, 210 op/s 2024-09-13T14:29:48.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:48 smithi172 ceph-mon[32794]: pgmap v114: 73 pgs: 73 active+clean; 3.6 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 540 KiB/s rd, 779 KiB/s wr, 210 op/s 2024-09-13T14:29:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:50 smithi154 ceph-mon[39074]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:29:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:50 smithi154 ceph-mon[39074]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 344 op/s 2024-09-13T14:29:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:50 smithi154 ceph-mon[29001]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:29:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:50 smithi154 ceph-mon[29001]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 344 op/s 2024-09-13T14:29:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:50 smithi172 ceph-mon[32794]: osdmap e86: 8 total, 8 up, 8 in 2024-09-13T14:29:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:50 smithi172 ceph-mon[32794]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s rd, 1.4 MiB/s wr, 344 op/s 2024-09-13T14:29:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:52 smithi154 ceph-mon[29001]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 820 KiB/s rd, 1.1 MiB/s wr, 275 op/s 2024-09-13T14:29:52.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:52 smithi154 ceph-mon[39074]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 820 KiB/s rd, 1.1 MiB/s wr, 275 op/s 2024-09-13T14:29:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:52 smithi172 ceph-mon[32794]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 820 KiB/s rd, 1.1 MiB/s wr, 275 op/s 2024-09-13T14:29:54.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:54 smithi154 ceph-mon[29001]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 394 KiB/s rd, 516 KiB/s wr, 117 op/s 2024-09-13T14:29:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:54 smithi154 ceph-mon[39074]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 394 KiB/s rd, 516 KiB/s wr, 117 op/s 2024-09-13T14:29:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:54 smithi172 ceph-mon[32794]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 394 KiB/s rd, 516 KiB/s wr, 117 op/s 2024-09-13T14:29:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:56 smithi154 ceph-mon[29001]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 388 KiB/s rd, 515 KiB/s wr, 107 op/s 2024-09-13T14:29:56.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:56 smithi154 ceph-mon[39074]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 388 KiB/s rd, 515 KiB/s wr, 107 op/s 2024-09-13T14:29:56.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:56 smithi172 ceph-mon[32794]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 388 KiB/s rd, 515 KiB/s wr, 107 op/s 2024-09-13T14:29:57.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:29:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:29:57] "GET /metrics HTTP/1.1" 200 33754 "" "Prometheus/2.43.0" 2024-09-13T14:29:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:58 smithi154 ceph-mon[29001]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 388 KiB/s rd, 515 KiB/s wr, 107 op/s 2024-09-13T14:29:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:29:58 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:58 smithi154 ceph-mon[39074]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 388 KiB/s rd, 515 KiB/s wr, 107 op/s 2024-09-13T14:29:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:29:58 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:29:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:58 smithi172 ceph-mon[32794]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 388 KiB/s rd, 515 KiB/s wr, 107 op/s 2024-09-13T14:29:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:29:58 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:00 smithi154 ceph-mon[29001]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:00 smithi154 ceph-mon[29001]: overall HEALTH_OK 2024-09-13T14:30:00.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:00 smithi154 ceph-mon[39074]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:00 smithi154 ceph-mon[39074]: overall HEALTH_OK 2024-09-13T14:30:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:00 smithi172 ceph-mon[32794]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:00 smithi172 ceph-mon[32794]: overall HEALTH_OK 2024-09-13T14:30:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:02 smithi154 ceph-mon[29001]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:02.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:02 smithi154 ceph-mon[39074]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:02 smithi172 ceph-mon[32794]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:04 smithi154 ceph-mon[29001]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:04 smithi154 ceph-mon[39074]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:04.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:04 smithi172 ceph-mon[32794]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:05.018 INFO:teuthology.orchestra.run.smithi154.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:30:05.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:30:05.987 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:30:06.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[29001]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:30:06.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:06.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:06.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[29001]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:06.360 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[39074]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:30:06.360 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:06.360 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:06.360 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:06 smithi154 ceph-mon[39074]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:06.437 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:06 smithi172 ceph-mon[32794]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:30:06.437 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:06 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:06.437 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:06 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:06.437 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:06 smithi172 ceph-mon[32794]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:07.706 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:30:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:30:07] "GET /metrics HTTP/1.1" 200 33754 "" "Prometheus/2.43.0" 2024-09-13T14:30:08.432 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:30:08.434 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:08 smithi154 ceph-mon[29001]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:08.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:08 smithi154 ceph-mon[39074]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:08.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:08 smithi172 ceph-mon[32794]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:08.976 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:30:08.976 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (2m) 82s ago 2m 16.3M - 0.25.0 c8568f914cd2 d46f4205cce4 2024-09-13T14:30:08.976 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (2m) 0s ago 2m 86.6M - 9.4.7 954c08fa6188 9507837610ff 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (107s) 82s ago 106s 14.3M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (104s) 0s ago 103s 14.8M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,8765 running (6m) 0s ago 6m 439M - 18.2.4-843-g2c01458a c129f5a2fd4e 8b093da69eb8 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:9283,8765 running (8m) 82s ago 8m 482M - 18.2.4-843-g2c01458a c129f5a2fd4e 7d0a3081e77d 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (8m) 82s ago 8m 44.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (7m) 0s ago 7m 42.0M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (7m) 82s ago 7m 36.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (2m) 82s ago 2m 8032k - 1.5.0 0da6a335fe13 93d3a2d692ca 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (2m) 0s ago 2m 7675k - 1.5.0 0da6a335fe13 6e12b9645ca4 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (6m) 82s ago 6m 46.7M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (6m) 82s ago 6m 53.3M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (5m) 82s ago 5m 48.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:30:08.977 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (5m) 82s ago 5m 47.9M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:30:08.978 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (4m) 0s ago 4m 54.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:30:08.978 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (4m) 0s ago 4m 53.9M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:30:08.978 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (4m) 0s ago 4m 52.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:30:08.978 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (3m) 0s ago 3m 52.2M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:30:08.978 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (116s) 0s ago 3m 45.0M - 2.43.0 a07b618ecd1d 33387ab51e88 2024-09-13T14:30:09.309 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:30:09.310 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:30:09.310 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:30:09.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:09.310 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:30:09.310 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 15 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:30:09.311 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:30:09.592 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:30:09.592 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", 2024-09-13T14:30:09.592 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:30:09.592 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:30:09.592 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:30:09.592 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "", 2024-09-13T14:30:09.592 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "", 2024-09-13T14:30:09.593 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:30:09.593 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:30:09.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[29001]: from='client.25385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:09.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:09.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[29001]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.594 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[29001]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.594 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[29001]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:09.594 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1359413833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:30:09.594 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[39074]: from='client.25385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.594 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:09.594 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:09.595 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[39074]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.595 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[39074]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.595 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[39074]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:09.595 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:09 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1359413833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:30:09.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:09 smithi172 ceph-mon[32794]: from='client.25385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:09 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:09 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:09 smithi172 ceph-mon[32794]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:09 smithi172 ceph-mon[32794]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:09 smithi172 ceph-mon[32794]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:09 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1359413833' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:30:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[29001]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:30:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:30:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[29001]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:30:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[39074]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:30:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:30:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:10 smithi154 ceph-mon[39074]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:30:11.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:10 smithi172 ceph-mon[32794]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:10 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:10 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:10 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:30:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:10 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:30:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:10 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:10 smithi172 ceph-mon[32794]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:30:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[29001]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:12.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[39074]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:12.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:12.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:12.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:11 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:12.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:11 smithi172 ceph-mon[32794]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:12.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:11 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:12.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:11 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:12.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:11 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[29001]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:30:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[29001]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:30:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[29001]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:30:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[29001]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi172 2024-09-13T14:30:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[39074]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:30:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[39074]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:30:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[39074]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:30:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[39074]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi172 2024-09-13T14:30:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:12 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:12 smithi172 ceph-mon[32794]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (unknown) 2024-09-13T14:30:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:12 smithi172 ceph-mon[32794]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:30:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:12 smithi172 ceph-mon[32794]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-13T14:30:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:12 smithi172 ceph-mon[32794]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f on smithi172 2024-09-13T14:30:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:12 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:13 smithi154 ceph-mon[29001]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:13 smithi154 ceph-mon[39074]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:13 smithi172 ceph-mon[32794]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:16.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:16 smithi172 ceph-mon[32794]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:16 smithi154 ceph-mon[29001]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:16.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:16 smithi154 ceph-mon[39074]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:17.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:30:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:30:17] "GET /metrics HTTP/1.1" 200 33751 "" "Prometheus/2.43.0" 2024-09-13T14:30:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:18 smithi172 ceph-mon[32794]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:18.621 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:18 smithi154 ceph-mon[29001]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:18.622 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:18 smithi154 ceph-mon[39074]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:20 smithi172 ceph-mon[32794]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:20 smithi154 ceph-mon[29001]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:20.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:20 smithi154 ceph-mon[39074]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:22.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:22 smithi172 ceph-mon[32794]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:22 smithi154 ceph-mon[29001]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:22.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:22 smithi154 ceph-mon[39074]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:24 smithi172 ceph-mon[32794]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:24.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:24 smithi154 ceph-mon[29001]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:24.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:24 smithi154 ceph-mon[39074]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:26 smithi172 ceph-mon[32794]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:26 smithi154 ceph-mon[29001]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:26.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:26 smithi154 ceph-mon[39074]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:27.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:27 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:27.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:30:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:30:27] "GET /metrics HTTP/1.1" 200 33748 "" "Prometheus/2.43.0" 2024-09-13T14:30:27.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:27 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:27.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:27 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:28 smithi154 ceph-mon[29001]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:28 smithi154 ceph-mon[39074]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:28.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:28 smithi172 ceph-mon[32794]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:30 smithi154 ceph-mon[29001]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:30 smithi154 ceph-mon[39074]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:30.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:30 smithi172 ceph-mon[32794]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:32.541 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:32 smithi172 ceph-mon[32794]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:32 smithi154 ceph-mon[29001]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:32 smithi154 ceph-mon[39074]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:34 smithi154 ceph-mon[29001]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:34.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:34 smithi154 ceph-mon[39074]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:34 smithi172 ceph-mon[32794]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:36 smithi154 ceph-mon[29001]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:36 smithi154 ceph-mon[39074]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:36 smithi172 ceph-mon[32794]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:37.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:30:37 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:30:37] "GET /metrics HTTP/1.1" 200 33748 "" "Prometheus/2.43.0" 2024-09-13T14:30:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:38 smithi154 ceph-mon[29001]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:38 smithi154 ceph-mon[39074]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:38 smithi172 ceph-mon[32794]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:39.892 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:30:40.416 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:30:40.416 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (2m) 30s ago 3m 28.3M - 0.25.0 c8568f914cd2 d46f4205cce4 2024-09-13T14:30:40.416 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (3m) 31s ago 3m 86.6M - 9.4.7 954c08fa6188 9507837610ff 2024-09-13T14:30:40.416 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (2m) 30s ago 2m 15.1M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:30:40.417 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (2m) 31s ago 2m 14.8M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:30:40.417 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,8765 running (7m) 31s ago 7m 439M - 18.2.4-843-g2c01458a c129f5a2fd4e 8b093da69eb8 2024-09-13T14:30:40.417 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:9283,8765 running (8m) 30s ago 8m 488M - 18.2.4-843-g2c01458a c129f5a2fd4e 7d0a3081e77d 2024-09-13T14:30:40.417 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (8m) 30s ago 9m 47.5M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:30:40.417 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (7m) 31s ago 7m 42.0M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:30:40.417 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (7m) 30s ago 7m 39.0M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:30:40.417 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (2m) 30s ago 2m 15.7M - 1.5.0 0da6a335fe13 93d3a2d692ca 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (2m) 31s ago 2m 7675k - 1.5.0 0da6a335fe13 6e12b9645ca4 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (7m) 30s ago 7m 52.6M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (6m) 30s ago 6m 59.5M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (6m) 30s ago 6m 55.6M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (5m) 30s ago 5m 51.0M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (5m) 31s ago 5m 54.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (5m) 31s ago 5m 53.9M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (4m) 31s ago 4m 52.8M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (4m) 31s ago 4m 52.2M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:30:40.418 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (2m) 31s ago 3m 45.0M - 2.43.0 a07b618ecd1d 33387ab51e88 2024-09-13T14:30:40.529 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:40 smithi172 ceph-mon[32794]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:40.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:40 smithi154 ceph-mon[29001]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:40.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:40 smithi154 ceph-mon[39074]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:40.732 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:30:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:30:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 15 2024-09-13T14:30:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:30:40.734 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:30:41.005 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:30:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:30:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:30:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:30:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:30:41.005 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "0/2 daemons upgraded", 2024-09-13T14:30:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f image on host smithi172", 2024-09-13T14:30:41.006 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:30:41.006 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:30:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:41 smithi172 ceph-mon[32794]: from='client.25415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:41 smithi172 ceph-mon[32794]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:41 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/2070052414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:30:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:41 smithi154 ceph-mon[29001]: from='client.25415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:41 smithi154 ceph-mon[29001]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:41 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/2070052414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:30:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:41 smithi154 ceph-mon[39074]: from='client.25415 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:41 smithi154 ceph-mon[39074]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:41 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/2070052414' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:30:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:42 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:30:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:30:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:30:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:42 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:43.491 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:43 smithi172 systemd[1]: Stopping Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:30:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:43 smithi172 ceph-mon[32794]: Upgrade: Updating mgr.x 2024-09-13T14:30:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:43 smithi172 ceph-mon[32794]: Deploying daemon mgr.x on smithi172 2024-09-13T14:30:43.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:43 smithi154 ceph-mon[39074]: Upgrade: Updating mgr.x 2024-09-13T14:30:43.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:43 smithi154 ceph-mon[39074]: Deploying daemon mgr.x on smithi172 2024-09-13T14:30:43.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:43 smithi154 ceph-mon[29001]: Upgrade: Updating mgr.x 2024-09-13T14:30:43.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:43 smithi154 ceph-mon[29001]: Deploying daemon mgr.x on smithi172 2024-09-13T14:30:43.721 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:30:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:30:43.523Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": read tcp 172.21.15.154:33252->172.21.15.172:8443: read: connection reset by peer" 2024-09-13T14:30:43.827 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:43 smithi172 podman[66333]: 2024-09-13 14:30:43.532884191 +0000 UTC m=+0.151171835 container died 8b093da69eb833ffe48af01ab9c64579bfab8d6e2d95c482a1fb0dd56a295ef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-2c01458, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-13T14:30:44.220 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:30:43 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:30:43.820Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-09-13T14:30:44.240 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:43 smithi172 podman[66333]: 2024-09-13 14:30:43.927102038 +0000 UTC m=+0.545389681 container cleanup 8b093da69eb833ffe48af01ab9c64579bfab8d6e2d95c482a1fb0dd56a295ef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T14:30:44.240 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:43 smithi172 bash[66333]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:30:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:44 smithi172 ceph-mon[32794]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:44.588 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:44 smithi172 podman[66346]: 2024-09-13 14:30:44.420890162 +0000 UTC m=+0.891150124 container remove 8b093da69eb833ffe48af01ab9c64579bfab8d6e2d95c482a1fb0dd56a295ef8 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T14:30:44.588 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:44 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:30:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:44 smithi154 ceph-mon[39074]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:44 smithi154 ceph-mon[29001]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:44.987 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:44 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:30:44.987 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:44 smithi172 systemd[1]: Stopped Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:30:44.987 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:44 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service: Consumed 10.953s CPU time. 2024-09-13T14:30:44.987 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:44 smithi172 systemd[1]: Starting Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:30:45.240 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:45 smithi172 podman[66475]: 2024-09-13 14:30:44.91145077 +0000 UTC m=+0.016135679 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:30:45.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:45 smithi172 podman[66475]: 2024-09-13 14:30:45.080346224 +0000 UTC m=+0.185031118 container create e47bd4e01c6ce7d7045c0f71776ddccd3fe08970332e3e9134affcc285a03ed8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:30:45.740 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:45 smithi172 podman[66475]: 2024-09-13 14:30:45.316256077 +0000 UTC m=+0.420940984 container init e47bd4e01c6ce7d7045c0f71776ddccd3fe08970332e3e9134affcc285a03ed8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-13T14:30:45.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:45 smithi172 podman[66475]: 2024-09-13 14:30:45.31910902 +0000 UTC m=+0.423793923 container start e47bd4e01c6ce7d7045c0f71776ddccd3fe08970332e3e9134affcc285a03ed8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T14:30:45.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:45 smithi172 ceph-mgr[66496]: -- 172.21.15.172:0/786690858 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56283dcf10e0 con 0x56283dcce800 2024-09-13T14:30:45.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:45 smithi172 bash[66475]: e47bd4e01c6ce7d7045c0f71776ddccd3fe08970332e3e9134affcc285a03ed8 2024-09-13T14:30:45.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:45 smithi172 systemd[1]: Started Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:30:46.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:46 smithi172 ceph-mon[32794]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:46.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:46 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:46.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:46 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:46.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:46 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[29001]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:46.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:46.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[39074]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:46 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:30:47.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:30:47 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:30:47] "GET /metrics HTTP/1.1" 200 33748 "" "Prometheus/2.43.0" 2024-09-13T14:30:48.621 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:48 smithi154 ceph-mon[29001]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:48.622 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:48 smithi154 ceph-mon[39074]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:48 smithi172 ceph-mon[32794]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:50.684 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-mon[32794]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:50.685 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.685 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.685 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.685 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[29001]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[39074]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:50.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:50 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:30:50.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:30:50] ENGINE Bus STARTING 2024-09-13T14:30:50.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: CherryPy Checker: 2024-09-13T14:30:50.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: The Application mounted at '' has an empty config. 2024-09-13T14:30:50.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:30:50.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:30:50] ENGINE Serving on http://:::9283 2024-09-13T14:30:50.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:30:50 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:30:50] ENGINE Bus STARTED 2024-09-13T14:30:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[29001]: Standby manager daemon x restarted 2024-09-13T14:30:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[29001]: Standby manager daemon x started 2024-09-13T14:30:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:30:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[39074]: Standby manager daemon x restarted 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[39074]: Standby manager daemon x started 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:30:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:51 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:30:51.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:51 smithi172 ceph-mon[32794]: Standby manager daemon x restarted 2024-09-13T14:30:51.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:51 smithi172 ceph-mon[32794]: Standby manager daemon x started 2024-09-13T14:30:51.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:51 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:30:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:51 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:30:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:51 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:30:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:51 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:30:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:52 smithi154 ceph-mon[29001]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:52 smithi154 ceph-mon[29001]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:30:52.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:52 smithi154 ceph-mon[39074]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:52 smithi154 ceph-mon[39074]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:30:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:52 smithi172 ceph-mon[32794]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:52 smithi172 ceph-mon[32794]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-13T14:30:53.970 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:30:53 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:30:53.508Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-09-13T14:30:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:54 smithi172 ceph-mon[32794]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:54.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:54 smithi154 ceph-mon[39074]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:54.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:54 smithi154 ceph-mon[29001]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:56 smithi154 ceph-mon[29001]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:56.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:56 smithi154 ceph-mon[39074]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:56.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:56 smithi172 ceph-mon[32794]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:57 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:57.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:57 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:57.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:30:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:30:57] "GET /metrics HTTP/1.1" 200 33750 "" "Prometheus/2.43.0" 2024-09-13T14:30:57.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:57 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:30:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:30:58 smithi154 ceph-mon[29001]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:58.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:30:58 smithi154 ceph-mon[39074]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:30:58.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:30:58 smithi172 ceph-mon[32794]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:00.625 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:00 smithi172 ceph-mon[32794]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:00 smithi154 ceph-mon[29001]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:00.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:00 smithi154 ceph-mon[39074]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:02.618 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:02 smithi172 ceph-mon[32794]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:02 smithi154 ceph-mon[29001]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:02.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:02 smithi154 ceph-mon[39074]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:04.490 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:04] ENGINE Bus STOPPING 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:04] ENGINE Bus STOPPED 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: Activating manager daemon x 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: mgrmap e22: x(active, starting, since 0.00675701s) 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:31:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:31:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:04 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:31:04.633 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:04.285+0000 7f7c85a4a640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-13T14:31:04.633 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setuser ceph since I am not root 2024-09-13T14:31:04.633 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ignoring --setgroup ceph since I am not root 2024-09-13T14:31:04.633 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:04.427+0000 7f6780c9a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:04.484+0000 7f6780c9a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:04.632+0000 7f6780c9a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:04.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: Activating manager daemon x 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: mgrmap e22: x(active, starting, since 0.00675701s) 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:31:04.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:04.636 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 172.21.15.154:0/1304398345' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: Activating manager daemon x 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: osdmap e87: 8 total, 8 up, 8 in 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.24329 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: mgrmap e22: x(active, starting, since 0.00675701s) 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:31:04.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:31:04.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:31:04.639 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:31:04.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:04.698+0000 7f6780c9a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T14:31:04.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:04.844+0000 7f6780c9a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:31:05.300 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:04.983+0000 7f6780c9a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T14:31:05.358 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: Manager daemon x is now available 2024-09-13T14:31:05.358 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.358 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.358 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:05.359 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3234b6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3234b6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3234b6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3234b6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3234b6640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3264bc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3264bc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3264bc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3264bc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.095+0000 7ff3264bc640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:05.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:05] ENGINE Bus STARTING 2024-09-13T14:31:05.560 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:05.299+0000 7f6780c9a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:05.365+0000 7f6780c9a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:05.559+0000 7f6780c9a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: Manager daemon x is now available 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: Manager daemon x is now available 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.562 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:05.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:05 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:05.740 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:05] ENGINE Serving on http://:::9283 2024-09-13T14:31:05.740 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:05] ENGINE Bus STARTED 2024-09-13T14:31:06.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:05.817+0000 7ff309a48640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:31:05] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:06.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: Traceback (most recent call last): 2024-09-13T14:31:06.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:06.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: self._connections.run(self.expiration_interval) 2024-09-13T14:31:06.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:06.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: self._run(expiration_interval) 2024-09-13T14:31:06.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: s = self.context.wrap_socket( 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: return self.sslsocket_class._create( 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: self.do_handshake() 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: self._sslobj.do_handshake() 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:06.242 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:06.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:06.443+0000 7f6780c9a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T14:31:06.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:06.511+0000 7f6780c9a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T14:31:06.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[39074]: Checking for cert/key for grafana.a 2024-09-13T14:31:06.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[39074]: Migrating grafana.a cert to cert store 2024-09-13T14:31:06.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[39074]: Migrating grafana.a key to cert store 2024-09-13T14:31:06.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[39074]: mgrmap e23: x(active, since 1.05168s) 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[39074]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[29001]: Checking for cert/key for grafana.a 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[29001]: Migrating grafana.a cert to cert store 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[29001]: Migrating grafana.a key to cert store 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[29001]: mgrmap e23: x(active, since 1.05168s) 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:06 smithi154 ceph-mon[29001]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:06.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:06 smithi172 ceph-mon[32794]: Checking for cert/key for grafana.a 2024-09-13T14:31:06.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:06 smithi172 ceph-mon[32794]: Migrating grafana.a cert to cert store 2024-09-13T14:31:06.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:06 smithi172 ceph-mon[32794]: Migrating grafana.a key to cert store 2024-09-13T14:31:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:06 smithi172 ceph-mon[32794]: mgrmap e23: x(active, since 1.05168s) 2024-09-13T14:31:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:06 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:06 smithi172 ceph-mon[32794]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:07.541 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:07.244+0000 7f6780c9a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T14:31:07.541 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:07.377+0000 7f6780c9a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: Deploying cephadm binary to smithi172 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: Deploying cephadm binary to smithi154 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:05] ENGINE Bus STARTING 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:05] ENGINE Serving on https://172.21.15.172:7150 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:05] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: self.do_handshake() 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:31:07.543 INFO:journalctl@ceph.mon.a.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:05] ENGINE Serving on http://172.21.15.172:8765 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:05] ENGINE Bus STARTED 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[29001]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: Deploying cephadm binary to smithi172 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: Deploying cephadm binary to smithi154 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:05] ENGINE Bus STARTING 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:05] ENGINE Serving on https://172.21.15.172:7150 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:05] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:07.544 INFO:journalctl@ceph.mon.c.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:31:07.545 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi154.stdout: self.do_handshake() 2024-09-13T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:05] ENGINE Serving on http://172.21.15.172:8765 2024-09-13T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:05] ENGINE Bus STARTED 2024-09-13T14:31:07.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-mon[39074]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: Deploying cephadm binary to smithi172 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: Deploying cephadm binary to smithi154 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:05] ENGINE Bus STARTING 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:05] ENGINE Serving on https://172.21.15.172:7150 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:05] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._run(expiration_interval) 2024-09-13T14:31:07.643 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: s = self.context.wrap_socket( 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.sslsocket_class._create( 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: self.do_handshake() 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._sslobj.do_handshake() 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:05] ENGINE Serving on http://172.21.15.172:8765 2024-09-13T14:31:07.644 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:05] ENGINE Bus STARTED 2024-09-13T14:31:07.645 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:07 smithi172 ceph-mon[32794]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:07.829 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:07.540+0000 7f6780c9a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T14:31:07.829 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:07.602+0000 7f6780c9a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T14:31:07.829 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:07.754+0000 7f6780c9a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T14:31:07.829 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:07.828+0000 7f6780c9a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T14:31:08.118 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:07.890+0000 7f6780c9a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T14:31:08.118 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:08.050+0000 7f6780c9a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T14:31:08.118 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:08.116+0000 7f6780c9a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T14:31:08.415 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-mon[29001]: mgrmap e24: x(active, since 3s) 2024-09-13T14:31:08.415 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-mon[39074]: mgrmap e24: x(active, since 3s) 2024-09-13T14:31:08.416 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:08.186+0000 7f6780c9a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T14:31:08.416 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:08.281+0000 7f6780c9a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T14:31:08.416 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:08.348+0000 7f6780c9a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T14:31:08.416 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:08.414+0000 7f6780c9a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T14:31:08.564 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:08 smithi172 ceph-mon[32794]: mgrmap e24: x(active, since 3s) 2024-09-13T14:31:09.556 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[39074]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:09.556 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[39074]: Standby manager daemon y started 2024-09-13T14:31:09.556 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:31:09.556 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:09.556 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[29001]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[29001]: Standby manager daemon y started 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:09.557 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:09.239+0000 7f6780c9a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T14:31:09.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:31:09] ENGINE Bus STARTING 2024-09-13T14:31:09.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: CherryPy Checker: 2024-09-13T14:31:09.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: The Application mounted at '' has an empty config. 2024-09-13T14:31:09.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: 2024-09-13T14:31:09.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:31:09] ENGINE Serving on http://:::9283 2024-09-13T14:31:09.558 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: [13/Sep/2024:14:31:09] ENGINE Bus STARTED 2024-09-13T14:31:09.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:09 smithi172 ceph-mon[32794]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:09 smithi172 ceph-mon[32794]: Standby manager daemon y started 2024-09-13T14:31:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:09 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:31:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:09 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:09 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:31:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:09 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/2141845139' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:10.621 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:10 smithi172 ceph-mon[32794]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-13T14:31:10.621 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:10 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:31:10.621 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:10 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:10.621 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:10 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:10.621 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:10 smithi172 ceph-mon[32794]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[29001]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-13T14:31:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:31:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[29001]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:10.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[39074]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-13T14:31:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:31:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:10 smithi154 ceph-mon[39074]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:11.304 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi154 to 2480M 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[29001]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi154 to 2480M 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:11 smithi154 ceph-mon[39074]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:11.866 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:31:11.866 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (3m) 1s ago 4m 28.4M - 0.25.0 c8568f914cd2 d46f4205cce4 2024-09-13T14:31:11.867 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (3m) 0s ago 3m 85.2M - 9.4.7 954c08fa6188 9507837610ff 2024-09-13T14:31:11.867 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (2m) 1s ago 2m 16.0M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:31:11.867 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (2m) 0s ago 2m 15.5M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:31:11.867 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (26s) 0s ago 7m 495M - 19.3.0-4697-gd58f7306 fec07016a2b0 e47bd4e01c6c 2024-09-13T14:31:11.867 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:9283,8765 running (9m) 1s ago 9m 292M - 18.2.4-843-g2c01458a c129f5a2fd4e 7d0a3081e77d 2024-09-13T14:31:11.868 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (9m) 1s ago 9m 49.6M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:31:11.868 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (8m) 0s ago 8m 43.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:31:11.868 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (8m) 1s ago 8m 40.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:31:11.868 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (3m) 1s ago 3m 15.2M - 1.5.0 0da6a335fe13 93d3a2d692ca 2024-09-13T14:31:11.868 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (3m) 0s ago 3m 14.7M - 1.5.0 0da6a335fe13 6e12b9645ca4 2024-09-13T14:31:11.869 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (7m) 1s ago 7m 53.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:31:11.869 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (7m) 1s ago 7m 60.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:31:11.869 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (6m) 1s ago 6m 55.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:31:11.869 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (6m) 1s ago 6m 51.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:31:11.869 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (6m) 0s ago 6m 53.9M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:31:11.870 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (5m) 0s ago 5m 54.3M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:31:11.870 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (5m) 0s ago 5m 53.4M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:31:11.870 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (4m) 0s ago 4m 53.1M 4016M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:31:11.870 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (2m) 0s ago 4m 46.7M - 2.43.0 a07b618ecd1d 33387ab51e88 2024-09-13T14:31:11.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi154 to 2480M 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:11.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:11 smithi172 ceph-mon[32794]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:12.200 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 1, 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 1 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:12.201 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:31:12.202 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:31:12.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:12.202 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:31:12.202 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 14, 2024-09-13T14:31:12.202 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 1 2024-09-13T14:31:12.202 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:31:12.202 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:31:12.486 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "1/2 daemons upgraded", 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "", 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:31:12.487 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:31:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[29001]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[29001]: from='client.25502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3823473336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:31:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[29001]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:31:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[39074]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[39074]: from='client.25502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3823473336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:31:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:12 smithi154 ceph-mon[39074]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:31:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-mon[32794]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-mon[32794]: from='client.25502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3823473336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:31:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-mon[32794]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:31:12.991 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:12.838Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T14:31:12.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:12.838Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T14:31:12.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:12.839Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T14:31:12.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:12.841Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T14:31:12.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:12.841Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T14:31:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:13 smithi154 ceph-mon[29001]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:13 smithi154 ceph-mon[39074]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:14.236 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:13 smithi172 ceph-mon[32794]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:15.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:15 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:31:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:31:15.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:15.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:15 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: Adjusting osd_memory_target on smithi172 to 2480M 2024-09-13T14:31:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:31:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:31:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: Adjusting osd_memory_target on smithi172 to 2480M 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:16 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: Adjusting osd_memory_target on smithi172 to 2480M 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:16 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:17.189 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:17 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:17.189 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:17 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:17.189 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:17 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:17.189 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:17 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:17.189 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:17 smithi172 ceph-mon[32794]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:31:17.189 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:17 smithi172 ceph-mon[32794]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:31:17.189 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:17 smithi172 ceph-mon[32794]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:31:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[29001]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:31:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[29001]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:31:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[29001]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[29230]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[39074]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[39074]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:31:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:17 smithi154 ceph-mon[39074]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:31:19.350 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 systemd[1]: Stopping Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:19.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:19 smithi154 ceph-mon[39074]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:19.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:19 smithi154 ceph-mon[29001]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:19.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-mon[32794]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:31:19.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:31:19.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[63244]: ts=2024-09-13T14:31:19.502Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:31:19.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 podman[75188]: 2024-09-13 14:31:19.512400701 +0000 UTC m=+0.192848228 container died 33387ab51e885f6374cc8e6536f44b9c286c92e32af7bed533f9b0ab8dcc6cdc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:20.240 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 podman[75188]: 2024-09-13 14:31:19.960325518 +0000 UTC m=+0.640773090 container cleanup 33387ab51e885f6374cc8e6536f44b9c286c92e32af7bed533f9b0ab8dcc6cdc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:20.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:19 smithi172 bash[75188]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:31:20.661 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:20.661 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 podman[75199]: 2024-09-13 14:31:20.283126665 +0000 UTC m=+0.772589410 container remove 33387ab51e885f6374cc8e6536f44b9c286c92e32af7bed533f9b0ab8dcc6cdc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:20.661 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:31:20.662 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 systemd[1]: Stopped Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:20.662 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.636s CPU time. 2024-09-13T14:31:20.662 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 systemd[1]: Starting Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:20.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:20 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:20.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:20 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:20.924 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 podman[75294]: 2024-09-13 14:31:20.564571639 +0000 UTC m=+0.016603080 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:31:20.925 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 podman[75294]: 2024-09-13 14:31:20.662658785 +0000 UTC m=+0.114690212 container create 8a1026bf1d79a29025511e90fc5fae2c654057ddd76d751539f54e5235228dc2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:20.925 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 podman[75294]: 2024-09-13 14:31:20.898093942 +0000 UTC m=+0.350125380 container init 8a1026bf1d79a29025511e90fc5fae2c654057ddd76d751539f54e5235228dc2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:20.925 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 podman[75294]: 2024-09-13 14:31:20.900965107 +0000 UTC m=+0.352996538 container start 8a1026bf1d79a29025511e90fc5fae2c654057ddd76d751539f54e5235228dc2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:21.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.924Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:31:21.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.924Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:31:21.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.924Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi172 (none))" 2024-09-13T14:31:21.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.924Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:31:21.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.924Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.925Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.926Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.926Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.927Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.929Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.929Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.655µs 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.929Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.930Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.937Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.939Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.939Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.525µs wal_replay_duration=9.661082ms wbl_replay_duration=143ns total_replay_duration=9.705358ms 2024-09-13T14:31:21.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.941Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:31:21.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.941Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:31:21.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.941Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:31:21.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.959Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.562128ms db_storage=696ns remote_storage=1.014µs web_handler=393ns query_engine=768ns scrape=403.809µs scrape_sd=123.922µs notify=16.985µs notify_sd=24.502µs rules=17.620768ms tracing=5.546µs 2024-09-13T14:31:21.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.959Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:31:21.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:20.959Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:31:21.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:20 smithi172 bash[75294]: 8a1026bf1d79a29025511e90fc5fae2c654057ddd76d751539f54e5235228dc2 2024-09-13T14:31:21.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:21 smithi172 systemd[1]: Started Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:21.650 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[29001]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:21.651 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:21.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:21.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:21.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:21.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:31:21.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:21 smithi154 ceph-mon[39074]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:31:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:21 smithi172 ceph-mon[32794]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-13T14:31:22.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:22 smithi172 ceph-mon[32794]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:22 smithi154 ceph-mon[29001]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:22.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:22 smithi154 ceph-mon[39074]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:23.669 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[29001]: Upgrade: Updating mgr.y 2024-09-13T14:31:23.669 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[29001]: Deploying daemon mgr.y on smithi154 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[39074]: Upgrade: Updating mgr.y 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:23.670 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:31:23.671 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:23.671 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:23 smithi154 ceph-mon[39074]: Deploying daemon mgr.y on smithi154 2024-09-13T14:31:23.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:23 smithi172 ceph-mon[32794]: Upgrade: Updating mgr.y 2024-09-13T14:31:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:23 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:23 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:23 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:23 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:31:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:23 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:23 smithi172 ceph-mon[32794]: Deploying daemon mgr.y on smithi154 2024-09-13T14:31:24.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:23 smithi154 systemd[1]: Stopping Ceph mgr.y for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:24.508 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:24 smithi154 ceph-mon[29001]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:24.508 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:24 smithi154 podman[84474]: 2024-09-13 14:31:24.254530193 +0000 UTC m=+0.157488113 container died 7d0a3081e77d89a01d7eee1b28c4522c8d0ccdad14b39935f7f78d7fc4197ae0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-2c01458, org.label-schema.build-date=20240731) 2024-09-13T14:31:24.509 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:24 smithi154 ceph-mon[39074]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:24.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:24 smithi172 ceph-mon[32794]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:24.800 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:24 smithi154 podman[84474]: 2024-09-13 14:31:24.590048605 +0000 UTC m=+0.493006514 container cleanup 7d0a3081e77d89a01d7eee1b28c4522c8d0ccdad14b39935f7f78d7fc4197ae0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux ) 2024-09-13T14:31:24.800 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:24 smithi154 bash[84474]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y 2024-09-13T14:31:25.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:24 smithi154 podman[84486]: 2024-09-13 14:31:24.799952662 +0000 UTC m=+0.540893490 container remove 7d0a3081e77d89a01d7eee1b28c4522c8d0ccdad14b39935f7f78d7fc4197ae0 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, RELEASE=reef-2c01458, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-13T14:31:25.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:24 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:31:25.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-13T14:31:25.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 systemd[1]: Stopped Ceph mgr.y for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:25.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service: Consumed 41.947s CPU time. 2024-09-13T14:31:25.568 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 systemd[1]: Starting Ceph mgr.y for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:25.568 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 podman[84614]: 2024-09-13 14:31:25.341525678 +0000 UTC m=+0.015873142 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:25.568 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 podman[84614]: 2024-09-13 14:31:25.476274308 +0000 UTC m=+0.150621821 container create fc4464b991bdf848adc3d9cdc7a0286f24c11ce3d960fbec45543707885b0966 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T14:31:25.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 podman[84614]: 2024-09-13 14:31:25.687035801 +0000 UTC m=+0.361383264 container init fc4464b991bdf848adc3d9cdc7a0286f24c11ce3d960fbec45543707885b0966 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-13T14:31:25.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 podman[84614]: 2024-09-13 14:31:25.689867158 +0000 UTC m=+0.364214617 container start fc4464b991bdf848adc3d9cdc7a0286f24c11ce3d960fbec45543707885b0966 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-13T14:31:25.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 ceph-mgr[84634]: -- 172.21.15.154:0/488653074 <== mon.2 v2:172.21.15.154:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c9ef8350e0 con 0x55c9ef812800 2024-09-13T14:31:25.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 bash[84614]: fc4464b991bdf848adc3d9cdc7a0286f24c11ce3d960fbec45543707885b0966 2024-09-13T14:31:25.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:25 smithi154 systemd[1]: Started Ceph mgr.y for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:27.140 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:26 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:27.140 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:26 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:27.140 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:26 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:27.140 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:26 smithi172 ceph-mon[32794]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:27.203 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:27.203 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:27.203 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:27.203 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[29001]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:27.204 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:27.204 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:27.204 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:27.204 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:26 smithi154 ceph-mon[39074]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:29.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:29 smithi154 ceph-mon[29001]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:29.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:29 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:29.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:29 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:29 smithi154 ceph-mon[39074]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:29 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:29 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:29.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:29 smithi172 ceph-mon[32794]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:29.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:29 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:29.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:29 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[29001]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:30.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:30.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-mon[39074]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:30 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:30 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:30 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:30 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:30 smithi172 ceph-mon[32794]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:31.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:30] ENGINE Bus STARTING 2024-09-13T14:31:31.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: CherryPy Checker: 2024-09-13T14:31:31.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: The Application mounted at '' has an empty config. 2024-09-13T14:31:31.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:30 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:31.294 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:31] ENGINE Serving on http://:::9283 2024-09-13T14:31:31.295 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:31] ENGINE Bus STARTED 2024-09-13T14:31:31.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:31 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:31:31] "GET /metrics HTTP/1.1" 200 33826 "" "Prometheus/2.43.0" 2024-09-13T14:31:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[29001]: Standby manager daemon y restarted 2024-09-13T14:31:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[29001]: Standby manager daemon y started 2024-09-13T14:31:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:31:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:31:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:32.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[39074]: Standby manager daemon y restarted 2024-09-13T14:31:32.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[39074]: Standby manager daemon y started 2024-09-13T14:31:32.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:31:32.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:32.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:31:32.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:31 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:32.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:31 smithi172 ceph-mon[32794]: Standby manager daemon y restarted 2024-09-13T14:31:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:31 smithi172 ceph-mon[32794]: Standby manager daemon y started 2024-09-13T14:31:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:31 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-13T14:31:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:31 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:31 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-13T14:31:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:31 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:32 smithi154 ceph-mon[29001]: mgrmap e26: x(active, since 27s), standbys: y 2024-09-13T14:31:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:32 smithi154 ceph-mon[29001]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:32.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:32 smithi154 ceph-mon[39074]: mgrmap e26: x(active, since 27s), standbys: y 2024-09-13T14:31:32.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:32 smithi154 ceph-mon[39074]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:33.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:32 smithi172 ceph-mon[32794]: mgrmap e26: x(active, since 27s), standbys: y 2024-09-13T14:31:33.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:32 smithi172 ceph-mon[32794]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:33.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:33] ENGINE Bus STOPPING 2024-09-13T14:31:33.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:33 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:33.295+0000 7ff3b2bbc640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-13T14:31:33.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:33 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: ignoring --setuser ceph since I am not root 2024-09-13T14:31:33.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:33 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: ignoring --setgroup ceph since I am not root 2024-09-13T14:31:33.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:33 smithi172 ceph-mgr[66496]: -- 172.21.15.172:0/162464063 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628398ab0e0 con 0x562839888800 2024-09-13T14:31:33.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:31:33.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:33] ENGINE Bus STOPPED 2024-09-13T14:31:34.314 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.314 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: Activating manager daemon y 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25111 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:31:34.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: mgrmap e27: y(active, starting, since 0.00690666s) 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:31:34.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: Activating manager daemon y 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25111 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: mgrmap e27: y(active, starting, since 0.00690666s) 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:31:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:31:34.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: Manager daemon y is now available 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:34.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: Manager daemon y is now available 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:34.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.984+0000 7fd3b4f54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.984+0000 7fd3b4f54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.984+0000 7fd3b4f54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.984+0000 7fd3b4f54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.984+0000 7fd3b4f54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.985+0000 7fd3b6f58640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.985+0000 7fd3b6f58640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.985+0000 7fd3b6f58640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.985+0000 7fd3b6f58640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:33 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:33.985+0000 7fd3b6f58640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:31:34.322 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:34] ENGINE Bus STARTING 2024-09-13T14:31:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 172.21.15.172:0/2061930405' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: Activating manager daemon y 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25111 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: mgrmap e27: y(active, starting, since 0.00690666s) 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:31:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:31:34.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: Manager daemon y is now available 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:34.494 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:34 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:31:34.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:34] ENGINE Serving on http://:::9283 2024-09-13T14:31:34.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:34 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:31:34] ENGINE Bus STARTED 2024-09-13T14:31:35.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:35.019+0000 7fd39649d640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:31:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: Traceback (most recent call last): 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self._connections.run(self.expiration_interval) 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self._run(expiration_interval) 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:35.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: s = self.context.wrap_socket( 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: return self.sslsocket_class._create( 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self.do_handshake() 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self._sslobj.do_handshake() 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:35.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:31:35.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-mon[29001]: mgrmap e28: y(active, since 1.00864s) 2024-09-13T14:31:35.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:35.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-mon[29001]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:35.551 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-mon[39074]: mgrmap e28: y(active, since 1.00864s) 2024-09-13T14:31:35.551 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:35.551 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:35 smithi154 ceph-mon[39074]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:35.631 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:35 smithi172 ceph-mon[32794]: mgrmap e28: y(active, since 1.00864s) 2024-09-13T14:31:35.631 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:35 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:31:35.631 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:35 smithi172 ceph-mon[32794]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:36.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:36 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:34] ENGINE Bus STARTING 2024-09-13T14:31:36.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:36 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:34] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:31:36.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:36 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:35] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:36 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:35] ENGINE Bus STARTED 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:36 smithi172 ceph-mon[32794]: [13/Sep/2024:14:31:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._run(expiration_interval) 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:36.594 INFO:journalctl@ceph.mon.b.smithi172.stdout: s = self.context.wrap_socket( 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.sslsocket_class._create( 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: self.do_handshake() 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._sslobj.do_handshake() 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:36 smithi172 ceph-mon[32794]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:36.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:36 smithi172 ceph-mon[32794]: mgrmap e29: y(active, since 2s) 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:34] ENGINE Bus STARTING 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:34] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:35] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:35] ENGINE Bus STARTED 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[29001]: [13/Sep/2024:14:31:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:31:36.608 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: self.do_handshake() 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:36.609 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.a.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[29001]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[29001]: mgrmap e29: y(active, since 2s) 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:34] ENGINE Bus STARTING 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:34] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:35] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:35] ENGINE Bus STARTED 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[39074]: [13/Sep/2024:14:31:35] ENGINE Error in HTTPServer.serve 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:31:36.610 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: self.do_handshake() 2024-09-13T14:31:36.611 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:31:36.612 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:31:36.612 INFO:journalctl@ceph.mon.c.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:31:36.612 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[39074]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:36.612 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:36 smithi154 ceph-mon[39074]: mgrmap e29: y(active, since 2s) 2024-09-13T14:31:38.637 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-mon[32794]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:38.637 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-mon[32794]: mgrmap e30: y(active, since 4s) 2024-09-13T14:31:38.637 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.637 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.637 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.638 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.638 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:38] ENGINE Bus STARTING 2024-09-13T14:31:38.638 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: CherryPy Checker: 2024-09-13T14:31:38.638 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: The Application mounted at '' has an empty config. 2024-09-13T14:31:38.638 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: 2024-09-13T14:31:38.638 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:38] ENGINE Serving on http://:::9283 2024-09-13T14:31:38.638 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:38 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: [13/Sep/2024:14:31:38] ENGINE Bus STARTED 2024-09-13T14:31:38.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[29001]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[29001]: mgrmap e30: y(active, since 4s) 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[39074]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[39074]: mgrmap e30: y(active, since 4s) 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:38.678 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:38 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: Standby manager daemon x started 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: Standby manager daemon x started 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.644 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:39 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: Standby manager daemon x started 2024-09-13T14:31:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:31:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:31:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/192746679' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:39.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:39.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:39 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:40.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:40 smithi154 ceph-mon[29001]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:40.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:40 smithi154 ceph-mon[29001]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-13T14:31:40.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:40 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:31:40.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:40 smithi154 ceph-mon[39074]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:40.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:40 smithi154 ceph-mon[39074]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-13T14:31:40.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:40 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:31:40.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:40 smithi172 ceph-mon[32794]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:40 smithi172 ceph-mon[32794]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-13T14:31:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:40 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:31:41.740 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:41 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[66492]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:31:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-13T14:31:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:31:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:42 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:42 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:42.803 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:31:43.357 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:31:43.357 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (3m) 5s ago 4m 28.4M - 0.25.0 c8568f914cd2 d46f4205cce4 2024-09-13T14:31:43.357 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (4m) 5s ago 4m 86.5M - 9.4.7 954c08fa6188 9507837610ff 2024-09-13T14:31:43.357 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (3m) 5s ago 3m 16.5M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:31:43.357 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (3m) 5s ago 3m 15.8M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:31:43.358 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (58s) 5s ago 8m 126M - 19.3.0-4697-gd58f7306 fec07016a2b0 e47bd4e01c6c 2024-09-13T14:31:43.358 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (18s) 5s ago 9m 487M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:31:43.358 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (9m) 5s ago 10m 52.4M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:31:43.358 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (8m) 5s ago 8m 47.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:31:43.358 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (8m) 5s ago 8m 43.0M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:31:43.359 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (4m) 5s ago 3m 15.6M - 1.5.0 0da6a335fe13 93d3a2d692ca 2024-09-13T14:31:43.359 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (3m) 5s ago 3m 15.3M - 1.5.0 0da6a335fe13 6e12b9645ca4 2024-09-13T14:31:43.359 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (8m) 5s ago 8m 52.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:31:43.359 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (7m) 5s ago 7m 60.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:31:43.359 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (7m) 5s ago 7m 56.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:31:43.359 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (6m) 5s ago 6m 51.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:31:43.360 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (6m) 5s ago 6m 54.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:31:43.360 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (6m) 5s ago 6m 54.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:31:43.360 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (5m) 5s ago 5m 53.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:31:43.360 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (5m) 5s ago 5m 53.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:31:43.360 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (22s) 5s ago 4m 35.9M - 2.43.0 a07b618ecd1d 8a1026bf1d79 2024-09-13T14:31:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:43 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:43 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:43 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:43 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:43 smithi172 ceph-mon[32794]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:31:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:43 smithi172 ceph-mon[32794]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:31:43.691 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:43.692 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:31:43.693 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:31:43.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:31:43.693 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:31:43.693 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:31:43.693 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:31:43.693 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:31:43.693 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:31:43.693 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[29001]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:43.693 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[29001]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[29001]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[29001]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[29001]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[29001]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[39074]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-13T14:31:43.694 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:43 smithi154 ceph-mon[39074]: Reconfiguring daemon prometheus.a on smithi172 2024-09-13T14:31:43.974 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:31:43.975 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:31:43.975 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:31:43.975 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:31:43.975 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [ 2024-09-13T14:31:43.975 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr" 2024-09-13T14:31:43.976 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:31:43.976 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:31:43.976 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "", 2024-09-13T14:31:43.976 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:31:43.976 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:31:44.516 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:44 smithi172 ceph-mon[32794]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.516 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:44 smithi172 ceph-mon[32794]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.516 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:44 smithi172 ceph-mon[32794]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:31:44.516 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:44 smithi172 ceph-mon[32794]: from='client.25574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.516 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:44 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/267033985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:31:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[29001]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[29001]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[29001]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:31:44.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[29001]: from='client.25574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/267033985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:31:44.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[39074]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[39074]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[39074]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:31:44.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[39074]: from='client.25574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:44.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:44 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/267033985' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:31:45.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-mon[32794]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:45.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:45 smithi154 ceph-mon[29001]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:45.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:45 smithi154 ceph-mon[39074]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:31:45.886 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 systemd[1]: Stopping Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.885Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:31:46.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:31:46.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:31:46.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:31:46.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:31:46.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[75309]: ts=2024-09-13T14:31:45.886Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:31:46.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:45 smithi172 podman[77744]: 2024-09-13 14:31:45.896172805 +0000 UTC m=+0.118866128 container died 8a1026bf1d79a29025511e90fc5fae2c654057ddd76d751539f54e5235228dc2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:46.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:46 smithi172 ceph-mon[32794]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:31:46.491 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:46 smithi172 podman[77744]: 2024-09-13 14:31:46.402656942 +0000 UTC m=+0.625350306 container cleanup 8a1026bf1d79a29025511e90fc5fae2c654057ddd76d751539f54e5235228dc2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:46.491 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:46 smithi172 bash[77744]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:31:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:46 smithi154 ceph-mon[29001]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:31:46.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:46 smithi154 ceph-mon[39074]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-13T14:31:46.890 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:46 smithi172 podman[77758]: 2024-09-13 14:31:46.637060204 +0000 UTC m=+0.743072883 container remove 8a1026bf1d79a29025511e90fc5fae2c654057ddd76d751539f54e5235228dc2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:47.220 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:46 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:31:47.220 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:46 smithi172 systemd[1]: Stopped Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:47.220 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:46 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.038s CPU time. 2024-09-13T14:31:47.220 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:46 smithi172 systemd[1]: Starting Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:47.221 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 podman[77849]: 2024-09-13 14:31:47.000482202 +0000 UTC m=+0.016934539 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-13T14:31:47.221 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 podman[77849]: 2024-09-13 14:31:47.129704866 +0000 UTC m=+0.146157196 container create 05f4ba04a6c02a5e44bd79be95529b477c9b3d43ac255b09e29d7d393096b45c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:47.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 podman[77849]: 2024-09-13 14:31:47.348584763 +0000 UTC m=+0.365037093 container init 05f4ba04a6c02a5e44bd79be95529b477c9b3d43ac255b09e29d7d393096b45c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:47.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 podman[77849]: 2024-09-13 14:31:47.351493487 +0000 UTC m=+0.367945817 container start 05f4ba04a6c02a5e44bd79be95529b477c9b3d43ac255b09e29d7d393096b45c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:31:47.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.375Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-13T14:31:47.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.375Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-13T14:31:47.741 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.375Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi172 (none))" 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.375Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.375Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.377Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.377Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.378Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.378Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.379Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.379Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.524µs 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.379Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.382Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.387Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-13T14:31:47.742 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.388Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.388Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.388Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.096µs wal_replay_duration=8.978892ms wbl_replay_duration=149ns total_replay_duration=9.032039ms 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.391Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.391Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.391Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.409Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.029442ms db_storage=792ns remote_storage=1.102µs web_handler=354ns query_engine=715ns scrape=413.89µs scrape_sd=102.524µs notify=10.981µs notify_sd=9.899µs rules=17.088615ms tracing=6.542µs 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.409Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:31:47.409Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 bash[77849]: 05f4ba04a6c02a5e44bd79be95529b477c9b3d43ac255b09e29d7d393096b45c 2024-09-13T14:31:47.743 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:31:47 smithi172 systemd[1]: Started Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[29001]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:48.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[39074]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:48.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:48.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:48.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:48.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:48.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:48 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:48 smithi172 ceph-mon[32794]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:48 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:48 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:48 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:48 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:31:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:48 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:49.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:49 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:49.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:49 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:49.668 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:49 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:49.668 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:49 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:31:49.668 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:49 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:31:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:31:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T14:31:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T14:31:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:49 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:31:50.708 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:50 smithi172 ceph-mon[32794]: Upgrade: Updating mgr.x 2024-09-13T14:31:50.708 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:50 smithi172 ceph-mon[32794]: Deploying daemon mgr.x on smithi172 2024-09-13T14:31:50.708 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:50 smithi172 ceph-mon[32794]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:50.887 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:50 smithi154 ceph-mon[29001]: Upgrade: Updating mgr.x 2024-09-13T14:31:50.887 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:50 smithi154 ceph-mon[29001]: Deploying daemon mgr.x on smithi172 2024-09-13T14:31:50.887 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:50 smithi154 ceph-mon[29001]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:50.887 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:50 smithi154 ceph-mon[39074]: Upgrade: Updating mgr.x 2024-09-13T14:31:50.887 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:50 smithi154 ceph-mon[39074]: Deploying daemon mgr.x on smithi172 2024-09-13T14:31:50.887 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:50 smithi154 ceph-mon[39074]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:50.990 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:50 smithi172 systemd[1]: Stopping Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:51.276 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:50 smithi172 podman[78181]: 2024-09-13 14:31:50.998178818 +0000 UTC m=+0.147012455 container died e47bd4e01c6ce7d7045c0f71776ddccd3fe08970332e3e9134affcc285a03ed8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:31:51.635 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 podman[78181]: 2024-09-13 14:31:51.358554827 +0000 UTC m=+0.507388463 container cleanup e47bd4e01c6ce7d7045c0f71776ddccd3fe08970332e3e9134affcc285a03ed8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-13T14:31:51.635 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 bash[78181]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x 2024-09-13T14:31:51.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 podman[78193]: 2024-09-13 14:31:51.635097621 +0000 UTC m=+0.636805880 container remove e47bd4e01c6ce7d7045c0f71776ddccd3fe08970332e3e9134affcc285a03ed8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-13T14:31:51.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:31:51.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-13T14:31:51.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 systemd[1]: Stopped Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:51.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service: Consumed 16.735s CPU time. 2024-09-13T14:31:52.319 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:51 smithi172 systemd[1]: Starting Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:31:52.319 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:52 smithi172 podman[78323]: 2024-09-13 14:31:52.10607545 +0000 UTC m=+0.016218020 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:31:52.319 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:52 smithi172 podman[78323]: 2024-09-13 14:31:52.219724171 +0000 UTC m=+0.129866728 container create b3bb4b6384823601d2cae6e5ce7da8821f119fdfa6b8794026847c236f488dbc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T14:31:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:52 smithi172 ceph-mon[32794]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:52.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:52 smithi172 podman[78323]: 2024-09-13 14:31:52.397223648 +0000 UTC m=+0.307366204 container init b3bb4b6384823601d2cae6e5ce7da8821f119fdfa6b8794026847c236f488dbc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-13T14:31:52.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:52 smithi172 podman[78323]: 2024-09-13 14:31:52.400126933 +0000 UTC m=+0.310269490 container start b3bb4b6384823601d2cae6e5ce7da8821f119fdfa6b8794026847c236f488dbc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:31:52.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:52 smithi172 ceph-mgr[78342]: -- 172.21.15.172:0/712612101 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e99b1590e0 con 0x55e99b137000 2024-09-13T14:31:52.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:52 smithi172 bash[78323]: b3bb4b6384823601d2cae6e5ce7da8821f119fdfa6b8794026847c236f488dbc 2024-09-13T14:31:52.741 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:52 smithi172 systemd[1]: Started Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:31:52.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:52 smithi154 ceph-mon[29001]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:52.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:52 smithi154 ceph-mon[39074]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-13T14:31:53.863 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:53 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:53.863 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:53 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:53.863 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:53 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:53.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:53 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:53.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:53 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:53.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:53 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:53.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:53 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:53.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:53 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:53.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:53 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:31:54.891 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:54 smithi172 ceph-mon[32794]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:54.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:54 smithi154 ceph-mon[29001]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:54.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:54 smithi154 ceph-mon[39074]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:56.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:56 smithi172 ceph-mon[32794]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:56.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:56 smithi154 ceph-mon[29001]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:56.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:56 smithi154 ceph-mon[39074]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:57.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:31:57] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.43.0" 2024-09-13T14:31:57.958 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: [13/Sep/2024:14:31:57] ENGINE Bus STARTING 2024-09-13T14:31:57.958 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: CherryPy Checker: 2024-09-13T14:31:57.958 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: The Application mounted at '' has an empty config. 2024-09-13T14:31:57.958 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: 2024-09-13T14:31:57.958 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: [13/Sep/2024:14:31:57] ENGINE Serving on http://:::9283 2024-09-13T14:31:57.958 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: [13/Sep/2024:14:31:57] ENGINE Bus STARTED 2024-09-13T14:31:57.958 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: Standby manager daemon x restarted 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: Standby manager daemon x started 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:31:57.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:57 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: Standby manager daemon x restarted 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: Standby manager daemon x started 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[29001]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:31:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:31:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: Standby manager daemon x restarted 2024-09-13T14:31:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: Standby manager daemon x started 2024-09-13T14:31:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:31:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:31:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:31:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:57 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/4052188486' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:31:59.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:31:59 smithi154 ceph-mon[29001]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-13T14:31:59.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:31:59 smithi154 ceph-mon[39074]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-13T14:31:59.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:31:59 smithi172 ceph-mon[32794]: mgrmap e32: y(active, since 25s), standbys: x 2024-09-13T14:32:00.647 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:00 smithi172 ceph-mon[32794]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:00 smithi154 ceph-mon[29001]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:00.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:00 smithi154 ceph-mon[39074]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:01.908 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:32:01.909 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.911 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:32:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:01.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all mgr 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all crash 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all rgw 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all nfs 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:32:02.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[29001]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:02.972 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:02 smithi154 systemd[1]: Stopping Ceph node-exporter.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all mgr 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all crash 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all rgw 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:32:02.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all nfs 2024-09-13T14:32:02.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:32:02.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:02 smithi154 ceph-mon[39074]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all mgr 2024-09-13T14:32:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all crash 2024-09-13T14:32:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all rgw 2024-09-13T14:32:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:32:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:32:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:32:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:32:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all nfs 2024-09-13T14:32:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:32:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:02 smithi172 ceph-mon[32794]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:03.470 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 podman[92743]: 2024-09-13 14:32:03.174711482 +0000 UTC m=+0.113101983 container died 93d3a2d692cab99b3413fe40f66b5f709edae21926d30dded6399960d337ca1b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:03.724 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:03 smithi154 ceph-mon[29001]: Upgrade: Updating node-exporter.a (1/2) 2024-09-13T14:32:03.724 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:03 smithi154 ceph-mon[29001]: Deploying daemon node-exporter.a on smithi154 2024-09-13T14:32:03.725 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 podman[92743]: 2024-09-13 14:32:03.47208669 +0000 UTC m=+0.410477186 container cleanup 93d3a2d692cab99b3413fe40f66b5f709edae21926d30dded6399960d337ca1b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:03.725 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 bash[92743]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T14:32:03.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:03 smithi154 ceph-mon[39074]: Upgrade: Updating node-exporter.a (1/2) 2024-09-13T14:32:03.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:03 smithi154 ceph-mon[39074]: Deploying daemon node-exporter.a on smithi154 2024-09-13T14:32:03.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:03 smithi172 ceph-mon[32794]: Upgrade: Updating node-exporter.a (1/2) 2024-09-13T14:32:03.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:03 smithi172 ceph-mon[32794]: Deploying daemon node-exporter.a on smithi154 2024-09-13T14:32:04.115 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 podman[92757]: 2024-09-13 14:32:03.723969745 +0000 UTC m=+0.549396199 container remove 93d3a2d692cab99b3413fe40f66b5f709edae21926d30dded6399960d337ca1b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:04.115 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:32:04.115 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T14:32:04.115 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 systemd[1]: Stopped Ceph node-exporter.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:04.115 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:03 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 2.129s CPU time. 2024-09-13T14:32:04.470 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:04 smithi154 systemd[1]: Starting Ceph node-exporter.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:04.470 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:04 smithi154 bash[92884]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:32:04.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:04 smithi154 ceph-mon[29001]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:04.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:04 smithi154 ceph-mon[39074]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:04.970 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:04 smithi154 bash[92884]: Getting image source signatures 2024-09-13T14:32:04.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:04 smithi154 bash[92884]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:32:04.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:04 smithi154 bash[92884]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:32:04.971 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:04 smithi154 bash[92884]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:32:04.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:04 smithi172 ceph-mon[32794]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:04.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:06.220 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:05 smithi154 bash[92884]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:32:06.220 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 bash[92884]: Writing manifest to image destination 2024-09-13T14:32:06.628 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 podman[92884]: 2024-09-13 14:32:06.29408313 +0000 UTC m=+2.085949906 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:32:06.628 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 podman[92884]: 2024-09-13 14:32:06.417584923 +0000 UTC m=+2.209451709 container create c2e00912b58cf27fe98516d0507ac06175dfd66f99c5604248d44c97134ab2e4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:06.900 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-mon[29001]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:06.900 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-mon[39074]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 podman[92884]: 2024-09-13 14:32:06.628039312 +0000 UTC m=+2.419906078 container init c2e00912b58cf27fe98516d0507ac06175dfd66f99c5604248d44c97134ab2e4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 podman[92884]: 2024-09-13 14:32:06.631595452 +0000 UTC m=+2.423462213 container start c2e00912b58cf27fe98516d0507ac06175dfd66f99c5604248d44c97134ab2e4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.635Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.635Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.635Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.635Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:32:06.901 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:32:06.902 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:32:06.903 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:32:06.904 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.636Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.637Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a[92959]: ts=2024-09-13T14:32:06.637Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 bash[92884]: c2e00912b58cf27fe98516d0507ac06175dfd66f99c5604248d44c97134ab2e4 2024-09-13T14:32:06.905 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 14:32:06 smithi154 systemd[1]: Started Ceph node-exporter.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:06.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:06 smithi172 ceph-mon[32794]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:07.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:32:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:32:07] "GET /metrics HTTP/1.1" 200 33825 "" "Prometheus/2.43.0" 2024-09-13T14:32:08.052 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:07 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:08.052 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:07 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:08.052 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:07 smithi172 ceph-mon[32794]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:07 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:07 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:07 smithi154 ceph-mon[29001]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:07 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:08.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:07 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:08.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:07 smithi154 ceph-mon[39074]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:08.741 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:08 smithi172 systemd[1]: Stopping Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:09.166 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:08 smithi172 ceph-mon[32794]: Upgrade: Updating node-exporter.b (2/2) 2024-09-13T14:32:09.166 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:08 smithi172 ceph-mon[32794]: Deploying daemon node-exporter.b on smithi172 2024-09-13T14:32:09.166 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:08 smithi172 podman[81849]: 2024-09-13 14:32:08.875642877 +0000 UTC m=+0.129634791 container died 6e12b9645ca4e7f9d326570001c6cfacc973103cf6d67f0e0d6175728bc4ac2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:32:09.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:08 smithi154 ceph-mon[29001]: Upgrade: Updating node-exporter.b (2/2) 2024-09-13T14:32:09.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:08 smithi154 ceph-mon[29001]: Deploying daemon node-exporter.b on smithi172 2024-09-13T14:32:09.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:08 smithi154 ceph-mon[39074]: Upgrade: Updating node-exporter.b (2/2) 2024-09-13T14:32:09.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:08 smithi154 ceph-mon[39074]: Deploying daemon node-exporter.b on smithi172 2024-09-13T14:32:09.490 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 podman[81849]: 2024-09-13 14:32:09.264704676 +0000 UTC m=+0.518696598 container cleanup 6e12b9645ca4e7f9d326570001c6cfacc973103cf6d67f0e0d6175728bc4ac2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:32:09.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 bash[81849]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T14:32:09.802 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 podman[81862]: 2024-09-13 14:32:09.524325744 +0000 UTC m=+0.652241914 container remove 6e12b9645ca4e7f9d326570001c6cfacc973103cf6d67f0e0d6175728bc4ac2e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:32:09.802 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:32:09.802 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T14:32:09.802 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 systemd[1]: Stopped Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:09.802 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.087s CPU time. 2024-09-13T14:32:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:09 smithi154 ceph-mon[29001]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:10.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:09 smithi154 ceph-mon[39074]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:10.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:09 smithi172 ceph-mon[32794]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:10.241 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:09 smithi172 systemd[1]: Starting Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:10.241 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:10 smithi172 bash[81984]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-13T14:32:10.990 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:10 smithi172 bash[81984]: Getting image source signatures 2024-09-13T14:32:10.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:10 smithi172 bash[81984]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-13T14:32:10.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:10 smithi172 bash[81984]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-13T14:32:10.991 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:10 smithi172 bash[81984]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-13T14:32:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:12 smithi154 ceph-mon[29001]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:12.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:12 smithi154 ceph-mon[39074]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:12.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:12 smithi172 ceph-mon[32794]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:12.740 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:12 smithi172 bash[81984]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-13T14:32:12.741 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:12 smithi172 bash[81984]: Writing manifest to image destination 2024-09-13T14:32:13.053 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:12 smithi172 podman[81984]: 2024-09-13 14:32:12.727032283 +0000 UTC m=+2.774909458 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-13T14:32:13.053 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:12 smithi172 podman[81984]: 2024-09-13 14:32:12.842616107 +0000 UTC m=+2.890493281 container create 60a8496c91efed099f0187e733f24cfdc9c3b83e14e40634edc67f83b280ffb6 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:32:13.053 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 podman[81984]: 2024-09-13 14:32:13.052891016 +0000 UTC m=+3.100768197 container init 60a8496c91efed099f0187e733f24cfdc9c3b83e14e40634edc67f83b280ffb6 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:32:13.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 podman[81984]: 2024-09-13 14:32:13.055884954 +0000 UTC m=+3.103762134 container start 60a8496c91efed099f0187e733f24cfdc9c3b83e14e40634edc67f83b280ffb6 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T14:32:13.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.059Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-13T14:32:13.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.059Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-13T14:32:13.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.059Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-13T14:32:13.491 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.059Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.059Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=arp 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-13T14:32:13.492 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-13T14:32:13.493 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=edac 2024-09-13T14:32:13.493 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-13T14:32:13.493 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-13T14:32:13.493 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-13T14:32:13.493 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-13T14:32:13.493 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=os 2024-09-13T14:32:13.494 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=stat 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=time 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=uname 2024-09-13T14:32:13.495 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-13T14:32:13.496 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-13T14:32:13.496 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.060Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-13T14:32:13.496 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.061Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-13T14:32:13.496 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b[82058]: ts=2024-09-13T14:32:13.061Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-13T14:32:13.496 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 bash[81984]: 60a8496c91efed099f0187e733f24cfdc9c3b83e14e40634edc67f83b280ffb6 2024-09-13T14:32:13.496 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 14:32:13 smithi172 systemd[1]: Started Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:14.287 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:32:14.468 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:14 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:14.468 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:14 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:14.468 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:14 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:14.468 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:14 smithi172 ceph-mon[32794]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:14.468 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:14.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:14.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:14.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[29001]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:14.469 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:14.469 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:14.469 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:14.469 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:14 smithi154 ceph-mon[39074]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (4m) 37s ago 5m 28.4M - 0.25.0 c8568f914cd2 d46f4205cce4 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (4m) 17s ago 4m 86.5M - 9.4.7 954c08fa6188 9507837610ff 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (3m) 37s ago 3m 16.5M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (3m) 17s ago 3m 16.0M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (22s) 17s ago 8m 65.9M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (49s) 37s ago 10m 487M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (10m) 37s ago 10m 52.4M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (9m) 17s ago 9m 48.5M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:32:14.858 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (9m) 37s ago 9m 43.0M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 starting - - - - 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 starting - - - - 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (8m) 37s ago 8m 52.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (8m) 37s ago 8m 60.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (7m) 37s ago 7m 56.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (7m) 37s ago 7m 51.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (7m) 17s ago 7m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (6m) 17s ago 6m 54.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (6m) 17s ago 6m 53.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (5m) 17s ago 5m 53.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:32:14.859 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (27s) 17s ago 5m 38.2M - 2.43.0 a07b618ecd1d 05f4ba04a6c0 2024-09-13T14:32:15.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:15 smithi154 ceph-mon[29001]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:15.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:15 smithi154 ceph-mon[29001]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:15.265 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:32:15.265 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:32:15.265 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:32:15.266 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:32:15.267 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:32:15.267 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:32:15.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:15 smithi154 ceph-mon[39074]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:15.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:15 smithi154 ceph-mon[39074]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:15.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:15 smithi172 ceph-mon[32794]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:15.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:15 smithi172 ceph-mon[32794]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:15.557 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:32:15.557 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [ 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr" 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading node-exporter daemons", 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:32:15.558 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:32:16.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:16 smithi172 ceph-mon[32794]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:16 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/3458756889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:16 smithi172 ceph-mon[32794]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:16 smithi172 ceph-mon[32794]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[39074]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/3458756889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[39074]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[39074]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[29001]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:16.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/3458756889' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:16.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[29001]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:16.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:16 smithi154 ceph-mon[29001]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:17.682 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:32:17] "GET /metrics HTTP/1.1" 200 33825 "" "Prometheus/2.43.0" 2024-09-13T14:32:17.682 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.683 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:17 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:17 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:17 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:17 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:17 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:17 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:17 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:18.715 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:18 smithi172 ceph-mon[32794]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:18.715 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:18 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:18.715 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:18 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:18.887 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:18 smithi154 ceph-mon[29001]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:18.887 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:18 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:18.887 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:18 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:18.887 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:18 smithi154 ceph-mon[39074]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:18.887 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:18 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:18.888 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:18 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:19.664 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:19 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:19.751 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:19 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:19.751 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:19 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:20 smithi154 ceph-mon[29001]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:20 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:20 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:20.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:20 smithi154 ceph-mon[39074]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:20.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:20 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:20.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:20 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:20.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:20 smithi172 ceph-mon[32794]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:20.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:20 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:20.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:20 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:22 smithi172 ceph-mon[32794]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[29001]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:22.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:22 smithi154 ceph-mon[39074]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:23.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:23 smithi154 ceph-mon[29001]: Upgrade: Updating prometheus.a 2024-09-13T14:32:23.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:23 smithi154 ceph-mon[39074]: Upgrade: Updating prometheus.a 2024-09-13T14:32:23.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:23 smithi172 ceph-mon[32794]: Upgrade: Updating prometheus.a 2024-09-13T14:32:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:24 smithi154 ceph-mon[29001]: Deploying daemon prometheus.a on smithi172 2024-09-13T14:32:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:24 smithi154 ceph-mon[29001]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:24 smithi154 ceph-mon[39074]: Deploying daemon prometheus.a on smithi172 2024-09-13T14:32:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:24 smithi154 ceph-mon[39074]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:24.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:24 smithi172 ceph-mon[32794]: Deploying daemon prometheus.a on smithi172 2024-09-13T14:32:24.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:24 smithi172 ceph-mon[32794]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:26 smithi154 ceph-mon[29001]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:26.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:26 smithi154 ceph-mon[39074]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:26.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:26 smithi172 ceph-mon[32794]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:27.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:32:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:32:27] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.43.0" 2024-09-13T14:32:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:28 smithi154 ceph-mon[29001]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:28 smithi154 ceph-mon[39074]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:28.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:28 smithi172 ceph-mon[32794]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:30 smithi154 ceph-mon[29001]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:30 smithi154 ceph-mon[39074]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:30.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:30 smithi172 ceph-mon[32794]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-mon[32794]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:32 smithi154 ceph-mon[29001]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:32.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:32 smithi154 ceph-mon[39074]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:32.792 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 systemd[1]: Stopping Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:33.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.791Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:32:33.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-13T14:32:33.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[77863]: ts=2024-09-13T14:32:32.792Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-13T14:32:33.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:32 smithi172 podman[86258]: 2024-09-13 14:32:32.802083693 +0000 UTC m=+0.135660193 container died 05f4ba04a6c02a5e44bd79be95529b477c9b3d43ac255b09e29d7d393096b45c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:34.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:34 smithi172 ceph-mon[32794]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:34.491 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:34 smithi172 podman[86258]: 2024-09-13 14:32:34.247963654 +0000 UTC m=+1.581540137 container cleanup 05f4ba04a6c02a5e44bd79be95529b477c9b3d43ac255b09e29d7d393096b45c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:34.491 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:34 smithi172 bash[86258]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T14:32:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:34 smithi154 ceph-mon[29001]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:34.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:34 smithi154 ceph-mon[39074]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:35.240 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:34 smithi172 podman[86273]: 2024-09-13 14:32:34.941150235 +0000 UTC m=+2.141390743 container remove 05f4ba04a6c02a5e44bd79be95529b477c9b3d43ac255b09e29d7d393096b45c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:35.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-13T14:32:35.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 systemd[1]: Stopped Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:35.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.094s CPU time. 2024-09-13T14:32:35.696 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 systemd[1]: Starting Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:35.696 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 podman[86411]: 2024-09-13 14:32:35.550472252 +0000 UTC m=+0.112760610 container create 12b1d8bc5be4b55197b1a17ea4a00c6f5cb064126e61751ff30a0c01a9aaf36d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:35.696 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 podman[86411]: 2024-09-13 14:32:35.454466154 +0000 UTC m=+0.016754470 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-13T14:32:35.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 podman[86411]: 2024-09-13 14:32:35.811162671 +0000 UTC m=+0.373451001 container init 12b1d8bc5be4b55197b1a17ea4a00c6f5cb064126e61751ff30a0c01a9aaf36d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:35.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 podman[86411]: 2024-09-13 14:32:35.814176086 +0000 UTC m=+0.376464407 container start 12b1d8bc5be4b55197b1a17ea4a00c6f5cb064126e61751ff30a0c01a9aaf36d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:35.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.834Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-13T14:32:35.992 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.834Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.834Z caller=main.go:623 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi172 (none))" 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.834Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.834Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.836Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.837Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.839Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.839Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.841Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.841Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.643µs 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.841Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-13T14:32:35.993 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.842Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.848Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.849Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.850Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.850Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.850Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.467µs wal_replay_duration=9.693237ms wbl_replay_duration=112ns total_replay_duration=9.753131ms 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.853Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.853Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.853Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.869Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.014384ms db_storage=908ns remote_storage=1.246µs web_handler=403ns query_engine=566ns scrape=445.482µs scrape_sd=103.444µs notify=18.157µs notify_sd=36.258µs rules=15.05439ms tracing=7.176µs 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.869Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T14:32:35.869Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 bash[86411]: 12b1d8bc5be4b55197b1a17ea4a00c6f5cb064126e61751ff30a0c01a9aaf36d 2024-09-13T14:32:35.994 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 14:32:35 smithi172 systemd[1]: Started Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:36.483 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[29001]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:36.483 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.483 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.483 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[39074]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.484 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:36 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.663 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:36 smithi172 ceph-mon[32794]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:36.664 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:36 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.664 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:36 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.664 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:36 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:36.664 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:36 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:36.664 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:36 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:38 smithi172 ceph-mon[32794]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:38 smithi154 ceph-mon[29001]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:38 smithi154 ceph-mon[39074]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:40 smithi154 ceph-mon[29001]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:40 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:40 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:40 smithi154 ceph-mon[39074]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:40 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:40 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:40.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:40 smithi172 ceph-mon[32794]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:40.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:40 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:40.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:40 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:41.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:41 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:41 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:41 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:41 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:42.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:41 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:41 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:42.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:32:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:32:41] "GET /metrics HTTP/1.1" 200 33824 "" "Prometheus/2.51.0" 2024-09-13T14:32:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:42 smithi172 ceph-mon[32794]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:43.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:42 smithi154 ceph-mon[29001]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:43.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:42 smithi154 ceph-mon[39074]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:43.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:43 smithi172 ceph-mon[32794]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:43 smithi154 ceph-mon[29001]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:43 smithi154 ceph-mon[39074]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:44.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:44.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:44.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:44.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:32:44.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:44.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:44 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:44 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:45.870 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:32:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:45 smithi154 ceph-mon[29001]: Upgrade: Updating alertmanager.a 2024-09-13T14:32:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:45 smithi154 ceph-mon[29001]: Deploying daemon alertmanager.a on smithi154 2024-09-13T14:32:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:45 smithi154 ceph-mon[29001]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:45 smithi154 ceph-mon[39074]: Upgrade: Updating alertmanager.a 2024-09-13T14:32:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:45 smithi154 ceph-mon[39074]: Deploying daemon alertmanager.a on smithi154 2024-09-13T14:32:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:45 smithi154 ceph-mon[39074]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:45 smithi172 ceph-mon[32794]: Upgrade: Updating alertmanager.a 2024-09-13T14:32:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:45 smithi172 ceph-mon[32794]: Deploying daemon alertmanager.a on smithi154 2024-09-13T14:32:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:45 smithi172 ceph-mon[32794]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (4m) 29s ago 5m 26.9M - 0.25.0 c8568f914cd2 d46f4205cce4 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (5m) 6s ago 5m 86.4M - 9.4.7 954c08fa6188 9507837610ff 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (4m) 29s ago 4m 17.0M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (4m) 6s ago 4m 16.2M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (54s) 6s ago 9m 452M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (81s) 29s ago 11m 506M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:32:46.449 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (11m) 29s ago 11m 54.2M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (9m) 6s ago 9m 52.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (9m) 29s ago 9m 45.7M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (40s) 29s ago 5m 7247k - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (33s) 6s ago 4m 10.1M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (9m) 29s ago 9m 52.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (8m) 29s ago 8m 60.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (8m) 29s ago 8m 56.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (7m) 29s ago 7m 52.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (7m) 6s ago 7m 54.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (7m) 6s ago 7m 54.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (6m) 6s ago 6m 54.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (6m) 6s ago 6m 54.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:32:46.450 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (10s) 6s ago 5m 31.3M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:32:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:32:46.795 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:32:47.085 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:46 smithi172 ceph-mon[32794]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.085 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:46 smithi172 ceph-mon[32794]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.085 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:46 smithi172 ceph-mon[32794]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.085 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:46 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/1728143599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:47.089 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:32:47.089 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:32:47.089 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [ 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr" 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading alertmanager daemons", 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:32:47.090 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:32:47.090 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[29001]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.090 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[29001]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.091 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[29001]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.091 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/1728143599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:47.091 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[39074]: from='client.15741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.091 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[39074]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.091 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[39074]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:47.091 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:46 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/1728143599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:32:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:47 smithi154 ceph-mon[29001]: from='client.15765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:47 smithi154 ceph-mon[29001]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:47 smithi154 ceph-mon[39074]: from='client.15765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:47 smithi154 ceph-mon[39074]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:47 smithi172 ceph-mon[32794]: from='client.15765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:32:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:47 smithi172 ceph-mon[32794]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:49.418 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:48 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:49.418 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:48 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:49.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:48 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:32:50.391 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:49 smithi154 ceph-mon[29001]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:50.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:49 smithi154 ceph-mon[39074]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:49 smithi172 ceph-mon[32794]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:52.075 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:32:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:32:51] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:32:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:52 smithi154 ceph-mon[29001]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:52.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:52 smithi154 ceph-mon[39074]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:52 smithi172 ceph-mon[32794]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:53.793 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:53 smithi154 systemd[1]: Stopping Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:54.094 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:53 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[71848]: ts=2024-09-13T14:32:53.793Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T14:32:54.094 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:53 smithi154 podman[97428]: 2024-09-13 14:32:53.833735312 +0000 UTC m=+0.155095812 container died d46f4205cce47ed7bf985c4b109664ed2a65cec53a7c32faee5aa6d08101b7ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:54.374 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:54 smithi154 podman[97428]: 2024-09-13 14:32:54.285035615 +0000 UTC m=+0.606396150 container cleanup d46f4205cce47ed7bf985c4b109664ed2a65cec53a7c32faee5aa6d08101b7ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:54.374 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:54 smithi154 bash[97428]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T14:32:54.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:54 smithi154 ceph-mon[29001]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:54.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:54 smithi154 ceph-mon[39074]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:54.720 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:54 smithi154 podman[97441]: 2024-09-13 14:32:54.50380386 +0000 UTC m=+0.660115288 container remove d46f4205cce47ed7bf985c4b109664ed2a65cec53a7c32faee5aa6d08101b7ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:54.721 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:54 smithi154 podman[97441]: 2024-09-13 14:32:54.587744781 +0000 UTC m=+0.744056208 volume remove 92e3a3475dda0753403871ec1af27331f63b54232be129ead2b20e23ad273c67 2024-09-13T14:32:54.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:54 smithi172 ceph-mon[32794]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:55.220 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:54 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T14:32:55.220 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:54 smithi154 systemd[1]: Stopped Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:55.220 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:54 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.133s CPU time. 2024-09-13T14:32:55.220 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:55 smithi154 systemd[1]: Starting Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:32:55.656 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:55 smithi154 podman[97568]: 2024-09-13 14:32:55.182553891 +0000 UTC m=+0.016472547 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-13T14:32:55.959 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:55 smithi154 podman[97568]: 2024-09-13 14:32:55.655796556 +0000 UTC m=+0.489715211 volume create 4e4e4eaf54cf399fb58c1b7cdfb8b9c98f9ea87184b3f22dd11c1c62ecfe86b7 2024-09-13T14:32:55.959 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:55 smithi154 podman[97568]: 2024-09-13 14:32:55.772919385 +0000 UTC m=+0.606838086 container create 4971176d5c4c08bc19e4a2410b51a6bca7643fb47425a3e9849cd0fd14c06fd2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:56.378 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 podman[97568]: 2024-09-13 14:32:56.084061557 +0000 UTC m=+0.917980226 container init 4971176d5c4c08bc19e4a2410b51a6bca7643fb47425a3e9849cd0fd14c06fd2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:56.378 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 podman[97568]: 2024-09-13 14:32:56.088004417 +0000 UTC m=+0.921923075 container start 4971176d5c4c08bc19e4a2410b51a6bca7643fb47425a3e9849cd0fd14c06fd2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T14:32:56.378 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.099Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-13T14:32:56.378 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.099Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-13T14:32:56.378 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.100Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.154 port=9094 2024-09-13T14:32:56.378 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.102Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-13T14:32:56.379 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.124Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:32:56.379 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.124Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-13T14:32:56.379 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.126Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-13T14:32:56.379 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:56.126Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-13T14:32:56.379 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 bash[97568]: 4971176d5c4c08bc19e4a2410b51a6bca7643fb47425a3e9849cd0fd14c06fd2 2024-09-13T14:32:56.379 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:56 smithi154 systemd[1]: Started Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:32:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[29001]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:56.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[39074]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:56.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:56.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:56.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:56 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:56.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:56 smithi172 ceph-mon[32794]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:56.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:56 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:56 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:56 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:32:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:57 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:57 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:57.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:57 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:57 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:57.914 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:57 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:57.914 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:57 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:32:58.447 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:32:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:32:58.102Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000481311s 2024-09-13T14:32:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:32:58 smithi154 ceph-mon[29001]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:58.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:32:58 smithi154 ceph-mon[39074]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:32:58.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:32:58 smithi172 ceph-mon[32794]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[29001]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[39074]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.711 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.712 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.712 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:00 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:00 smithi172 ceph-mon[32794]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:00 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:00 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:00 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:00 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:01.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:01.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:01.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:01.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:01.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:33:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:33:01] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:33:02.863 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:02 smithi154 ceph-mon[29001]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:02.864 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:02 smithi154 ceph-mon[39074]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:02.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:02 smithi172 ceph-mon[32794]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:04 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:04.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:04.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:04 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:06.470 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T14:33:06.106Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004243129s 2024-09-13T14:33:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[29001]: Upgrade: Updating grafana.a 2024-09-13T14:33:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[29001]: Deploying daemon grafana.a on smithi172 2024-09-13T14:33:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[29001]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[29001]: mgrmap e33: y(active, since 92s), standbys: x 2024-09-13T14:33:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[39074]: Upgrade: Updating grafana.a 2024-09-13T14:33:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[39074]: Deploying daemon grafana.a on smithi172 2024-09-13T14:33:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[39074]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:06 smithi154 ceph-mon[39074]: mgrmap e33: y(active, since 92s), standbys: x 2024-09-13T14:33:06.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:06 smithi172 ceph-mon[32794]: Upgrade: Updating grafana.a 2024-09-13T14:33:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:06 smithi172 ceph-mon[32794]: Deploying daemon grafana.a on smithi172 2024-09-13T14:33:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:06 smithi172 ceph-mon[32794]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:06 smithi172 ceph-mon[32794]: mgrmap e33: y(active, since 92s), standbys: x 2024-09-13T14:33:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:08 smithi154 ceph-mon[29001]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:08.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:08 smithi154 ceph-mon[39074]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:08.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:08 smithi172 ceph-mon[32794]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:10 smithi154 ceph-mon[29001]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:10.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:10 smithi154 ceph-mon[39074]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:10.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:10 smithi172 ceph-mon[32794]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:33:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:33:11] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:33:12.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:12 smithi154 ceph-mon[39074]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:12 smithi154 ceph-mon[29001]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:12.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:12 smithi172 ceph-mon[32794]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:14.427 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:14 smithi172 ceph-mon[32794]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:14 smithi154 ceph-mon[29001]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:14.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:14 smithi154 ceph-mon[39074]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:16 smithi172 ceph-mon[32794]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:16 smithi154 ceph-mon[29001]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:16.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:16 smithi154 ceph-mon[39074]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:33:16.904 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:16 smithi172 systemd[1]: Stopping Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:33:17.193 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:16 smithi172 podman[91269]: 2024-09-13 14:33:16.903123406 +0000 UTC m=+0.136668613 container died 9507837610ffa2ee36997c9564a119383e7cd9f58072197292bce9edcf6c88ac (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, maintainer=Paul Cuzner , distribution-scope=public, release=239.1651231664, vcs-type=git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-13T14:33:17.399 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:33:17.490 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 podman[91269]: 2024-09-13 14:33:17.259248357 +0000 UTC m=+0.492793564 container cleanup 9507837610ffa2ee36997c9564a119383e7cd9f58072197292bce9edcf6c88ac (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, version=8.5, name=ubi8, description=Ceph Grafana Container, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git) 2024-09-13T14:33:17.491 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 bash[91269]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a 2024-09-13T14:33:17.925 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 podman[91283]: 2024-09-13 14:33:17.577526763 +0000 UTC m=+0.673163236 container remove 9507837610ffa2ee36997c9564a119383e7cd9f58072197292bce9edcf6c88ac (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, vcs-type=git, distribution-scope=public, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, architecture=x86_64, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, io.buildah.version=1.32.0) 2024-09-13T14:33:17.925 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T14:33:17.925 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-13T14:33:17.925 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 systemd[1]: Stopped Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:33:17.926 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@grafana.a.service: Consumed 7.478s CPU time. 2024-09-13T14:33:17.954 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:33:17.954 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (22s) 18s ago 6m 16.5M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:33:17.954 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (5m) 17s ago 5m 86.7M - 9.4.7 954c08fa6188 9507837610ff 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (4m) 18s ago 4m 17.3M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (4m) 17s ago 4m 16.3M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (85s) 17s ago 9m 452M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (112s) 18s ago 11m 510M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (11m) 18s ago 11m 53.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (10m) 17s ago 10m 46.4M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (10m) 18s ago 10m 43.1M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (71s) 18s ago 5m 9.96M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (65s) 17s ago 5m 11.0M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (9m) 18s ago 9m 52.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (9m) 18s ago 9m 61.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:33:17.955 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (8m) 18s ago 8m 56.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:33:17.956 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (8m) 18s ago 8m 52.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:33:17.956 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (8m) 17s ago 8m 54.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:33:17.956 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (7m) 17s ago 7m 55.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:33:17.956 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (7m) 17s ago 7m 54.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:33:17.956 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (6m) 17s ago 6m 54.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:33:17.956 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (42s) 17s ago 6m 35.0M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:33:18.240 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:17 smithi172 systemd[1]: Starting Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:33:18.241 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 podman[91408]: 2024-09-13 14:33:18.128478365 +0000 UTC m=+0.111903908 container create 4bed2859671631cd5728c3df744e1b3622445dc1abcdb69a58ee7e4867892335 (image=quay.io/ceph/grafana:10.4.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T14:33:18.241 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 podman[91408]: 2024-09-13 14:33:18.033216615 +0000 UTC m=+0.016642167 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-09-13T14:33:18.289 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:33:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:33:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:33:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:33:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:33:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:33:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:33:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:33:18.291 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:33:18.515 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-mon[32794]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:18.515 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-mon[32794]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:18.515 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-mon[32794]: from='client.25664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:18.515 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/923305801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:18.516 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 podman[91408]: 2024-09-13 14:33:18.372916306 +0000 UTC m=+0.356341853 container init 4bed2859671631cd5728c3df744e1b3622445dc1abcdb69a58ee7e4867892335 (image=quay.io/ceph/grafana:10.4.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T14:33:18.516 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 podman[91408]: 2024-09-13 14:33:18.375911615 +0000 UTC m=+0.359337159 container start 4bed2859671631cd5728c3df744e1b3622445dc1abcdb69a58ee7e4867892335 (image=quay.io/ceph/grafana:10.4.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T14:33:18.516 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 bash[91408]: 4bed2859671631cd5728c3df744e1b3622445dc1abcdb69a58ee7e4867892335 2024-09-13T14:33:18.516 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 systemd[1]: Started Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:33:18.516 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502262516Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-09-13T14:33:18Z 2024-09-13T14:33:18.516 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502499322Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502507279Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502510895Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502513738Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502516956Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502519724Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502523654Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502527035Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502529731Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502532748Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502535295Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-13T14:33:18.517 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.50253798Z level=info msg=Target target=[all] 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.50254422Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502546738Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.50254925Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502551656Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.50255423Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=settings t=2024-09-13T14:33:18.502556745Z level=info msg="App mode production" 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=sqlstore t=2024-09-13T14:33:18.502843887Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=sqlstore t=2024-09-13T14:33:18.502857841Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.503313168Z level=info msg="Starting DB migrations" 2024-09-13T14:33:18.518 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.514843242Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-13T14:33:18.590 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [ 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr" 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "2/2 daemons upgraded", 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading grafana daemons", 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:33:18.591 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[29001]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[29001]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[29001]: from='client.25664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/923305801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[39074]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[39074]: from='client.15771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[39074]: from='client.25664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:18.592 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:18 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/923305801' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:18.767 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.520864897Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=6.01576ms 2024-09-13T14:33:18.767 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.521383211Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-13T14:33:18.767 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.521546645Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=162.297µs 2024-09-13T14:33:18.767 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.52197784Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-13T14:33:18.767 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.522756498Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=778.838µs 2024-09-13T14:33:18.767 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.524932843Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-13T14:33:18.767 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.529981378Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=5.048275ms 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.532322452Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.53768939Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=5.367127ms 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.539892864Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.545020245Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=5.126403ms 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.545419257Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.545560333Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=139.171µs 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.545977394Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.546751257Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=773.392µs 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.548506235Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-13T14:33:18.768 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.549216968Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=708.23µs 2024-09-13T14:33:18.769 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.549608376Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-13T14:33:18.770 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.555021238Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=5.411586ms 2024-09-13T14:33:18.770 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.55545802Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-13T14:33:18.770 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.560517145Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=5.058091ms 2024-09-13T14:33:18.770 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.560905386Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-13T14:33:18.770 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.560946622Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=41.02µs 2024-09-13T14:33:18.770 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.561701887Z level=info msg="Executing migration" id="permission kind migration" 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.566828263Z level=info msg="Migration successfully executed" id="permission kind migration" duration=5.125494ms 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.56719623Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.572432733Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=5.236471ms 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.572819419Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.578342255Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=5.522759ms 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.578725125Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.579534623Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=809.815µs 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.579917179Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.580704989Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=788.019µs 2024-09-13T14:33:18.771 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.581064065Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.581800915Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=735.437µs 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.58214578Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.582169975Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=23.517µs 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.582559425Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.5832782Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=718.79µs 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.583636677Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.5843395Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=701.242µs 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.584702061Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.600289516Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=15.585531ms 2024-09-13T14:33:18.772 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.600771106Z level=info msg="Executing migration" id="create correlation v2" 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.601507987Z level=info msg="Migration successfully executed" id="create correlation v2" duration=737.663µs 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.60189605Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.602608364Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=712.243µs 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.602971194Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.603660177Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=688.348µs 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.604040939Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.604739244Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=698.066µs 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.605101369Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.605277899Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=176.405µs 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.605893055Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-13T14:33:18.773 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.606469943Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=575.414µs 2024-09-13T14:33:18.774 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.606810974Z level=info msg="Executing migration" id="add provisioning column" 2024-09-13T14:33:18.774 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.612032769Z level=info msg="Migration successfully executed" id="add provisioning column" duration=5.220975ms 2024-09-13T14:33:18.774 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.612446962Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-13T14:33:18.774 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.612588745Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=141.446µs 2024-09-13T14:33:18.774 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.613210695Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-13T14:33:18.776 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.613254722Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=43.71µs 2024-09-13T14:33:18.776 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.613981437Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-13T14:33:18.776 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.619540182Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.557665ms 2024-09-13T14:33:18.776 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.619915023Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-13T14:33:18.776 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.620120091Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=203.654µs 2024-09-13T14:33:18.776 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.620695822Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.620815474Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=119.79µs 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.621497573Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.622436068Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=938.01µs 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.622797255Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.623496844Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=698.968µs 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.623836468Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.624602908Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=766.057µs 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.624959087Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-13T14:33:18.777 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.625435746Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=476.523µs 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.625778397Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.625978033Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=199.622µs 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.626585632Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.627292621Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=706.917µs 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.62762599Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.628323512Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=697.3µs 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.628658959Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.629325085Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=665.991µs 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.629655961Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-13T14:33:18.778 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.630366201Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=709.841µs 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.630709546Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.631376771Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=667.129µs 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.631754323Z level=info msg="Executing migration" id="create anon_device table" 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.63231643Z level=info msg="Migration successfully executed" id="create anon_device table" duration=561.748µs 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.634277123Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.63498763Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=710.339µs 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.636934058Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.637612069Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=677.915µs 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.639667418Z level=info msg="Executing migration" id="create signing_key table" 2024-09-13T14:33:18.779 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.640266689Z level=info msg="Migration successfully executed" id="create signing_key table" duration=599.032µs 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.640638785Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.64132529Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=686.478µs 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.643340249Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.644040519Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=700.23µs 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.645988788Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.646422399Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=431.783µs 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.648617444Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.65389107Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.273032ms 2024-09-13T14:33:18.780 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.654257324Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-13T14:33:18.781 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.655985606Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=1.729016ms 2024-09-13T14:33:18.781 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.657142877Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-13T14:33:18.781 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.658065613Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=922.528µs 2024-09-13T14:33:18.781 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.659707817Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-13T14:33:18.781 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.660441067Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=732.955µs 2024-09-13T14:33:18.781 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.66078271Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.661484129Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=701.351µs 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.661844325Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.66272236Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=878.127µs 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.664511176Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.665347314Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=836.072µs 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.66719643Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.667966178Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=771.624µs 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.669917268Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.670617034Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=700.659µs 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.6709752Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-13T14:33:18.782 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.671376605Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=401.724µs 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.673304198Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.673346907Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=43.438µs 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.675816251Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.681284447Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.468328ms 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.681657047Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.687054308Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.397249ms 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.687435629Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.687683395Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=254.322µs 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=migrator t=2024-09-13T14:33:18.689804702Z level=info msg="migrations completed" performed=65 skipped=482 duration=175.034416ms 2024-09-13T14:33:18.783 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=sqlstore t=2024-09-13T14:33:18.690518297Z level=info msg="Created default organization" 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=secrets t=2024-09-13T14:33:18.69269796Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=plugin.store t=2024-09-13T14:33:18.708550137Z level=info msg="Loading plugins..." 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=local.finder t=2024-09-13T14:33:18.75260144Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=plugin.store t=2024-09-13T14:33:18.752644306Z level=info msg="Plugins loaded" count=55 duration=44.111898ms 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=query_data t=2024-09-13T14:33:18.754712712Z level=info msg="Query Service initialization" 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=live.push_http t=2024-09-13T14:33:18.757387734Z level=info msg="Live Push Gateway initialization" 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ngalert.migration t=2024-09-13T14:33:18.758819612Z level=info msg=Starting 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ngalert.state.manager t=2024-09-13T14:33:18.76650822Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-13T14:33:18.784 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=infra.usagestats.collector t=2024-09-13T14:33:18.767960228Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-13T14:33:19.201 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=provisioning.datasources t=2024-09-13T14:33:18.770577687Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-13T14:33:19.201 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=provisioning.datasources t=2024-09-13T14:33:18.770864881Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-13T14:33:19.201 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=provisioning.alerting t=2024-09-13T14:33:18.808789323Z level=info msg="starting to provision alerting" 2024-09-13T14:33:19.201 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=provisioning.alerting t=2024-09-13T14:33:18.808813049Z level=info msg="finished to provision alerting" 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafanaStorageLogger t=2024-09-13T14:33:18.809031364Z level=info msg="Storage starting" 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ngalert.multiorg.alertmanager t=2024-09-13T14:33:18.809964725Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ngalert.state.manager t=2024-09-13T14:33:18.809960589Z level=info msg="Warming state cache for startup" 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=provisioning.dashboard t=2024-09-13T14:33:18.810053469Z level=info msg="starting to provision dashboards" 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ngalert.state.manager t=2024-09-13T14:33:18.81033447Z level=info msg="State cache has been initialized" states=0 duration=373.378µs 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ngalert.scheduler t=2024-09-13T14:33:18.810396251Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ticker t=2024-09-13T14:33:18.81048652Z level=info msg=starting first_tick=2024-09-13T14:33:20Z 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=http.server t=2024-09-13T14:33:18.812460588Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=http.server t=2024-09-13T14:33:18.81295123Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=plugins.update.checker t=2024-09-13T14:33:18.885159632Z level=info msg="Update check succeeded" duration=75.598589ms 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana.update.checker t=2024-09-13T14:33:18.895828642Z level=info msg="Update check succeeded" duration=86.898914ms 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=sqlstore.transactions t=2024-09-13T14:33:18.907349312Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:33:19.202 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=sqlstore.transactions t=2024-09-13T14:33:18.9181099Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:33:19.203 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=provisioning.dashboard t=2024-09-13T14:33:19.010602742Z level=info msg="finished to provision dashboards" 2024-09-13T14:33:19.203 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana-apiserver t=2024-09-13T14:33:19.199899512Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-13T14:33:19.203 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana-apiserver t=2024-09-13T14:33:19.200485428Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-13T14:33:19.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-mon[32794]: from='client.25670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-mon[32794]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:19 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:19.871 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[29001]: from='client.25670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:19.871 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[29001]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[39074]: from='client.25670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[39074]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:19.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:19 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:20.877 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:20 smithi172 ceph-mon[32794]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:20 smithi154 ceph-mon[29001]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:20 smithi154 ceph-mon[39074]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:33:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:33:21] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:33:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:22 smithi172 ceph-mon[32794]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:22 smithi154 ceph-mon[29001]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:22.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:22 smithi154 ceph-mon[39074]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:24.204 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:23 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.204 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:23 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.204 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:23 smithi172 ceph-mon[32794]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:24.204 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:23 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.204 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:23 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[29001]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[39074]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:23 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:26.665 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:26 smithi172 ceph-mon[32794]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:26 smithi154 ceph-mon[29001]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:26.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:26 smithi154 ceph-mon[39074]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:33:27.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:33:27.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:33:27.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:33:27.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:33:27.980 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:33:27.980 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:33:27.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:33:27.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:33:27.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:33:27.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:33:27.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:27 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:33:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:33:27.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:33:27.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:33:27.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:33:27.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:33:27.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:33:27.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:33:27.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:33:27.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:27.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:27.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:27.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:27 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[29001]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:33:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[29001]: Upgrade: Finalizing container_image settings 2024-09-13T14:33:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[29001]: Upgrade: Complete! 2024-09-13T14:33:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[29001]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:33:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[39074]: Upgrade: Finalizing container_image settings 2024-09-13T14:33:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[39074]: Upgrade: Complete! 2024-09-13T14:33:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:28 smithi154 ceph-mon[39074]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:28.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:28 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T14:33:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:28 smithi172 ceph-mon[32794]: Upgrade: Finalizing container_image settings 2024-09-13T14:33:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:28 smithi172 ceph-mon[32794]: Upgrade: Complete! 2024-09-13T14:33:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:28 smithi172 ceph-mon[32794]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:30.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:29 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:30.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:29 smithi154 ceph-mon[29001]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:30.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:29 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:30.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:29 smithi154 ceph-mon[39074]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:30.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:29 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:29 smithi172 ceph-mon[32794]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:33:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:33:31] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:33:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:32 smithi154 ceph-mon[29001]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:32.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:32 smithi154 ceph-mon[39074]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:32.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:32 smithi172 ceph-mon[32794]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:34.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:34 smithi154 ceph-mon[29001]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:34.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:34 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:34 smithi154 ceph-mon[39074]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:34 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:34 smithi172 ceph-mon[32794]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:34 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:36 smithi154 ceph-mon[29001]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:36 smithi154 ceph-mon[39074]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:36 smithi172 ceph-mon[32794]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:38 smithi154 ceph-mon[29001]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:38 smithi154 ceph-mon[39074]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:38 smithi172 ceph-mon[32794]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:40 smithi154 ceph-mon[29001]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:40.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:40 smithi154 ceph-mon[39074]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:40.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:40 smithi172 ceph-mon[32794]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:33:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:33:41] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:33:42.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:42 smithi154 ceph-mon[29001]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:42.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:42 smithi154 ceph-mon[39074]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:42.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:42 smithi172 ceph-mon[32794]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:44 smithi154 ceph-mon[29001]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:44.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:44 smithi154 ceph-mon[39074]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:44.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:44 smithi172 ceph-mon[32794]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:46 smithi154 ceph-mon[29001]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:46.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:46 smithi154 ceph-mon[39074]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:46.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:46 smithi172 ceph-mon[32794]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:48 smithi154 ceph-mon[29001]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:48.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:48 smithi154 ceph-mon[39074]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:48 smithi172 ceph-mon[32794]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:49 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:49.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:49 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:49.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-13T14:33:49.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:49 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:33:49.828 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:50 smithi154 ceph-mon[39074]: from='client.25688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:50 smithi154 ceph-mon[39074]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:50 smithi154 ceph-mon[29001]: from='client.25688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:50 smithi154 ceph-mon[29001]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:50.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:50 smithi172 ceph-mon[32794]: from='client.25688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:50 smithi172 ceph-mon[32794]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:51.554 INFO:teuthology.orchestra.run.smithi154.stdout:wait for minority of mons to upgrade 2024-09-13T14:33:52.138 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:33:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:33:51] "GET /metrics HTTP/1.1" 200 33830 "" "Prometheus/2.51.0" 2024-09-13T14:33:52.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-09-13T14:33:52.342 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:52 smithi154 ceph-mon[29001]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:52.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:52 smithi154 ceph-mon[39074]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:52 smithi172 ceph-mon[32794]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:54.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:54 smithi154 ceph-mon[29001]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:54.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:54 smithi154 ceph-mon[39074]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:54.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:54 smithi172 ceph-mon[32794]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:55.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:55 smithi154 ceph-mon[29001]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:55.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:55 smithi154 ceph-mon[39074]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:55.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:55 smithi172 ceph-mon[32794]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:33:56.033 INFO:teuthology.orchestra.run.smithi154.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[29001]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[39074]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:56.642 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:56.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:56.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:56.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:56 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:56.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:33:56.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:56 smithi172 ceph-mon[32794]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:56 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:56 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:33:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:56 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:33:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:56 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:33:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:56 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:33:56.874 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:33:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:57 smithi154 ceph-mon[29001]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:33:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:57 smithi154 ceph-mon[29001]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:33:57.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:57 smithi154 ceph-mon[39074]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:33:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:57 smithi154 ceph-mon[39074]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:33:57.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:57 smithi172 ceph-mon[32794]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:33:57.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:57 smithi172 ceph-mon[32794]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:33:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:33:58 smithi154 ceph-mon[29001]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:58.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:33:58 smithi154 ceph-mon[39074]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:33:58.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:33:58 smithi172 ceph-mon[32794]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:00.050 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (65s) 61s ago 6m 16.5M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (42s) 37s ago 6m 51.5M - c8b91775d855 4bed28596716 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (5m) 61s ago 5m 17.3M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (5m) 37s ago 5m 16.4M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (2m) 37s ago 10m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (2m) 61s ago 12m 510M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (12m) 61s ago 12m 53.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e d8d448f265b6 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (11m) 37s ago 11m 47.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (10m) 61s ago 10m 43.1M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e a87978af9c40 2024-09-13T14:34:00.624 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (114s) 61s ago 6m 9.96M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (107s) 37s ago 6m 11.0M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (10m) 61s ago 10m 52.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (10m) 61s ago 9m 61.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (9m) 61s ago 9m 56.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (9m) 61s ago 9m 52.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (8m) 37s ago 8m 55.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (8m) 37s ago 8m 55.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (8m) 37s ago 8m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (7m) 37s ago 7m 54.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:34:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (85s) 37s ago 7m 36.6M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: Upgrade: Setting container_image for all mgr 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:34:00.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[29001]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: Upgrade: Setting container_image for all mgr 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:34:00.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:00 smithi154 ceph-mon[39074]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all mgr 2024-09-13T14:34:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:00.743 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:34:00.743 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-13T14:34:00.743 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:00 smithi172 ceph-mon[32794]: Upgrade: It appears safe to stop mon.a 2024-09-13T14:34:00.959 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 3 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:00.960 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:34:00.961 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 13, 2024-09-13T14:34:00.961 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:34:00.961 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:34:00.961 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:34:01.264 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:34:01.264 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:34:01.265 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:34:01.265 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-09-13T14:34:01.265 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:34:01.265 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "0/3 daemons upgraded", 2024-09-13T14:34:01.265 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:34:01.265 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:34:01.265 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:34:01.515 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.515 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='client.25706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.515 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.516 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='client.? 172.21.15.154:0/199800997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:01.516 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:01.516 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:34:01.516 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:34:01.516 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[29001]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:01.516 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.517 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='client.25706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.517 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.517 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='client.? 172.21.15.154:0/199800997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:01.517 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:01.517 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:34:01.517 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:34:01.517 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='client.25706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='client.15825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/199800997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='mgr.y' 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:34:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:01 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:34:01] "GET /metrics HTTP/1.1" 200 33830 "" "Prometheus/2.51.0" 2024-09-13T14:34:02.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:02 smithi172 ceph-mon[32794]: Upgrade: Updating mon.a 2024-09-13T14:34:02.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:02 smithi172 ceph-mon[32794]: Deploying daemon mon.a on smithi154 2024-09-13T14:34:02.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:02 smithi172 ceph-mon[32794]: from='client.25724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:02.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:02 smithi172 ceph-mon[32794]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[29001]: Upgrade: Updating mon.a 2024-09-13T14:34:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[29001]: Deploying daemon mon.a on smithi154 2024-09-13T14:34:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[29001]: from='client.25724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[29001]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[39074]: Upgrade: Updating mon.a 2024-09-13T14:34:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[39074]: Deploying daemon mon.a on smithi154 2024-09-13T14:34:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[39074]: from='client.25724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-mon[39074]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:02.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:02 smithi154 systemd[1]: Stopping Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:34:03.343 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[28997]: 2024-09-13T14:34:02.992+0000 7fd44ad28640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:34:03.343 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[28997]: 2024-09-13T14:34:02.992+0000 7fd44ad28640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-13T14:34:03.343 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:03 smithi154 podman[102157]: 2024-09-13 14:34:03.043567939 +0000 UTC m=+0.156512684 container died d8d448f265b63d0d0361fbb76de9c445c0acf7cb21d3bb1ba45b8b5142d56317 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, ceph=True, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-13T14:34:03.693 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:03 smithi154 podman[102157]: 2024-09-13 14:34:03.433594472 +0000 UTC m=+0.546539192 container cleanup d8d448f265b63d0d0361fbb76de9c445c0acf7cb21d3bb1ba45b8b5142d56317 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS) 2024-09-13T14:34:03.693 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:03 smithi154 bash[102157]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a 2024-09-13T14:34:03.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:03 smithi154 podman[102170]: 2024-09-13 14:34:03.693274678 +0000 UTC m=+0.653191681 container remove d8d448f265b63d0d0361fbb76de9c445c0acf7cb21d3bb1ba45b8b5142d56317 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-09-13T14:34:03.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:03 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T14:34:03.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:03 smithi154 systemd[1]: Stopped Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:34:03.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:03 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service: Consumed 12.922s CPU time. 2024-09-13T14:34:04.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 systemd[1]: Starting Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:34:04.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 podman[102295]: 2024-09-13 14:34:04.146813504 +0000 UTC m=+0.016102034 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:34:04.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 podman[102295]: 2024-09-13 14:34:04.255157495 +0000 UTC m=+0.124446067 container create c1bb51eb0c909c7900de8536e457b99383828a0a8794dc4d8d534fc5746b3065 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-13T14:34:04.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 podman[102295]: 2024-09-13 14:34:04.52267902 +0000 UTC m=+0.391967556 container init c1bb51eb0c909c7900de8536e457b99383828a0a8794dc4d8d534fc5746b3065 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-13T14:34:04.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 podman[102295]: 2024-09-13 14:34:04.525843805 +0000 UTC m=+0.395132341 container start c1bb51eb0c909c7900de8536e457b99383828a0a8794dc4d8d534fc5746b3065 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T14:34:04.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:34:04.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 2 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: pidfile_write: ignore empty --pid-file 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: load: jerasure load: lrc 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Git sha 0 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: DB SUMMARY 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: DB Session ID: OE5P2UEJICLUGGGN66FW 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1518 Bytes 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000032.sst 2024-09-13T14:34:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000030.log size: 1719632 ; 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.env: 0x5595238db160 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.info_log: 0x559524185880 2024-09-13T14:34:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:34:04.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.statistics: (nil) 2024-09-13T14:34:04.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.use_fsync: 0 2024-09-13T14:34:04.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:34:04.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:34:04.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:34:04.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.db_log_dir: 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.wal_dir: 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:34:04.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.write_buffer_manager: 0x559524188960 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:34:04.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.unordered_write: 0 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.row_cache: None 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.wal_filter: None 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:34:04.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.wal_compression: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:34:04.978 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:34:04.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_open_files: -1 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Compression algorithms supported: 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kZSTD supported: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:34:04.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.merge_operator: 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_filter: None 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:34:04.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5595241854e0) 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: index_type: 0 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: data_block_index_type: 0 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: index_shortening: 1 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: checksum: 4 2024-09-13T14:34:04.982 INFO:journalctl@ceph.mon.a.smithi154.stdout: no_block_cache: 0 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache: 0x559524178850 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache_options: 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: capacity : 536870912 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: num_shard_bits : 4 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: strict_capacity_limit : 0 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_cache_compressed: (nil) 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: persistent_cache: (nil) 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_size: 4096 2024-09-13T14:34:04.983 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_size_deviation: 10 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_restart_interval: 16 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: index_block_restart_interval: 1 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: metadata_block_size: 4096 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: partition_filters: 0 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: use_delta_encoding: 1 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: filter_policy: bloomfilter 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: whole_key_filtering: 1 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: verify_compression: 0 2024-09-13T14:34:04.984 INFO:journalctl@ceph.mon.a.smithi154.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout: format_version: 5 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout: enable_index_compression: 1 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout: block_align: 0 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout: max_auto_readahead_size: 262144 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout: prepopulate_block_cache: 0 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression: NoCompression 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:34:04.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.num_levels: 7 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:34:04.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:34:04.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:34:04.988 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:34:04.989 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:34:04.990 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.ttl: 2592000 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:34:04.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12322, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 296d8068-e1f6-4def-acd2-8bd759767a90 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238044552839, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238044560622, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 1541526, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12323, "largest_seqno": 13631, "table_properties": {"data_size": 1535221, "index_size": 3728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1541, "raw_key_size": 14650, "raw_average_key_size": 24, "raw_value_size": 1522232, "raw_average_value_size": 2507, "num_data_blocks": 170, "num_entries": 607, "num_filter_entries": 607, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726238044, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "296d8068-e1f6-4def-acd2-8bd759767a90", "db_session_id": "OE5P2UEJICLUGGGN66FW", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238044560719, "job": 1, "event": "recovery_finished"} 2024-09-13T14:34:04.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5595241a8e00 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: DB pointer 0x5595242a8000 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** DB Stats ** 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:34:04.993 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** Compaction Stats [default] ** 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: L0 1/0 1.47 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 438.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: L6 1/0 7.64 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Sum 2/0 9.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 438.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:34:04.994 INFO:journalctl@ceph.mon.a.smithi154.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 438.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** Compaction Stats [default] ** 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 438.6 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:34:04.995 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: Cumulative compaction: 0.00 GB write, 117.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: Interval compaction: 0.00 GB write, 117.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: Block cache BinnedLRUCache@0x559524178850#2 capacity: 512.00 MB usage: 5.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: starting mon.a rank 0 at public addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] at bind addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???) e3 preinit fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???).mds e9 new map 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???).mds e9 print_map 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: e9 2024-09-13T14:34:04.996 INFO:journalctl@ceph.mon.a.smithi154.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: legacy client fscid: 1 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: Filesystem 'foo' (1) 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: fs_name foo 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: epoch 8 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: created 2024-09-13T14:28:14.301827+0000 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: modified 2024-09-13T14:28:28.364158+0000 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: tableserver 0 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: root 0 2024-09-13T14:34:04.997 INFO:journalctl@ceph.mon.a.smithi154.stdout: session_timeout 60 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: session_autoclose 300 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: max_file_size 1099511627776 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: max_xattr_size 65536 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: required_client_features {} 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: last_failure 0 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: last_failure_osd_epoch 0 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: max_mds 1 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: in 0 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: up {0=24539} 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: failed 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: damaged 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: stopped 2024-09-13T14:34:04.998 INFO:journalctl@ceph.mon.a.smithi154.stdout: data_pools [4] 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: metadata_pool 3 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: inline_data disabled 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: balancer 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: bal_rank_mask -1 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: standby_count_wanted 1 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: [mds.foo.smithi154.daekzd{0:24539} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: Standby daemons: 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout: [mds.foo.smithi172.pmmovd{-1:24472} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:34:04.999 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???).osd e88 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:34:05.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???).osd e88 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:34:05.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???).osd e88 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:34:05.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???).osd e88 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:34:05.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 ceph-mon[102314]: mon.a@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-09-13T14:34:05.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 bash[102295]: c1bb51eb0c909c7900de8536e457b99383828a0a8794dc4d8d534fc5746b3065 2024-09-13T14:34:05.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:04 smithi154 systemd[1]: Started Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:34:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: mon.a calling monitor election 2024-09-13T14:34:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:34:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: monmap epoch 3 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: last_changed 2024-09-13T14:23:04.765616+0000 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: min_mon_release 18 (reef) 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: election_strategy: 1 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: mgrmap e33: y(active, since 2m), standbys: x 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: overall HEALTH_OK 2024-09-13T14:34:07.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: from='mgr.25520 ' entity='' 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[102314]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: mon.a calling monitor election 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: monmap epoch 3 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:07.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: last_changed 2024-09-13T14:23:04.765616+0000 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: min_mon_release 18 (reef) 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: election_strategy: 1 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: mgrmap e33: y(active, since 2m), standbys: x 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: overall HEALTH_OK 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: from='mgr.25520 ' entity='' 2024-09-13T14:34:07.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mon[39074]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-13T14:34:07.975 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ignoring --setuser ceph since I am not root 2024-09-13T14:34:07.975 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ignoring --setgroup ceph since I am not root 2024-09-13T14:34:07.975 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:07 smithi154 ceph-mgr[84634]: -- 172.21.15.154:0/2594291995 <== mon.2 v2:172.21.15.154:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cded5d10e0 con 0x55cded5af400 2024-09-13T14:34:07.990 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: ignoring --setuser ceph since I am not root 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: ignoring --setgroup ceph since I am not root 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mgr[78342]: -- 172.21.15.172:0/460960839 <== mon.1 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584b841b0e0 con 0x5584b83f9000 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: from='mgr.25520 172.21.15.154:0/3957573787' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: mon.a calling monitor election 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: monmap epoch 3 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: last_changed 2024-09-13T14:23:04.765616+0000 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:34:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: min_mon_release 18 (reef) 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: election_strategy: 1 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: osdmap e88: 8 total, 8 up, 8 in 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: mgrmap e33: y(active, since 2m), standbys: x 2024-09-13T14:34:07.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: overall HEALTH_OK 2024-09-13T14:34:07.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: from='mgr.25520 ' entity='' 2024-09-13T14:34:07.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:07 smithi172 ceph-mon[32794]: mgrmap e34: y(active, since 2m), standbys: x 2024-09-13T14:34:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: Standby manager daemon x restarted 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: Standby manager daemon x started 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: Active manager daemon y restarted 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: Activating manager daemon y 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: mgrmap e35: y(active, starting, since 0.00606089s), standbys: x 2024-09-13T14:34:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:34:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[102314]: Manager daemon y is now available 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: Standby manager daemon x restarted 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: Standby manager daemon x started 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:34:12.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: Active manager daemon y restarted 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: Activating manager daemon y 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: mgrmap e35: y(active, starting, since 0.00606089s), standbys: x 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:34:12.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:34:12.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-mon[39074]: Manager daemon y is now available 2024-09-13T14:34:12.990 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: [13/Sep/2024:14:34:12] ENGINE Bus STARTING 2024-09-13T14:34:12.990 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: CherryPy Checker: 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: The Application mounted at '' has an empty config. 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: [13/Sep/2024:14:34:12] ENGINE Serving on http://:::9283 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x[78338]: [13/Sep/2024:14:34:12] ENGINE Bus STARTED 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: Standby manager daemon x restarted 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: Standby manager daemon x started 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-13T14:34:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.? 172.21.15.172:0/3781025496' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: Active manager daemon y restarted 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: Activating manager daemon y 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: mgrmap e35: y(active, starting, since 0.00606089s), standbys: x 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi172.pmmovd"}]: dispatch 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi154.daekzd"}]: dispatch 2024-09-13T14:34:12.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T14:34:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:12 smithi172 ceph-mon[32794]: Manager daemon y is now available 2024-09-13T14:34:13.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.990+0000 7f9d4a04a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d4a04a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d4a04a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d4a04a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d4a04a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d47845640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d47845640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d47845640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d47845640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:12.991+0000 7f9d47845640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-13T14:34:13.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:13.656+0000 7f9d7ab79640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-13T14:34:13.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:13] ENGINE Bus STARTING 2024-09-13T14:34:13.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: CherryPy Checker: 2024-09-13T14:34:13.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: The Application mounted at '' has an empty config. 2024-09-13T14:34:13.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:13] ENGINE Serving on http://:::9283 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:13] ENGINE Bus STARTED 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:34:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: mgrmap e36: y(active, since 1.01414s), standbys: x 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[102314]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:34:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: mgrmap e36: y(active, since 1.01414s), standbys: x 2024-09-13T14:34:13.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:34:13.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:13 smithi154 ceph-mon[39074]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: mgrmap e36: y(active, since 1.01414s), standbys: x 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-13T14:34:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:13 smithi172 ceph-mon[32794]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:14.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:14.102+0000 7f9d28e4e640 -1 log_channel(cephadm) log [ERR] : [13/Sep/2024:14:34:14] ENGINE Error in HTTPServer.serve 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: Traceback (most recent call last): 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self._connections.run(self.expiration_interval) 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self._run(expiration_interval) 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: s = self.context.wrap_socket( 2024-09-13T14:34:14.471 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:34:14.472 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: return self.sslsocket_class._create( 2024-09-13T14:34:14.472 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:34:14.472 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self.do_handshake() 2024-09-13T14:34:14.472 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:34:14.472 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: self._sslobj.do_handshake() 2024-09-13T14:34:14.472 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:34:14.472 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:14 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:15 smithi172 ceph-mon[32794]: [13/Sep/2024:14:34:13] ENGINE Bus STARTING 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:15 smithi172 ceph-mon[32794]: [13/Sep/2024:14:34:13] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:15 smithi172 ceph-mon[32794]: [13/Sep/2024:14:34:14] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:15 smithi172 ceph-mon[32794]: [13/Sep/2024:14:34:14] ENGINE Bus STARTED 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:15 smithi172 ceph-mon[32794]: [13/Sep/2024:14:34:14] ENGINE Error in HTTPServer.serve 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:34:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._run(expiration_interval) 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: s = self.context.wrap_socket( 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.sslsocket_class._create( 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: self.do_handshake() 2024-09-13T14:34:15.742 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:34:15.743 INFO:journalctl@ceph.mon.b.smithi172.stdout: self._sslobj.do_handshake() 2024-09-13T14:34:15.743 INFO:journalctl@ceph.mon.b.smithi172.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:34:15.743 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:15 smithi172 ceph-mon[32794]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:15.743 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:15 smithi172 ceph-mon[32794]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-13T14:34:15.911 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[102314]: [13/Sep/2024:14:34:13] ENGINE Bus STARTING 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[102314]: [13/Sep/2024:14:34:13] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[102314]: [13/Sep/2024:14:34:14] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[102314]: [13/Sep/2024:14:34:14] ENGINE Bus STARTED 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[102314]: [13/Sep/2024:14:34:14] ENGINE Error in HTTPServer.serve 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:34:15.912 INFO:journalctl@ceph.mon.a.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: self.do_handshake() 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[102314]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:15.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[102314]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[39074]: [13/Sep/2024:14:34:13] ENGINE Bus STARTING 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[39074]: [13/Sep/2024:14:34:13] ENGINE Serving on http://172.21.15.154:8765 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[39074]: [13/Sep/2024:14:34:14] ENGINE Serving on https://172.21.15.154:7150 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[39074]: [13/Sep/2024:14:34:14] ENGINE Bus STARTED 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[39074]: [13/Sep/2024:14:34:14] ENGINE Error in HTTPServer.serve 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout: Traceback (most recent call last): 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._run(expiration_interval) 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T14:34:15.914 INFO:journalctl@ceph.mon.c.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: s = self.context.wrap_socket( 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: return self.sslsocket_class._create( 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: self.do_handshake() 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: self._sslobj.do_handshake() 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[39074]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:15.915 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:15 smithi154 ceph-mon[39074]: mgrmap e37: y(active, since 2s), standbys: x 2024-09-13T14:34:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-13T14:34:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:17 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:17.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: mgrmap e38: y(active, since 4s), standbys: x 2024-09-13T14:34:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:17 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:19.104 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:18 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:19.105 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:18 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:19.105 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:18 smithi154 ceph-mon[102314]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:19.105 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:18 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:19.105 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:18 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:19.105 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:18 smithi154 ceph-mon[39074]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:19.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:18 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:19.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:18 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:18 smithi172 ceph-mon[32794]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:20.918 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:20 smithi172 ceph-mon[32794]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:34:21.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:20 smithi154 ceph-mon[102314]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:34:21.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:20 smithi154 ceph-mon[39074]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-13T14:34:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:34:21] "GET /metrics HTTP/1.1" 200 31137 "" "Prometheus/2.51.0" 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:34:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:34:22.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:21 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.conf 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:34:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: Updating smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:21 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:22.957 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:22.957 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:22.957 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:34:22.957 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[102314]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[102314]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[39074]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:34:22.958 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:22 smithi154 ceph-mon[39074]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:34:23.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:22 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:22 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:22 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:34:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:22 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-13T14:34:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:22 smithi172 ceph-mon[32794]: Upgrade: It appears safe to stop mon.c 2024-09-13T14:34:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:22 smithi172 ceph-mon[32794]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-13T14:34:24.673 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[102314]: Upgrade: Updating mon.c 2024-09-13T14:34:24.673 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:24.673 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:34:24.673 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:34:24.673 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:24.674 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[102314]: Deploying daemon mon.c on smithi154 2024-09-13T14:34:24.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[39074]: Upgrade: Updating mon.c 2024-09-13T14:34:24.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[39074]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:24.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:34:24.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:34:24.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[39074]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:24.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:24 smithi154 ceph-mon[39074]: Deploying daemon mon.c on smithi154 2024-09-13T14:34:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:24 smithi172 ceph-mon[32794]: Upgrade: Updating mon.c 2024-09-13T14:34:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:24 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:24 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:34:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:24 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:34:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:24 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:24 smithi172 ceph-mon[32794]: Deploying daemon mon.c on smithi154 2024-09-13T14:34:25.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:24 smithi154 systemd[1]: Stopping Ceph mon.c for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:34:25.637 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:25] ENGINE Bus STOPPING 2024-09-13T14:34:25.637 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:34:25.637 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:25] ENGINE Bus STOPPED 2024-09-13T14:34:25.637 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c[39070]: 2024-09-13T14:34:25.235+0000 7f75be22e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:34:25.638 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c[39070]: 2024-09-13T14:34:25.235+0000 7f75be22e640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:34:25.638 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:25 smithi154 podman[106879]: 2024-09-13 14:34:25.327839008 +0000 UTC m=+0.224063664 container died a87978af9c4081b3297f6863e2fc919e3a4dfef291e73c95900444dcf609571a (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:34:25.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:25 smithi154 podman[106879]: 2024-09-13 14:34:25.80321546 +0000 UTC m=+0.699440103 container cleanup a87978af9c4081b3297f6863e2fc919e3a4dfef291e73c95900444dcf609571a (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:34:25.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:25 smithi154 bash[106879]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c 2024-09-13T14:34:26.440 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 podman[106893]: 2024-09-13 14:34:26.063130338 +0000 UTC m=+0.726370599 container remove a87978af9c4081b3297f6863e2fc919e3a4dfef291e73c95900444dcf609571a (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:34:26.440 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-13T14:34:26.441 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 systemd[1]: Stopped Ceph mon.c for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:34:26.441 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.c.service: Consumed 8.012s CPU time. 2024-09-13T14:34:26.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 systemd[1]: Starting Ceph mon.c for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:34:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 podman[107020]: 2024-09-13 14:34:26.631242103 +0000 UTC m=+0.094928128 container create fcefc4676e768076d37e06c4f614daf2581dd877ad0a3f6d7bdbb1fdf55e84c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-13T14:34:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 podman[107020]: 2024-09-13 14:34:26.552366523 +0000 UTC m=+0.016052508 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:34:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 podman[107020]: 2024-09-13 14:34:26.867416139 +0000 UTC m=+0.331102122 container init fcefc4676e768076d37e06c4f614daf2581dd877ad0a3f6d7bdbb1fdf55e84c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-13T14:34:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 podman[107020]: 2024-09-13 14:34:26.871306653 +0000 UTC m=+0.334992633 container start fcefc4676e768076d37e06c4f614daf2581dd877ad0a3f6d7bdbb1fdf55e84c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-13T14:34:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:34:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 2 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: pidfile_write: ignore empty --pid-file 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: load: jerasure load: lrc 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Git sha 0 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: DB SUMMARY 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: DB Session ID: XKA5LAD0YZM39DXLBEV3 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1411 Bytes 2024-09-13T14:34:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 5346738 ; 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.env: 0x55cfec85e160 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.info_log: 0x55cfed93f880 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:34:27.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.statistics: (nil) 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.use_fsync: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:34:27.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.db_log_dir: 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.wal_dir: 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.write_buffer_manager: 0x55cfed942960 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:34:27.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.unordered_write: 0 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.row_cache: None 2024-09-13T14:34:27.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.wal_filter: None 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.wal_compression: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:34:27.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:34:27.229 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_open_files: -1 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:34:27.230 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Compression algorithms supported: 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kZSTD supported: 0 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:34:27.231 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.merge_operator: 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_filter: None 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cfed93f4e0) 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:34:27.232 INFO:journalctl@ceph.mon.c.smithi154.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: index_type: 0 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: data_block_index_type: 0 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: index_shortening: 1 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: checksum: 4 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: no_block_cache: 0 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache: 0x55cfed932850 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache_options: 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: capacity : 536870912 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: num_shard_bits : 4 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: strict_capacity_limit : 0 2024-09-13T14:34:27.233 INFO:journalctl@ceph.mon.c.smithi154.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_cache_compressed: (nil) 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: persistent_cache: (nil) 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_size: 4096 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_size_deviation: 10 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_restart_interval: 16 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: index_block_restart_interval: 1 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: metadata_block_size: 4096 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: partition_filters: 0 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: use_delta_encoding: 1 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: filter_policy: bloomfilter 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: whole_key_filtering: 1 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: verify_compression: 0 2024-09-13T14:34:27.234 INFO:journalctl@ceph.mon.c.smithi154.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout: format_version: 5 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout: enable_index_compression: 1 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout: block_align: 0 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout: max_auto_readahead_size: 262144 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout: prepopulate_block_cache: 0 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression: NoCompression 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:34:27.235 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.num_levels: 7 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:34:27.236 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:34:27.237 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:34:27.238 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:34:27.239 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:34:27.240 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.ttl: 2592000 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:34:27.241 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12954, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e2d4abd6-5cc7-471d-a9ab-f3771df2c245 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238066897674, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238066909757, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3283849, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12959, "largest_seqno": 14865, "table_properties": {"data_size": 3275502, "index_size": 5253, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2053, "raw_key_size": 19618, "raw_average_key_size": 24, "raw_value_size": 3257991, "raw_average_value_size": 3987, "num_data_blocks": 239, "num_entries": 817, "num_filter_entries": 817, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726238066, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e2d4abd6-5cc7-471d-a9ab-f3771df2c245", "db_session_id": "XKA5LAD0YZM39DXLBEV3", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238066909878, "job": 1, "event": "recovery_finished"} 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:34:27.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cfed962e00 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: DB pointer 0x55cfeda62000 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** DB Stats ** 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.243 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** Compaction Stats [default] ** 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: L0 1/0 3.13 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 679.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: L6 1/0 7.64 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: Sum 2/0 10.77 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 679.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 679.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** Compaction Stats [default] ** 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:34:27.244 INFO:journalctl@ceph.mon.c.smithi154.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 679.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: Cumulative compaction: 0.00 GB write, 209.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:34:27.245 INFO:journalctl@ceph.mon.c.smithi154.stdout: Interval compaction: 0.00 GB write, 209.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: Block cache BinnedLRUCache@0x55cfed932850#2 capacity: 512.00 MB usage: 8.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.28 KB,0.000435114%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: starting mon.c rank 2 at public addrs [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] at bind addrs [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???) e3 preinit fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???).mds e9 new map 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???).mds e9 print_map 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: e9 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:34:27.246 INFO:journalctl@ceph.mon.c.smithi154.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: legacy client fscid: 1 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: Filesystem 'foo' (1) 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: fs_name foo 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: epoch 8 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: created 2024-09-13T14:28:14.301827+0000 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: modified 2024-09-13T14:28:28.364158+0000 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: tableserver 0 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: root 0 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: session_timeout 60 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: session_autoclose 300 2024-09-13T14:34:27.247 INFO:journalctl@ceph.mon.c.smithi154.stdout: max_file_size 1099511627776 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: max_xattr_size 65536 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: required_client_features {} 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: last_failure 0 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: last_failure_osd_epoch 0 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: max_mds 1 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: in 0 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: up {0=24539} 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: failed 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: damaged 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: stopped 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: data_pools [4] 2024-09-13T14:34:27.248 INFO:journalctl@ceph.mon.c.smithi154.stdout: metadata_pool 3 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: inline_data disabled 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: balancer 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: bal_rank_mask -1 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: standby_count_wanted 1 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: [mds.foo.smithi154.daekzd{0:24539} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: Standby daemons: 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout: [mds.foo.smithi172.pmmovd{-1:24472} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???).osd e89 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:34:27.249 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:34:27.250 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:34:27.250 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:34:27.250 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 ceph-mon[107038]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-13T14:34:27.250 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 bash[107020]: fcefc4676e768076d37e06c4f614daf2581dd877ad0a3f6d7bdbb1fdf55e84c9 2024-09-13T14:34:27.250 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:26 smithi154 systemd[1]: Started Ceph mon.c for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:34:32.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:32] ENGINE Bus STARTING 2024-09-13T14:34:32.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:32] ENGINE Serving on http://:::9283 2024-09-13T14:34:32.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:34:32] ENGINE Bus STARTED 2024-09-13T14:34:32.988 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: mon.c calling monitor election 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: mon.b calling monitor election 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: mon.a calling monitor election 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: monmap epoch 3 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: last_changed 2024-09-13T14:23:04.765616+0000 2024-09-13T14:34:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: min_mon_release 18 (reef) 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: election_strategy: 1 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: mgrmap e38: y(active, since 20s), standbys: x 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[107038]: overall HEALTH_OK 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: mon.c calling monitor election 2024-09-13T14:34:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: mon.b calling monitor election 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: mon.a calling monitor election 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: monmap epoch 3 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: last_changed 2024-09-13T14:23:04.765616+0000 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: min_mon_release 18 (reef) 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: election_strategy: 1 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:34:33.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:34:33.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: mgrmap e38: y(active, since 20s), standbys: x 2024-09-13T14:34:33.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:33 smithi154 ceph-mon[102314]: overall HEALTH_OK 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: mon.c calling monitor election 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: mon.b calling monitor election 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: mon.a calling monitor election 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: monmap epoch 3 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: last_changed 2024-09-13T14:23:04.765616+0000 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: min_mon_release 18 (reef) 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: election_strategy: 1 2024-09-13T14:34:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:34:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:34:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:34:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:34:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:34:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: mgrmap e38: y(active, since 20s), standbys: x 2024-09-13T14:34:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:33 smithi172 ceph-mon[32794]: overall HEALTH_OK 2024-09-13T14:34:33.576 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:34:33.576 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (98s) 16s ago 7m 18.9M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:34:33.576 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (75s) 16s ago 7m 79.6M - c8b91775d855 4bed28596716 2024-09-13T14:34:33.576 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (6m) 16s ago 6m 18.0M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:34:33.576 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (6m) 16s ago 6m 16.8M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (2m) 16s ago 11m 450M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (3m) 16s ago 12m 486M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (29s) 16s ago 13m 28.0M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 c1bb51eb0c90 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (11m) 16s ago 11m 51.2M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 starting - - - 2048M 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (2m) 16s ago 6m 11.2M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (2m) 16s ago 6m 11.1M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (10m) 16s ago 10m 53.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (10m) 16s ago 10m 61.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (10m) 16s ago 10m 57.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (9m) 16s ago 9m 52.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (9m) 16s ago 9m 55.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (9m) 16s ago 9m 55.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:34:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (8m) 16s ago 8m 54.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:34:33.578 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (8m) 16s ago 8m 53.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:34:33.578 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (118s) 16s ago 7m 40.9M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:34:33.920 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 1, 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:34:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:34:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:34:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:34:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 11, 2024-09-13T14:34:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:34:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:34:33.922 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:34:34.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[102314]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[102314]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[102314]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/96852835' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:34.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[107038]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[107038]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[107038]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:34 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/96852835' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:34.228 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "1/3 daemons upgraded", 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:34:34.229 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:34:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:34 smithi172 ceph-mon[32794]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:34 smithi172 ceph-mon[32794]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:34 smithi172 ceph-mon[32794]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:34 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/96852835' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:34:35.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:35 smithi154 ceph-mon[102314]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:35.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:35 smithi154 ceph-mon[102314]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:35.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:35 smithi154 ceph-mon[107038]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:35.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:35 smithi154 ceph-mon[107038]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:35.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:35 smithi172 ceph-mon[32794]: from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:34:35.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:35 smithi172 ceph-mon[32794]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:36.816 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.816 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.816 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.816 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.816 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.816 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.817 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.817 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:36 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:36 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:36 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:36.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:36 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:37.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:36 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:34:36.918+0000 7f9d7ab79640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T14:34:37.896 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:37 smithi154 ceph-mon[102314]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:37.897 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:37 smithi154 ceph-mon[107038]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:37.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:37 smithi172 ceph-mon[32794]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:38.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:38 smithi172 ceph-mon[32794]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:39.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:38 smithi154 ceph-mon[102314]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:39.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:38 smithi154 ceph-mon[107038]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:34:40.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.976 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.977 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.978 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:34:40.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:34:40.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:34:40.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:34:40.979 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:34:40.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:34:40.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:34:40.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:34:40.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:34:40.981 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:34:40.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:34:40.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.983 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.984 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:34:40.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:40 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:34:40.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-13T14:34:40.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:34:40.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-13T14:34:40.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:34:40.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-13T14:34:40.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:34:40.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-13T14:34:40.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:34:40.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:34:40.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:34:40.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:40 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:41.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all grafana 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all loki 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all promtail 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Finalizing container_image settings 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: Upgrade: Complete! 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[102314]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:41.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all grafana 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all loki 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all promtail 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Finalizing container_image settings 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: Upgrade: Complete! 2024-09-13T14:34:41.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-mon[107038]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:34:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:34:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:34:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:34:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all grafana 2024-09-13T14:34:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all loki 2024-09-13T14:34:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all promtail 2024-09-13T14:34:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Finalizing container_image settings 2024-09-13T14:34:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: Upgrade: Complete! 2024-09-13T14:34:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:41 smithi172 ceph-mon[32794]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:34:41] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:34:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:42 smithi172 ceph-mon[32794]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:43.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:42 smithi154 ceph-mon[102314]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:43.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:42 smithi154 ceph-mon[107038]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:43 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:43 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:43 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:34:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:43 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:44 smithi154 ceph-mon[102314]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:45.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:44 smithi154 ceph-mon[107038]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:45.234 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:44 smithi172 ceph-mon[32794]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:46 smithi172 ceph-mon[32794]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:47.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:46 smithi154 ceph-mon[102314]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:47.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:46 smithi154 ceph-mon[107038]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:48.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:48 smithi172 ceph-mon[32794]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:49.022 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:48 smithi154 ceph-mon[102314]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:49.023 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:48 smithi154 ceph-mon[107038]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:50 smithi154 ceph-mon[102314]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:50.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:50 smithi154 ceph-mon[107038]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:50.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:50 smithi172 ceph-mon[32794]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:34:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:34:51] "GET /metrics HTTP/1.1" 200 33816 "" "Prometheus/2.51.0" 2024-09-13T14:34:52.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:52 smithi172 ceph-mon[32794]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:53.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:52 smithi154 ceph-mon[102314]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:53.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:52 smithi154 ceph-mon[107038]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:54.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:54 smithi172 ceph-mon[32794]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:54 smithi154 ceph-mon[102314]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:54 smithi154 ceph-mon[107038]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:56.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:56 smithi172 ceph-mon[32794]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:56.991 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:34:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=infra.usagestats t=2024-09-13T14:34:56.815873722Z level=info msg="Usage stats are ready to report" 2024-09-13T14:34:57.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:56 smithi154 ceph-mon[102314]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:57.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:56 smithi154 ceph-mon[107038]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:57 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:34:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:34:58 smithi154 ceph-mon[102314]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:34:58 smithi154 ceph-mon[107038]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:34:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:34:58 smithi172 ceph-mon[32794]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:00.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:00 smithi172 ceph-mon[32794]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:01.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:00 smithi154 ceph-mon[102314]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:01.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:00 smithi154 ceph-mon[107038]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:35:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:35:01] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:35:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:02 smithi172 ceph-mon[32794]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:03.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:02 smithi154 ceph-mon[102314]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:03.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:02 smithi154 ceph-mon[107038]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:04.868 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:04 smithi154 ceph-mon[102314]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:04.868 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:04 smithi154 ceph-mon[102314]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:04.868 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:04 smithi154 ceph-mon[107038]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:04.868 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:04 smithi154 ceph-mon[107038]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:04.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:04 smithi172 ceph-mon[32794]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:35:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:04 smithi172 ceph-mon[32794]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:05.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:35:05.463 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:35:06.864 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:06 smithi154 ceph-mon[102314]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:06.864 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:06 smithi154 ceph-mon[107038]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:06.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:06 smithi172 ceph-mon[32794]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:08.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:08 smithi172 ceph-mon[32794]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:09.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:08 smithi154 ceph-mon[102314]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:09.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:08 smithi154 ceph-mon[107038]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:10 smithi172 ceph-mon[32794]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:10 smithi154 ceph-mon[102314]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:11.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:10 smithi154 ceph-mon[107038]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:35:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:35:11] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:35:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:12 smithi172 ceph-mon[32794]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:12 smithi154 ceph-mon[102314]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:13.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:12 smithi154 ceph-mon[107038]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:13 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:14 smithi154 ceph-mon[102314]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:14 smithi154 ceph-mon[107038]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:14 smithi172 ceph-mon[32794]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:16.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:16 smithi172 ceph-mon[32794]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:17.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:16 smithi154 ceph-mon[102314]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:17.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:16 smithi154 ceph-mon[107038]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:18.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:18 smithi154 ceph-mon[102314]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:18 smithi154 ceph-mon[107038]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:18.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:18 smithi172 ceph-mon[32794]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:20 smithi172 ceph-mon[32794]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:21.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:20 smithi154 ceph-mon[102314]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:21.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:20 smithi154 ceph-mon[107038]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:35:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:35:21] "GET /metrics HTTP/1.1" 200 33815 "" "Prometheus/2.51.0" 2024-09-13T14:35:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:22 smithi172 ceph-mon[32794]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:23.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:22 smithi154 ceph-mon[102314]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:23.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:22 smithi154 ceph-mon[107038]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:24.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:24 smithi172 ceph-mon[32794]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:25.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:24 smithi154 ceph-mon[102314]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:25.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:24 smithi154 ceph-mon[107038]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:26.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:26 smithi172 ceph-mon[32794]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:26 smithi154 ceph-mon[102314]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:27.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:26 smithi154 ceph-mon[107038]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:27 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:29 smithi154 ceph-mon[102314]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:29.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:29 smithi154 ceph-mon[107038]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:29.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:29 smithi172 ceph-mon[32794]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:30.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:30 smithi172 ceph-mon[32794]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:31.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:30 smithi154 ceph-mon[102314]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:31.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:30 smithi154 ceph-mon[107038]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:35:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:35:31] "GET /metrics HTTP/1.1" 200 33815 "" "Prometheus/2.51.0" 2024-09-13T14:35:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:32 smithi172 ceph-mon[32794]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:33.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:32 smithi154 ceph-mon[102314]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:33.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:32 smithi154 ceph-mon[107038]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:34.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:34 smithi172 ceph-mon[32794]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:35.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:34 smithi154 ceph-mon[102314]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:35.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:34 smithi154 ceph-mon[107038]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:36.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:36 smithi172 ceph-mon[32794]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:36 smithi154 ceph-mon[102314]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:37.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:36 smithi154 ceph-mon[107038]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:38.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:38 smithi172 ceph-mon[32794]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:39.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:38 smithi154 ceph-mon[102314]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:39.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:38 smithi154 ceph-mon[107038]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:35:40.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:35:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:40 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:35:41.374 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:41 smithi172 ceph-mon[32794]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:41.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:41 smithi154 ceph-mon[102314]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:41.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:41 smithi154 ceph-mon[107038]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:42.145 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:35:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:35:41] "GET /metrics HTTP/1.1" 200 33815 "" "Prometheus/2.51.0" 2024-09-13T14:35:42.989 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:42 smithi172 ceph-mon[32794]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:42.992 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:42 smithi154 ceph-mon[107038]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:42.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:42 smithi154 ceph-mon[102314]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:43.977 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:43 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:43.977 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:43 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:43.977 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:43 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:43 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:43 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:43 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:43 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:35:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:35:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[102314]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:35:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:35:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:45 smithi154 ceph-mon[107038]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:45 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:45 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:45 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:35:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:45 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:35:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:45 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:35:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:45 smithi172 ceph-mon[32794]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:46 smithi154 ceph-mon[102314]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-13T14:35:46.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:46 smithi154 ceph-mon[107038]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-13T14:35:46.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:46 smithi172 ceph-mon[32794]: mgrmap e39: y(active, since 92s), standbys: x 2024-09-13T14:35:47.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:47 smithi154 ceph-mon[102314]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:47.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:47 smithi154 ceph-mon[107038]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:47.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:47 smithi172 ceph-mon[32794]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:49.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:48 smithi154 ceph-mon[102314]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:49.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:48 smithi154 ceph-mon[107038]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:49.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:48 smithi172 ceph-mon[32794]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:51.023 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:50 smithi154 ceph-mon[102314]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:51.023 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:50 smithi154 ceph-mon[107038]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:51.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:50 smithi172 ceph-mon[32794]: pgmap v52: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:35:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:35:51] "GET /metrics HTTP/1.1" 200 33815 "" "Prometheus/2.51.0" 2024-09-13T14:35:53.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:52 smithi154 ceph-mon[102314]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:53.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:52 smithi154 ceph-mon[107038]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:53.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:52 smithi172 ceph-mon[32794]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:54.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:54 smithi172 ceph-mon[32794]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:54 smithi154 ceph-mon[102314]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:54 smithi154 ceph-mon[107038]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T14:35:57.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:56 smithi154 ceph-mon[102314]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:57.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:56 smithi154 ceph-mon[107038]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:57.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:56 smithi172 ceph-mon[32794]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:57 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:35:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:35:59 smithi154 ceph-mon[102314]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:35:59 smithi154 ceph-mon[107038]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:35:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:35:59 smithi172 ceph-mon[32794]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:01.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:00 smithi154 ceph-mon[102314]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:01.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:00 smithi154 ceph-mon[107038]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:01.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:00 smithi172 ceph-mon[32794]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:36:01] "GET /metrics HTTP/1.1" 200 33813 "" "Prometheus/2.51.0" 2024-09-13T14:36:03.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:02 smithi154 ceph-mon[102314]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:03.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:02 smithi154 ceph-mon[107038]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:03.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:02 smithi172 ceph-mon[32794]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:05.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:04 smithi172 ceph-mon[32794]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:05.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:04 smithi154 ceph-mon[102314]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:05.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:04 smithi154 ceph-mon[107038]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:07.010 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:06 smithi154 ceph-mon[102314]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:07.011 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:06 smithi154 ceph-mon[107038]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:07.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:06 smithi172 ceph-mon[32794]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:07.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-13T14:36:07.900 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:09.053 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:08 smithi154 ceph-mon[107038]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:09.054 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:08 smithi154 ceph-mon[102314]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:09.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:08 smithi172 ceph-mon[32794]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:09.557 INFO:teuthology.orchestra.run.smithi154.stdout:wait for majority of mons to upgrade 2024-09-13T14:36:10.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --limit 1' 2024-09-13T14:36:10.391 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:10.823 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:10 smithi154 ceph-mon[102314]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:10.823 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:10 smithi154 ceph-mon[107038]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:11.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:10 smithi172 ceph-mon[32794]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:12.207 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:36:11] "GET /metrics HTTP/1.1" 200 33813 "" "Prometheus/2.51.0" 2024-09-13T14:36:13.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:12 smithi154 ceph-mon[102314]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:13.002 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:12 smithi154 ceph-mon[102314]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:13.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:12 smithi154 ceph-mon[107038]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:13.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:12 smithi154 ceph-mon[107038]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:12 smithi172 ceph-mon[32794]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:12 smithi172 ceph-mon[32794]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:13.941 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:36:13.941 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:36:13.974 INFO:teuthology.orchestra.run.smithi154.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:36:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:13 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T14:36:14.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:36:14.790 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:36:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[102314]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:36:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:14 smithi154 ceph-mon[107038]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:36:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:36:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:36:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T14:36:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:36:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:14 smithi172 ceph-mon[32794]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:16 smithi154 ceph-mon[102314]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:16 smithi154 ceph-mon[107038]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:17.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:16 smithi172 ceph-mon[32794]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:17.375 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:36:17.942 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:36:17.942 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (3m) 34s ago 9m 20.2M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:36:17.942 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (2m) 33s ago 8m 85.4M - c8b91775d855 4bed28596716 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (7m) 34s ago 7m 18.6M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (7m) 33s ago 7m 17.1M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (4m) 33s ago 12m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (4m) 34s ago 14m 507M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (2m) 34s ago 14m 40.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 c1bb51eb0c90 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (13m) 33s ago 13m 56.9M 2048M 18.2.4-843-g2c01458a c129f5a2fd4e 3e07b61db92d 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (111s) 34s ago 13m 31.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 fcefc4676e76 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (4m) 34s ago 8m 10.8M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (4m) 33s ago 8m 10.9M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (12m) 34s ago 12m 53.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (12m) 34s ago 12m 61.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (11m) 34s ago 11m 57.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:36:17.943 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (11m) 34s ago 11m 53.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:36:17.944 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (11m) 33s ago 11m 55.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:36:17.944 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (10m) 33s ago 10m 56.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:36:17.944 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (10m) 33s ago 10m 55.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:36:17.944 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (9m) 33s ago 9m 54.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:36:17.944 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (3m) 33s ago 9m 43.1M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 1, 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:36:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 11, 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 4 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:36:18.279 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:36:18.568 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (1 remaining).", 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "2/3 daemons upgraded", 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:36:18.569 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:36:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:36:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:36:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all mgr 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:36:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/612094425' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all mgr 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:18.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:36:18.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-13T14:36:18.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:36:18.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/612094425' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: Upgrade: Setting container_image for all mgr 2024-09-13T14:36:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:18.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:18.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-13T14:36:18.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-13T14:36:18.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: Upgrade: It appears safe to stop mon.b 2024-09-13T14:36:18.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:18 smithi172 ceph-mon[32794]: from='client.? 172.21.15.154:0/612094425' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:19.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:19 smithi172 ceph-mon[32794]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:19.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:19 smithi172 ceph-mon[32794]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:19.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:19 smithi172 ceph-mon[32794]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:19.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:19 smithi172 ceph-mon[32794]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:19.830 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:19 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:36:19.831 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:19 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:36:19.831 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:19 smithi172 ceph-mon[32794]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:36:19.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[102314]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:19.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[102314]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:19.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[102314]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:19.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[102314]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[107038]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[107038]: from='client.44138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[107038]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[107038]: from='mgr.25736 ' entity='mgr.y' 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T14:36:19.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:19 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T14:36:20.912 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:20 smithi172 ceph-mon[32794]: Upgrade: Updating mon.b 2024-09-13T14:36:20.912 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:20 smithi172 ceph-mon[32794]: Deploying daemon mon.b on smithi172 2024-09-13T14:36:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:20 smithi154 ceph-mon[102314]: Upgrade: Updating mon.b 2024-09-13T14:36:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:20 smithi154 ceph-mon[102314]: Deploying daemon mon.b on smithi172 2024-09-13T14:36:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:20 smithi154 ceph-mon[107038]: Upgrade: Updating mon.b 2024-09-13T14:36:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:20 smithi154 ceph-mon[107038]: Deploying daemon mon.b on smithi172 2024-09-13T14:36:21.163 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:20 smithi172 systemd[1]: Stopping Ceph mon.b for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:36:21.443 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b[32790]: 2024-09-13T14:36:21.161+0000 7f52e5374640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:36:21.443 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b[32790]: 2024-09-13T14:36:21.161+0000 7f52e5374640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-13T14:36:21.444 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 podman[100537]: 2024-09-13 14:36:21.182338514 +0000 UTC m=+0.126289533 container died 3e07b61db92d035a03362912b113fc7481cdb3c0d1154a648eb808f0b9a2a965 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-13T14:36:21.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:36:21] ENGINE Bus STOPPING 2024-09-13T14:36:21.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:36:21] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T14:36:21.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:36:21] ENGINE Bus STOPPED 2024-09-13T14:36:21.470 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:36:21] ENGINE Bus STARTING 2024-09-13T14:36:21.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 podman[100537]: 2024-09-13 14:36:21.544734123 +0000 UTC m=+0.488685146 container cleanup 3e07b61db92d035a03362912b113fc7481cdb3c0d1154a648eb808f0b9a2a965 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-13T14:36:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 bash[100537]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b 2024-09-13T14:36:21.876 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:36:21] ENGINE Serving on http://:::9283 2024-09-13T14:36:21.876 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:14:36:21] ENGINE Bus STARTED 2024-09-13T14:36:22.132 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 podman[100548]: 2024-09-13 14:36:21.804751409 +0000 UTC m=+0.622583346 container remove 3e07b61db92d035a03362912b113fc7481cdb3c0d1154a648eb808f0b9a2a965 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-13T14:36:22.132 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-13T14:36:22.132 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 systemd[1]: Stopped Ceph mon.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:36:22.132 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:21 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.b.service: Consumed 9.869s CPU time. 2024-09-13T14:36:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:36:21] "GET /metrics HTTP/1.1" 200 33747 "" "Prometheus/2.51.0" 2024-09-13T14:36:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 systemd[1]: Starting Ceph mon.b for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:36:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 podman[100678]: 2024-09-13 14:36:22.242501444 +0000 UTC m=+0.016057889 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:36:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 podman[100678]: 2024-09-13 14:36:22.380947856 +0000 UTC m=+0.154504289 container create 50c3380c21bb6a99cfa6fdbea685b4c17371383311dbd2663c639bf0fb072421 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-13T14:36:22.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 podman[100678]: 2024-09-13 14:36:22.608939045 +0000 UTC m=+0.382495486 container init 50c3380c21bb6a99cfa6fdbea685b4c17371383311dbd2663c639bf0fb072421 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-13T14:36:22.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 podman[100678]: 2024-09-13 14:36:22.612327172 +0000 UTC m=+0.385883604 container start 50c3380c21bb6a99cfa6fdbea685b4c17371383311dbd2663c639bf0fb072421 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:36:22.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev), process ceph-mon, pid 2 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: pidfile_write: ignore empty --pid-file 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: load: jerasure load: lrc 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: RocksDB version: 7.9.2 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Git sha 0 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Compile date 2024-09-12 20:46:14 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: DB SUMMARY 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: DB Session ID: KU994SBWVYDCPB9PETL7 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: CURRENT file: CURRENT 2024-09-13T14:36:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1411 Bytes 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 7485141 ; 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.error_if_exists: 0 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.create_if_missing: 0 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.paranoid_checks: 1 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.env: 0x55ad9e155160 2024-09-13T14:36:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.info_log: 0x55ada06f9880 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.statistics: (nil) 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.use_fsync: 0 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_log_file_size: 0 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.allow_fallocate: 1 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T14:36:22.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.use_direct_reads: 0 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.db_log_dir: 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.wal_dir: 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.write_buffer_manager: 0x55ada06fc960 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T14:36:22.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.unordered_write: 0 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.row_cache: None 2024-09-13T14:36:22.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.wal_filter: None 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.two_write_queues: 0 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.wal_compression: 0 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.atomic_flush: 0 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T14:36:22.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.log_readahead_size: 0 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_background_jobs: 2 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_background_compactions: -1 2024-09-13T14:36:22.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_subcompactions: 1 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_open_files: -1 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T14:36:23.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_background_flushes: -1 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Compression algorithms supported: 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kZSTD supported: 0 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kXpressCompression supported: 0 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kBZip2Compression supported: 0 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kLZ4Compression supported: 1 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kZlibCompression supported: 1 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: kSnappyCompression supported: 1 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T14:36:23.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.merge_operator: 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_filter: None 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ada06f94e0) 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout: cache_index_and_filter_blocks: 1 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T14:36:23.002 INFO:journalctl@ceph.mon.b.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: pin_top_level_index_and_filter: 1 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: index_type: 0 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: data_block_index_type: 0 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: index_shortening: 1 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: checksum: 4 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: no_block_cache: 0 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache: 0x55ada06ec850 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache_name: BinnedLRUCache 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache_options: 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: capacity : 536870912 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: num_shard_bits : 4 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: strict_capacity_limit : 0 2024-09-13T14:36:23.003 INFO:journalctl@ceph.mon.b.smithi172.stdout: high_pri_pool_ratio: 0.000 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache_compressed: (nil) 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: persistent_cache: (nil) 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_size: 4096 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_size_deviation: 10 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_restart_interval: 16 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: index_block_restart_interval: 1 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: metadata_block_size: 4096 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: partition_filters: 0 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: use_delta_encoding: 1 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: filter_policy: bloomfilter 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: whole_key_filtering: 1 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: verify_compression: 0 2024-09-13T14:36:23.004 INFO:journalctl@ceph.mon.b.smithi172.stdout: read_amp_bytes_per_bit: 0 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout: format_version: 5 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout: enable_index_compression: 1 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_align: 0 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout: max_auto_readahead_size: 262144 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout: prepopulate_block_cache: 0 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout: initial_auto_readahead_size: 8192 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression: NoCompression 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T14:36:23.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.num_levels: 7 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:36:23.006 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T14:36:23.007 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T14:36:23.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T14:36:23.009 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.inplace_update_support: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.bloom_locality: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.max_successive_merges: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.ttl: 2592000 2024-09-13T14:36:23.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.enable_blob_files: false 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.min_blob_size: 0 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:36:23.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 13026, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 992c62d9-eadf-499c-bc79-6e379dc05885 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238182638976, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238182657856, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4594335, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13031, "largest_seqno": 16684, "table_properties": {"data_size": 4580327, "index_size": 8865, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4101, "raw_key_size": 36661, "raw_average_key_size": 22, "raw_value_size": 4548150, "raw_average_value_size": 2791, "num_data_blocks": 399, "num_entries": 1629, "num_filter_entries": 1629, "num_deletions": 261, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726238182, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "992c62d9-eadf-499c-bc79-6e379dc05885", "db_session_id": "KU994SBWVYDCPB9PETL7", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726238182657985, "job": 1, "event": "recovery_finished"} 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ada071ce00 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: DB pointer 0x55ada081c000 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T14:36:23.012 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** DB Stats ** 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** Compaction Stats [default] ** 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T14:36:23.013 INFO:journalctl@ceph.mon.b.smithi172.stdout: L0 1/0 4.38 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: L6 1/0 7.64 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: Sum 2/0 12.02 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 526.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** Compaction Stats [default] ** 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 526.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-13T14:36:23.014 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative compaction: 0.00 GB write, 202.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval compaction: 0.00 GB write, 202.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: Block cache BinnedLRUCache@0x55ada06ec850#2 capacity: 512.00 MB usage: 13.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: starting mon.b rank 1 at public addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] at bind addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???) e3 preinit fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:36:23.015 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???).mds e9 new map 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???).mds e9 print_map 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: e9 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: legacy client fscid: 1 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: Filesystem 'foo' (1) 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: fs_name foo 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: epoch 8 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: created 2024-09-13T14:28:14.301827+0000 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: modified 2024-09-13T14:28:28.364158+0000 2024-09-13T14:36:23.016 INFO:journalctl@ceph.mon.b.smithi172.stdout: tableserver 0 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: root 0 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: session_timeout 60 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: session_autoclose 300 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: max_file_size 1099511627776 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: max_xattr_size 65536 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: required_client_features {} 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: last_failure 0 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: last_failure_osd_epoch 0 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: max_mds 1 2024-09-13T14:36:23.017 INFO:journalctl@ceph.mon.b.smithi172.stdout: in 0 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: up {0=24539} 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: failed 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: damaged 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: stopped 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: data_pools [4] 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: metadata_pool 3 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: inline_data disabled 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: balancer 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: bal_rank_mask -1 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: standby_count_wanted 1 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: qdb_cluster leader: 0 members: 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: [mds.foo.smithi154.daekzd{0:24539} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.154:6834/214113580,v1:172.21.15.154:6835/214113580] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:36:23.018 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout: Standby daemons: 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout: [mds.foo.smithi172.pmmovd{-1:24472} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.172:6832/3027611841,v1:172.21.15.172:6833/3027611841] compat {c=[1],r=[1],i=[7ff]}] 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???).osd e89 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 ceph-mon[100696]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 bash[100678]: 50c3380c21bb6a99cfa6fdbea685b4c17371383311dbd2663c639bf0fb072421 2024-09-13T14:36:23.019 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:22 smithi172 systemd[1]: Started Ceph mon.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: overall HEALTH_OK 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: mon.b calling monitor election 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: mon.a calling monitor election 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: mon.c calling monitor election 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:36:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: monmap epoch 4 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: last_changed 2024-09-13T14:36:24.213041+0000 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: min_mon_release 19 (squid) 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: election_strategy: 1 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: overall HEALTH_OK 2024-09-13T14:36:25.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[102314]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: overall HEALTH_OK 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: mon.b calling monitor election 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: mon.a calling monitor election 2024-09-13T14:36:25.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: mon.c calling monitor election 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: monmap epoch 4 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: last_changed 2024-09-13T14:36:24.213041+0000 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: min_mon_release 19 (squid) 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: election_strategy: 1 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:36:25.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: overall HEALTH_OK 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:25.475 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:25 smithi154 ceph-mon[107038]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: overall HEALTH_OK 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: mon.b calling monitor election 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: mon.a calling monitor election 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: mon.c calling monitor election 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-13T14:36:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: monmap epoch 4 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: last_changed 2024-09-13T14:36:24.213041+0000 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: created 2024-09-13T14:21:27.916625+0000 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: min_mon_release 19 (squid) 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: election_strategy: 1 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.a 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: 1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: 2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: fsmap foo:1 {0=foo.smithi154.daekzd=up:active} 1 up:standby 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: osdmap e89: 8 total, 8 up, 8 in 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: mgrmap e39: y(active, since 2m), standbys: x 2024-09-13T14:36:25.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: overall HEALTH_OK 2024-09-13T14:36:25.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:25.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:25.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:25 smithi172 ceph-mon[100696]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:26 smithi172 ceph-mon[100696]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:26 smithi154 ceph-mon[102314]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:27.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:26 smithi154 ceph-mon[107038]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:27.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:36:27.667+0000 7f9d7ab79640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T14:36:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:28.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:28.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:28.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:28.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:29.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:29 smithi172 ceph-mon[100696]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:29 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:29 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:29 smithi154 ceph-mon[102314]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:29 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:29 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:29.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:29 smithi154 ceph-mon[107038]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:29.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:29 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:29.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:29 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:31.176 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:30 smithi172 ceph-mon[100696]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:31.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:30 smithi154 ceph-mon[102314]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:31.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:30 smithi154 ceph-mon[107038]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:36:31] "GET /metrics HTTP/1.1" 200 33824 "" "Prometheus/2.51.0" 2024-09-13T14:36:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: Reconfiguring daemon mon.a on smithi154 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[102314]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:32.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: Reconfiguring daemon mon.a on smithi154 2024-09-13T14:36:32.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:32 smithi154 ceph-mon[107038]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: Reconfiguring mon.a (monmap changed)... 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: Reconfiguring daemon mon.a on smithi154 2024-09-13T14:36:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:32 smithi172 ceph-mon[100696]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[102314]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:36:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:36:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:36:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[102314]: Reconfiguring daemon mgr.y on smithi154 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[107038]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:34 smithi154 ceph-mon[107038]: Reconfiguring daemon mgr.y on smithi154 2024-09-13T14:36:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:34 smithi172 ceph-mon[100696]: Reconfiguring mgr.y (monmap changed)... 2024-09-13T14:36:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:36:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:36:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:34 smithi172 ceph-mon[100696]: Reconfiguring daemon mgr.y on smithi154 2024-09-13T14:36:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[102314]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[107038]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:35 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:35 smithi172 ceph-mon[100696]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:35 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:35 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:35 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:35 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:35 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:36.405 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[102314]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:36:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:36 smithi172 ceph-mon[100696]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:36:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:36 smithi172 ceph-mon[100696]: Reconfiguring daemon mon.c on smithi154 2024-09-13T14:36:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:36:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:36.717 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[102314]: Reconfiguring daemon mon.c on smithi154 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[107038]: Reconfiguring mon.c (monmap changed)... 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[107038]: Reconfiguring daemon mon.c on smithi154 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:36.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:36:36.719 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:37 smithi154 ceph-mon[102314]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:36:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:37 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.0 on smithi154 2024-09-13T14:36:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:37 smithi154 ceph-mon[102314]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:37.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:37 smithi154 ceph-mon[107038]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:36:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:37 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.0 on smithi154 2024-09-13T14:36:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:37 smithi154 ceph-mon[107038]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:37.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:37 smithi172 ceph-mon[100696]: Reconfiguring osd.0 (monmap changed)... 2024-09-13T14:36:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:37 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.0 on smithi154 2024-09-13T14:36:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:37 smithi172 ceph-mon[100696]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[107038]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:36:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:36:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.1 on smithi154 2024-09-13T14:36:38.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:38.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:38.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[102314]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:36:38.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:36:38.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:38.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:38 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.1 on smithi154 2024-09-13T14:36:38.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:38.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:38.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:38 smithi172 ceph-mon[100696]: Reconfiguring osd.1 (monmap changed)... 2024-09-13T14:36:38.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:36:38.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:38.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:38 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.1 on smithi154 2024-09-13T14:36:39.701 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[102314]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:39.701 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:39.701 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:39.701 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:36:39.702 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:39.702 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[107038]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:39.702 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:39.702 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:39.702 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:36:39.702 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:39 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:39.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:39 smithi172 ceph-mon[100696]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:39.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:39 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:39.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:39 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:39.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:39 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:36:39.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:39 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:40.673 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:40 smithi154 ceph-mon[102314]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:36:40.673 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:40 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.2 on smithi154 2024-09-13T14:36:40.673 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:40 smithi154 ceph-mon[107038]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:36:40.674 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:40 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.2 on smithi154 2024-09-13T14:36:40.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:40 smithi172 ceph-mon[100696]: Reconfiguring osd.2 (monmap changed)... 2024-09-13T14:36:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:40 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.2 on smithi154 2024-09-13T14:36:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[102314]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:36:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:41.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[107038]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:36:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:41.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:41 smithi172 ceph-mon[100696]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:36:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:42.144 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:36:41] "GET /metrics HTTP/1.1" 200 33824 "" "Prometheus/2.51.0" 2024-09-13T14:36:42.792 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[102314]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:36:42.792 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.3 on smithi154 2024-09-13T14:36:42.792 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[107038]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.3 on smithi154 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:36:42.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:42 smithi172 ceph-mon[100696]: Reconfiguring osd.3 (monmap changed)... 2024-09-13T14:36:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:42 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.3 on smithi154 2024-09-13T14:36:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi154.daekzd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:36:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:43.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[102314]: Reconfiguring mds.foo.smithi154.daekzd (monmap changed)... 2024-09-13T14:36:43.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[102314]: Reconfiguring daemon mds.foo.smithi154.daekzd on smithi154 2024-09-13T14:36:43.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[102314]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:43.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:43.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[107038]: Reconfiguring mds.foo.smithi154.daekzd (monmap changed)... 2024-09-13T14:36:43.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[107038]: Reconfiguring daemon mds.foo.smithi154.daekzd on smithi154 2024-09-13T14:36:43.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[107038]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:43.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:43.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:43 smithi172 ceph-mon[100696]: Reconfiguring mds.foo.smithi154.daekzd (monmap changed)... 2024-09-13T14:36:43.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:43 smithi172 ceph-mon[100696]: Reconfiguring daemon mds.foo.smithi154.daekzd on smithi154 2024-09-13T14:36:43.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:43 smithi172 ceph-mon[100696]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:43.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[102314]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:36:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[102314]: Reconfiguring daemon mon.b on smithi172 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[107038]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:44 smithi154 ceph-mon[107038]: Reconfiguring daemon mon.b on smithi172 2024-09-13T14:36:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:44 smithi172 ceph-mon[100696]: Reconfiguring mon.b (monmap changed)... 2024-09-13T14:36:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-13T14:36:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-13T14:36:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:44 smithi172 ceph-mon[100696]: Reconfiguring daemon mon.b on smithi172 2024-09-13T14:36:45.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:45 smithi172 ceph-mon[100696]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:45.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:45.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:45.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:36:45.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:36:45.820 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:45.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[102314]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:45.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:45.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:45.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:36:45.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:36:45.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:45.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[107038]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:45.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:45.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:45.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-13T14:36:45.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-13T14:36:45.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[102314]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:36:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[102314]: Reconfiguring daemon mgr.x on smithi172 2024-09-13T14:36:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:36:46.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:46.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[107038]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:36:46.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[107038]: Reconfiguring daemon mgr.x on smithi172 2024-09-13T14:36:46.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:46.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:46.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:36:46.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:46.973 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:46 smithi172 ceph-mon[100696]: Reconfiguring mgr.x (monmap changed)... 2024-09-13T14:36:46.973 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:46 smithi172 ceph-mon[100696]: Reconfiguring daemon mgr.x on smithi172 2024-09-13T14:36:46.973 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:46.973 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:46.973 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-13T14:36:46.973 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:47.947 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:47 smithi172 ceph-mon[100696]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:36:47.948 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:47 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.4 on smithi172 2024-09-13T14:36:47.948 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:47 smithi172 ceph-mon[100696]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:47.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:47 smithi154 ceph-mon[102314]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:36:47.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:47 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.4 on smithi172 2024-09-13T14:36:47.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:47 smithi154 ceph-mon[102314]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:47.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:47 smithi154 ceph-mon[107038]: Reconfiguring osd.4 (monmap changed)... 2024-09-13T14:36:47.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:47 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.4 on smithi172 2024-09-13T14:36:47.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:47 smithi154 ceph-mon[107038]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:48.880 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:36:49.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:49.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:49.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[102314]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:36:49.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:36:49.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:49.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.5 on smithi172 2024-09-13T14:36:49.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[102314]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:49.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:49.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:49.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[107038]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:36:49.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:36:49.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:49.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.5 on smithi172 2024-09-13T14:36:49.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:49 smithi154 ceph-mon[107038]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:49.430 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:49 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:49.430 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:49 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:49.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:49 smithi172 ceph-mon[100696]: Reconfiguring osd.5 (monmap changed)... 2024-09-13T14:36:49.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:49 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-13T14:36:49.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:49 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:49.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:49 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.5 on smithi172 2024-09-13T14:36:49.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:49 smithi172 ceph-mon[100696]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:49.442 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:36:49.442 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (3m) 66s ago 9m 20.2M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:36:49.443 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (3m) 21s ago 9m 85.5M - c8b91775d855 4bed28596716 2024-09-13T14:36:49.443 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (8m) 66s ago 8m 18.6M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:36:49.443 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (8m) 21s ago 8m 17.4M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:36:49.443 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (4m) 21s ago 13m 453M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:36:49.443 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (5m) 66s ago 15m 507M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:36:49.443 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (2m) 66s ago 15m 40.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 c1bb51eb0c90 2024-09-13T14:36:49.444 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (27s) 21s ago 13m 24.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 50c3380c21bb 2024-09-13T14:36:49.444 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (2m) 66s ago 13m 31.9M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 fcefc4676e76 2024-09-13T14:36:49.444 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (4m) 66s ago 9m 10.8M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:36:49.444 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (4m) 21s ago 9m 11.3M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:36:49.444 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (13m) 66s ago 13m 53.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:36:49.445 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (12m) 66s ago 12m 61.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:36:49.445 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (12m) 66s ago 12m 57.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:36:49.445 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (12m) 66s ago 12m 53.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:36:49.445 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (11m) 21s ago 11m 56.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:36:49.445 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (11m) 21s ago 11m 56.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:36:49.446 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (10m) 21s ago 10m 55.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:36:49.446 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (10m) 21s ago 10m 54.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:36:49.446 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (4m) 21s ago 9m 43.2M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:49.786 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:36:49.787 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:36:49.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:36:49.787 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:36:49.787 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 10, 2024-09-13T14:36:49.787 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 5 2024-09-13T14:36:49.787 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:36:49.787 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:36:50.072 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:36:50.072 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:36:50.072 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) mon. Upgrade limited to 1 daemons (0 remaining).", 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [ 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout: "mon" 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "3/3 daemons upgraded", 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading mon daemons", 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:36:50.073 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:36:50.787 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.787 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.787 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:50.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:50.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:36:50.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:36:50.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:50.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.6 on smithi172 2024-09-13T14:36:50.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1225683758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:50.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:36:50.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:36:50.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:50.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.6 on smithi172 2024-09-13T14:36:50.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1225683758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:50.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='client.34221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='client.44156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:50.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:50.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:50.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: Reconfiguring osd.6 (monmap changed)... 2024-09-13T14:36:50.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-13T14:36:50.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:50.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.6 on smithi172 2024-09-13T14:36:50.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1225683758' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:51.872 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[102314]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:51.872 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[102314]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[107038]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[107038]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:36:51.873 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:51.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:51 smithi172 ceph-mon[100696]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:36:51.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:51 smithi172 ceph-mon[100696]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-13T14:36:51.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:36:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:36:51] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:36:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:52 smithi172 ceph-mon[100696]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:36:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:52 smithi172 ceph-mon[100696]: Reconfiguring daemon osd.7 on smithi172 2024-09-13T14:36:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:36:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:52.888 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[102314]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:36:52.888 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[102314]: Reconfiguring daemon osd.7 on smithi172 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[107038]: Reconfiguring osd.7 (monmap changed)... 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[107038]: Reconfiguring daemon osd.7 on smithi172 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi172.pmmovd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-13T14:36:52.889 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:53.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:53 smithi172 ceph-mon[100696]: Reconfiguring mds.foo.smithi172.pmmovd (monmap changed)... 2024-09-13T14:36:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:53 smithi172 ceph-mon[100696]: Reconfiguring daemon mds.foo.smithi172.pmmovd on smithi172 2024-09-13T14:36:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:53 smithi172 ceph-mon[100696]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:53.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:53 smithi154 ceph-mon[102314]: Reconfiguring mds.foo.smithi172.pmmovd (monmap changed)... 2024-09-13T14:36:53.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:53 smithi154 ceph-mon[102314]: Reconfiguring daemon mds.foo.smithi172.pmmovd on smithi172 2024-09-13T14:36:53.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:53 smithi154 ceph-mon[102314]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:53.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:53 smithi154 ceph-mon[107038]: Reconfiguring mds.foo.smithi172.pmmovd (monmap changed)... 2024-09-13T14:36:53.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:53 smithi154 ceph-mon[107038]: Reconfiguring daemon mds.foo.smithi172.pmmovd on smithi172 2024-09-13T14:36:53.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:53 smithi154 ceph-mon[107038]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all grafana 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all loki 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all promtail 2024-09-13T14:36:55.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Finalizing container_image settings 2024-09-13T14:36:55.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:36:55.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all grafana 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all loki 2024-09-13T14:36:55.225 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all promtail 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Finalizing container_image settings 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:36:55.226 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: Upgrade: Complete! 2024-09-13T14:36:55.227 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[107038]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:36:55.228 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.229 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.230 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.230 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: Upgrade: Complete! 2024-09-13T14:36:55.230 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:36:55.230 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:36:55.230 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:55.230 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:54 smithi154 ceph-mon[102314]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all grafana 2024-09-13T14:36:55.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all loki 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all promtail 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Finalizing container_image settings 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: Upgrade: Complete! 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:55.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:54 smithi172 ceph-mon[100696]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:56.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:56 smithi154 ceph-mon[102314]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:56.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:56 smithi154 ceph-mon[107038]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:57.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:56 smithi172 ceph-mon[100696]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:36:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:36:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:36:59 smithi154 ceph-mon[102314]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:36:59 smithi154 ceph-mon[107038]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:36:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:36:59 smithi172 ceph-mon[100696]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:01.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:00 smithi154 ceph-mon[102314]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:01.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:00 smithi154 ceph-mon[107038]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:01.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:00 smithi172 ceph-mon[100696]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:37:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:37:01] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:37:03.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:02 smithi154 ceph-mon[102314]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:03.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:02 smithi154 ceph-mon[107038]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:03.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:02 smithi172 ceph-mon[100696]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:05.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:04 smithi172 ceph-mon[100696]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:05.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:04 smithi154 ceph-mon[102314]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:05.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:04 smithi154 ceph-mon[107038]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:07.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:06 smithi154 ceph-mon[102314]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:07.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:06 smithi154 ceph-mon[107038]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:07.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:06 smithi172 ceph-mon[100696]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:09.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:08 smithi154 ceph-mon[102314]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:09.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:08 smithi154 ceph-mon[107038]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:09.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:08 smithi172 ceph-mon[100696]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:10 smithi154 ceph-mon[102314]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:11.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:10 smithi154 ceph-mon[107038]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:11.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:10 smithi172 ceph-mon[100696]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:37:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:37:11] "GET /metrics HTTP/1.1" 200 33818 "" "Prometheus/2.51.0" 2024-09-13T14:37:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:12 smithi154 ceph-mon[102314]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:13.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:12 smithi154 ceph-mon[107038]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:13.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:12 smithi172 ceph-mon[100696]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:14 smithi154 ceph-mon[102314]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:14 smithi154 ceph-mon[107038]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:14 smithi172 ceph-mon[100696]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:17.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:16 smithi154 ceph-mon[102314]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:17.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:16 smithi154 ceph-mon[107038]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:17.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:16 smithi172 ceph-mon[100696]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:19.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:18 smithi154 ceph-mon[102314]: pgmap v96: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:19.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:18 smithi154 ceph-mon[107038]: pgmap v96: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:19.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:18 smithi172 ceph-mon[100696]: pgmap v96: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:21.091 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:20 smithi154 ceph-mon[102314]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:21.091 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:20 smithi154 ceph-mon[102314]: pgmap v97: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:21.091 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:20 smithi154 ceph-mon[107038]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:21.091 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:20 smithi154 ceph-mon[107038]: pgmap v97: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:21.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:37:21.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:20 smithi172 ceph-mon[100696]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:37:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:20 smithi172 ceph-mon[100696]: pgmap v97: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:21.317 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:37:21.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:37:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:37:21] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:37:22.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:22 smithi154 ceph-mon[102314]: pgmap v98: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:22.889 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:22 smithi154 ceph-mon[107038]: pgmap v98: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:23.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:22 smithi172 ceph-mon[100696]: pgmap v98: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:25.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:24 smithi154 ceph-mon[107038]: pgmap v99: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:25.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:24 smithi154 ceph-mon[102314]: pgmap v99: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:25.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:24 smithi172 ceph-mon[100696]: pgmap v99: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:26 smithi154 ceph-mon[102314]: pgmap v100: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:27.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:26 smithi154 ceph-mon[107038]: pgmap v100: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:27.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:26 smithi172 ceph-mon[100696]: pgmap v100: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:29 smithi154 ceph-mon[102314]: pgmap v101: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:29.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:29 smithi154 ceph-mon[107038]: pgmap v101: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:29.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:29 smithi172 ceph-mon[100696]: pgmap v101: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:31.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:30 smithi154 ceph-mon[107038]: pgmap v102: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:31.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:30 smithi154 ceph-mon[102314]: pgmap v102: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:31.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:30 smithi172 ceph-mon[100696]: pgmap v102: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:37:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:37:31] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:37:33.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:32 smithi154 ceph-mon[102314]: pgmap v103: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:33.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:32 smithi154 ceph-mon[107038]: pgmap v103: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:33.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:32 smithi172 ceph-mon[100696]: pgmap v103: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:35.943 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:35 smithi172 ceph-mon[100696]: pgmap v104: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:35.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:35 smithi154 ceph-mon[102314]: pgmap v104: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:35.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:35 smithi154 ceph-mon[107038]: pgmap v104: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:36 smithi154 ceph-mon[102314]: pgmap v105: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:37.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:36 smithi154 ceph-mon[107038]: pgmap v105: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:37.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:36 smithi172 ceph-mon[100696]: pgmap v105: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:39.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:38 smithi154 ceph-mon[102314]: pgmap v106: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:39.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:38 smithi154 ceph-mon[107038]: pgmap v106: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:39.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:38 smithi172 ceph-mon[100696]: pgmap v106: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:42.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:41 smithi154 ceph-mon[107038]: pgmap v107: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:41 smithi154 ceph-mon[102314]: pgmap v107: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:42.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:37:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:37:41] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:37:42.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:41 smithi172 ceph-mon[100696]: pgmap v107: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:43 smithi154 ceph-mon[102314]: pgmap v108: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:43 smithi154 ceph-mon[107038]: pgmap v108: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:43 smithi172 ceph-mon[100696]: pgmap v108: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:45 smithi154 ceph-mon[102314]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:45 smithi154 ceph-mon[107038]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:45 smithi172 ceph-mon[100696]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:47 smithi154 ceph-mon[102314]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:47 smithi154 ceph-mon[107038]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:47 smithi172 ceph-mon[100696]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:49 smithi154 ceph-mon[102314]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:49 smithi154 ceph-mon[107038]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:49 smithi172 ceph-mon[100696]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:52.044 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:51 smithi154 ceph-mon[102314]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:52.045 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:51 smithi154 ceph-mon[107038]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:52.045 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:37:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:37:51] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-09-13T14:37:52.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:51 smithi172 ceph-mon[100696]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:54.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:53 smithi154 ceph-mon[102314]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:54.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:53 smithi154 ceph-mon[107038]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:54.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:53 smithi172 ceph-mon[100696]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:55 smithi154 ceph-mon[102314]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:56.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:55 smithi154 ceph-mon[107038]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:56.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:55 smithi172 ceph-mon[100696]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:57 smithi154 ceph-mon[102314]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:58.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:57 smithi154 ceph-mon[107038]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:58.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:58.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:57 smithi172 ceph-mon[100696]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:37:58.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:37:59.101 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:59.147 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:37:59.147 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:00.106 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:37:59 smithi154 ceph-mon[102314]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:00.106 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:37:59 smithi154 ceph-mon[107038]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:00.157 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:37:59 smithi172 ceph-mon[100696]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:02.131 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:38:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:38:01] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[102314]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[107038]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:02 smithi172 ceph-mon[100696]: pgmap v117: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:03 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:03 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:03 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:03.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:03 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:03.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:03 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:03 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:03 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:03 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:03 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:04 smithi154 ceph-mon[102314]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:04 smithi154 ceph-mon[107038]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:04.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:04 smithi172 ceph-mon[100696]: pgmap v118: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:06.543 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:06 smithi172 ceph-mon[100696]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:06.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:06 smithi154 ceph-mon[102314]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:06.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:06 smithi154 ceph-mon[107038]: pgmap v119: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:08 smithi154 ceph-mon[102314]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:08.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:08 smithi154 ceph-mon[107038]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:08 smithi172 ceph-mon[100696]: pgmap v120: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:10 smithi154 ceph-mon[102314]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:10.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:10 smithi154 ceph-mon[107038]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:10.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:10 smithi172 ceph-mon[100696]: pgmap v121: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:38:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:38:11] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:38:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:12 smithi154 ceph-mon[102314]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:12.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:12 smithi154 ceph-mon[107038]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:12.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:12 smithi172 ceph-mon[100696]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:13.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:14 smithi154 ceph-mon[102314]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:14.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:14 smithi154 ceph-mon[107038]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:14.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:14 smithi172 ceph-mon[100696]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:16 smithi154 ceph-mon[102314]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:16.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:16 smithi154 ceph-mon[107038]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:16.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:16 smithi172 ceph-mon[100696]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:18 smithi154 ceph-mon[102314]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:18 smithi154 ceph-mon[107038]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:18.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:18 smithi172 ceph-mon[100696]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:20 smithi154 ceph-mon[102314]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:20.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:20 smithi154 ceph-mon[107038]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:20.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:20 smithi172 ceph-mon[100696]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:38:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:38:21] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:38:22.495 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:22 smithi154 ceph-mon[102314]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:22.496 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:22 smithi154 ceph-mon[107038]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:22.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:22 smithi172 ceph-mon[100696]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:23.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for all mons to upgrade' 2024-09-13T14:38:23.636 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:24.609 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:24 smithi154 ceph-mon[102314]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:24.609 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:24 smithi154 ceph-mon[107038]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:24 smithi172 ceph-mon[100696]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:25.105 INFO:teuthology.orchestra.run.smithi154.stdout:wait for all mons to upgrade 2024-09-13T14:38:25.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon' 2024-09-13T14:38:25.921 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:26.354 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:26 smithi154 ceph-mon[102314]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:26.354 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:26 smithi154 ceph-mon[107038]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:26.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:26 smithi172 ceph-mon[100696]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:28.555 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:28 smithi154 ceph-mon[102314]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:28.555 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:28 smithi154 ceph-mon[102314]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:38:28.555 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:28.555 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:28 smithi154 ceph-mon[107038]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:28.555 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:28 smithi154 ceph-mon[107038]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:38:28.555 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:28 smithi172 ceph-mon[100696]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:28 smithi172 ceph-mon[100696]: from='client.44180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:38:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:29.565 INFO:teuthology.orchestra.run.smithi154.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:38:30.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:38:30.428 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:30.949 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:30.949 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:38:30.949 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[102314]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:30.950 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:30.951 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:30.951 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:30.951 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:30 smithi154 ceph-mon[107038]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:38:30.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:30.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:38:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:30 smithi172 ceph-mon[100696]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:38:31.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:38:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:38:31] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:38:32.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:32 smithi154 ceph-mon[102314]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:32.939 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:32 smithi154 ceph-mon[107038]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:32.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:32 smithi172 ceph-mon[100696]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:33.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all mgr 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all mon 2024-09-13T14:38:33.980 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all crash 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all rgw 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.981 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all nfs 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.982 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all mgr 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all mon 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all crash 2024-09-13T14:38:33.983 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all rgw 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.984 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all nfs 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:38:33.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all grafana 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all loki 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all promtail 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Finalizing container_image settings 2024-09-13T14:38:33.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:38:33.987 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.988 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all grafana 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all loki 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all promtail 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Finalizing container_image settings 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:38:33.989 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:38:33.990 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: Upgrade: Complete! 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:33.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[107038]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:38:33.992 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: Upgrade: Complete! 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:33.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.994 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.994 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:33 smithi154 ceph-mon[102314]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:38:33.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:33.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:38:33.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all mgr 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all mon 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all crash 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all rgw 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all rbd-mirror 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all cephfs-mirror 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all ceph-exporter 2024-09-13T14:38:33.996 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all iscsi 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all nfs 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.997 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all node-exporter 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all prometheus 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all alertmanager 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all grafana 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all loki 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all promtail 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Finalizing container_image settings 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:38:33.998 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-13T14:38:33.999 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:38:34.000 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: Upgrade: Complete! 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:38:34.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:33 smithi172 ceph-mon[100696]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:38:34.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'sleep 60' 2024-09-13T14:38:34.199 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:38:36.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:36 smithi154 ceph-mon[102314]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:36.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:36 smithi154 ceph-mon[107038]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:36.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:36 smithi172 ceph-mon[100696]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:38 smithi154 ceph-mon[102314]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:38 smithi154 ceph-mon[107038]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:38.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:38 smithi172 ceph-mon[100696]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:40 smithi154 ceph-mon[102314]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:40.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:40 smithi154 ceph-mon[107038]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:40.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:40 smithi172 ceph-mon[100696]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:38:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:38:41] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:38:42.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:42 smithi154 ceph-mon[107038]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:42 smithi154 ceph-mon[102314]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:42 smithi172 ceph-mon[100696]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:43.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:43.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:43.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:44 smithi154 ceph-mon[102314]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:44.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:44 smithi154 ceph-mon[107038]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:44.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:44 smithi172 ceph-mon[100696]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:46 smithi154 ceph-mon[102314]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:46.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:46 smithi154 ceph-mon[107038]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:46.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:46 smithi172 ceph-mon[100696]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:48.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:48 smithi154 ceph-mon[102314]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:48.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:48 smithi154 ceph-mon[107038]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:48.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:48 smithi172 ceph-mon[100696]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:50.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:50 smithi154 ceph-mon[102314]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:50.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:50 smithi154 ceph-mon[107038]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:50.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:50 smithi172 ceph-mon[100696]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:38:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:38:51] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-09-13T14:38:52.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:52 smithi154 ceph-mon[102314]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:52.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:52 smithi154 ceph-mon[107038]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:52.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:52 smithi172 ceph-mon[100696]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:54.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:54 smithi154 ceph-mon[102314]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:54.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:54 smithi154 ceph-mon[107038]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:54.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:54 smithi172 ceph-mon[100696]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:56.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:56 smithi154 ceph-mon[102314]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:56.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:56 smithi154 ceph-mon[107038]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:56 smithi172 ceph-mon[100696]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:58.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:58 smithi154 ceph-mon[102314]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:58.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:38:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:58.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:58 smithi154 ceph-mon[107038]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:58.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:38:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:38:58.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:58 smithi172 ceph-mon[100696]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:38:58.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:38:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:00 smithi154 ceph-mon[102314]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:00.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:00 smithi154 ceph-mon[107038]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:00.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:00 smithi172 ceph-mon[100696]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:39:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:39:01] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:39:02.966 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:02 smithi154 ceph-mon[102314]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:02.966 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:02 smithi154 ceph-mon[107038]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:02 smithi172 ceph-mon[100696]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:04.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:04 smithi154 ceph-mon[102314]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:04 smithi154 ceph-mon[107038]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:04.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:04 smithi172 ceph-mon[100696]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:06 smithi154 ceph-mon[102314]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:06.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:06 smithi154 ceph-mon[107038]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:06.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:06 smithi172 ceph-mon[100696]: pgmap v149: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:08 smithi154 ceph-mon[102314]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:08.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:08 smithi154 ceph-mon[107038]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:08.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:08 smithi172 ceph-mon[100696]: pgmap v150: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:10 smithi154 ceph-mon[102314]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:10.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:10 smithi154 ceph-mon[107038]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:10 smithi172 ceph-mon[100696]: pgmap v151: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:39:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:39:11] "GET /metrics HTTP/1.1" 200 33821 "" "Prometheus/2.51.0" 2024-09-13T14:39:12.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:12 smithi154 ceph-mon[107038]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:12 smithi154 ceph-mon[102314]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:12 smithi172 ceph-mon[100696]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:13.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:13.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:13.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:14.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:14 smithi154 ceph-mon[102314]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:14.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:14 smithi154 ceph-mon[107038]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:14.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:14 smithi172 ceph-mon[100696]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:16 smithi154 ceph-mon[102314]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:16 smithi154 ceph-mon[107038]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:16.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:16 smithi172 ceph-mon[100696]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:18.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:18 smithi154 ceph-mon[102314]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:18 smithi154 ceph-mon[107038]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:18.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:18 smithi172 ceph-mon[100696]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:20 smithi154 ceph-mon[102314]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:20 smithi154 ceph-mon[107038]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:20.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:20 smithi172 ceph-mon[100696]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:39:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:39:21] "GET /metrics HTTP/1.1" 200 33819 "" "Prometheus/2.51.0" 2024-09-13T14:39:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:22 smithi154 ceph-mon[102314]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:22.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:22 smithi154 ceph-mon[107038]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:22 smithi172 ceph-mon[100696]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:24.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:24 smithi154 ceph-mon[102314]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:24.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:24 smithi154 ceph-mon[107038]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:24.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:24 smithi172 ceph-mon[100696]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:26.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:26 smithi154 ceph-mon[102314]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:26.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:26 smithi154 ceph-mon[107038]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:26 smithi172 ceph-mon[100696]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:28 smithi154 ceph-mon[102314]: pgmap v160: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:28 smithi154 ceph-mon[107038]: pgmap v160: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:28.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:28 smithi172 ceph-mon[100696]: pgmap v160: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:30 smithi154 ceph-mon[102314]: pgmap v161: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:30.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:30 smithi154 ceph-mon[107038]: pgmap v161: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:30 smithi172 ceph-mon[100696]: pgmap v161: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:39:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:39:31] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:39:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:32 smithi154 ceph-mon[102314]: pgmap v162: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:32.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:32 smithi154 ceph-mon[107038]: pgmap v162: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:32.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:32 smithi172 ceph-mon[100696]: pgmap v162: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:33.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:33.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:33.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:34.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:34 smithi154 ceph-mon[102314]: pgmap v163: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:34.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:34 smithi154 ceph-mon[107038]: pgmap v163: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:34.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:34 smithi172 ceph-mon[100696]: pgmap v163: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:35.951 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:35 smithi154 ceph-mon[102314]: pgmap v164: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:35.951 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:35 smithi154 ceph-mon[107038]: pgmap v164: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:35.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:35 smithi172 ceph-mon[100696]: pgmap v164: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:36.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-13T14:39:37.012 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:38.088 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: pgmap v165: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: pgmap v165: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:38.089 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.090 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.090 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.090 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.090 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:38.090 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:38.090 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: pgmap v165: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:38.600 INFO:teuthology.orchestra.run.smithi154.stdout:wait for half of osds to upgrade 2024-09-13T14:39:39.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 4' 2024-09-13T14:39:39.530 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:40.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:39 smithi154 ceph-mon[102314]: pgmap v166: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:40.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:39 smithi154 ceph-mon[107038]: pgmap v166: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:39 smithi172 ceph-mon[100696]: pgmap v166: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:42.151 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:41 smithi154 ceph-mon[102314]: pgmap v167: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:42.151 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:41 smithi154 ceph-mon[102314]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:42.152 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:39:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:39:41] "GET /metrics HTTP/1.1" 200 33822 "" "Prometheus/2.51.0" 2024-09-13T14:39:42.152 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:41 smithi154 ceph-mon[107038]: pgmap v167: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:42.152 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:41 smithi154 ceph-mon[107038]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:41 smithi172 ceph-mon[100696]: pgmap v167: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:41 smithi172 ceph-mon[100696]: from='client.44192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9", "daemon_types": "osd", "limit": 4, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:43.159 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:43.159 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:43.193 INFO:teuthology.orchestra.run.smithi154.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:39:43.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:43.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -e sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-13T14:39:44.035 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:39:44.410 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: pgmap v168: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:44.410 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:39:44.410 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:44.410 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[102314]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: pgmap v168: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:44.411 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:44.412 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:44 smithi154 ceph-mon[107038]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:39:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: pgmap v168: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:39:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:44.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:44.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:39:44.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:44.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:44 smithi172 ceph-mon[100696]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: pgmap v169: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: pgmap v169: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:39:46.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: pgmap v169: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:46.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:46.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[102314]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[102314]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all mgr 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all mon 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all crash 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[102314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[102314]: Upgrade: osd.0 is safe to restart 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[107038]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:39:47.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[107038]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:39:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all mgr 2024-09-13T14:39:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all mon 2024-09-13T14:39:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all crash 2024-09-13T14:39:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[107038]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:39:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:47 smithi154 ceph-mon[107038]: Upgrade: osd.0 is safe to restart 2024-09-13T14:39:47.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:47 smithi172 ceph-mon[100696]: Upgrade: Target is version 19.3.0-4697-gd58f7306 (squid) 2024-09-13T14:39:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:47 smithi172 ceph-mon[100696]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f'] 2024-09-13T14:39:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:47 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all mgr 2024-09-13T14:39:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:47 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all mon 2024-09-13T14:39:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:47 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all crash 2024-09-13T14:39:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:47 smithi172 ceph-mon[100696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-13T14:39:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:47 smithi172 ceph-mon[100696]: Upgrade: osd.0 is safe to restart 2024-09-13T14:39:48.000 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:39:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[102314]: pgmap v170: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[102314]: Upgrade: Updating osd.0 2024-09-13T14:39:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:39:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[102314]: Deploying daemon osd.0 on smithi154 2024-09-13T14:39:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[107038]: pgmap v170: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[107038]: Upgrade: Updating osd.0 2024-09-13T14:39:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:39:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:48 smithi154 ceph-mon[107038]: Deploying daemon osd.0 on smithi154 2024-09-13T14:39:48.571 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:39:48.571 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (6m) 11s ago 12m 18.9M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:39:48.571 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (6m) 11s ago 12m 86.7M - c8b91775d855 4bed28596716 2024-09-13T14:39:48.571 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (11m) 11s ago 11m 20.5M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:39:48.571 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (11m) 11s ago 11m 18.1M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (7m) 11s ago 16m 455M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (8m) 11s ago 18m 517M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (5m) 11s ago 18m 50.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 c1bb51eb0c90 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (3m) 11s ago 16m 39.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 50c3380c21bb 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (5m) 11s ago 16m 43.5M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 fcefc4676e76 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (7m) 11s ago 12m 11.7M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (7m) 11s ago 12m 12.0M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (16m) 11s ago 16m 54.5M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 4a2892f34e4f 2024-09-13T14:39:48.572 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (15m) 11s ago 15m 62.4M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:39:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (15m) 11s ago 15m 57.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:39:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (15m) 11s ago 14m 53.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:39:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (14m) 11s ago 14m 57.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:39:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (14m) 11s ago 14m 57.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:39:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (13m) 11s ago 13m 56.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:39:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (13m) 11s ago 13m 55.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:39:48.573 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (7m) 11s ago 12m 47.7M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:39:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:48 smithi172 ceph-mon[100696]: pgmap v170: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:48 smithi172 ceph-mon[100696]: Upgrade: Updating osd.0 2024-09-13T14:39:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:48 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:48 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-13T14:39:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:48 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:39:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:48 smithi172 ceph-mon[100696]: Deploying daemon osd.0 on smithi154 2024-09-13T14:39:48.913 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:39:48.913 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:39:48.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:39:48.915 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:39:48.915 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 10, 2024-09-13T14:39:48.915 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 5 2024-09-13T14:39:48.915 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:39:48.915 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (4 remaining).", 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "0/8 daemons upgraded", 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:39:49.220 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:39:49.221 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:39:49.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[102314]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[102314]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[102314]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/842190028' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:49.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[107038]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[107038]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[107038]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:49 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/842190028' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:49.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:49 smithi172 ceph-mon[100696]: from='client.44198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:49 smithi172 ceph-mon[100696]: from='client.34275 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:49 smithi172 ceph-mon[100696]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:49 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/842190028' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:39:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:50 smithi154 ceph-mon[102314]: pgmap v171: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:50 smithi154 ceph-mon[102314]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:50 smithi154 ceph-mon[107038]: pgmap v171: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:50 smithi154 ceph-mon[107038]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:50.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:50 smithi154 systemd[1]: Stopping Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:39:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:50 smithi172 ceph-mon[100696]: pgmap v171: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:50 smithi172 ceph-mon[100696]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:39:51.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:50 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[42918]: 2024-09-13T14:39:50.736+0000 7fe3ec520640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:39:51.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:50 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[42918]: 2024-09-13T14:39:50.736+0000 7fe3ec520640 -1 osd.0 89 *** Got signal Terminated *** 2024-09-13T14:39:51.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:50 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[42918]: 2024-09-13T14:39:50.736+0000 7fe3ec520640 -1 osd.0 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:39:51.640 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:51 smithi154 ceph-mon[102314]: osd.0 marked itself down and dead 2024-09-13T14:39:51.641 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:51 smithi154 ceph-mon[107038]: osd.0 marked itself down and dead 2024-09-13T14:39:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:51 smithi172 ceph-mon[100696]: osd.0 marked itself down and dead 2024-09-13T14:39:51.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:39:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:39:51] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-09-13T14:39:51.970 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:51 smithi154 podman[117952]: 2024-09-13 14:39:51.840375479 +0000 UTC m=+1.227489154 container died 4a2892f34e4ff808ae647a2f7cc2ee0274e84b5d9a2d203bebee4dd3747ff93d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, ceph=True, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:39:52.464 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:52 smithi154 ceph-mon[102314]: pgmap v172: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:52.464 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:52 smithi154 ceph-mon[102314]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:39:52.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:52 smithi154 ceph-mon[102314]: osdmap e90: 8 total, 7 up, 8 in 2024-09-13T14:39:52.465 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:52 smithi154 ceph-mon[107038]: pgmap v172: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:52.465 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:52 smithi154 ceph-mon[107038]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:39:52.465 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:52 smithi154 ceph-mon[107038]: osdmap e90: 8 total, 7 up, 8 in 2024-09-13T14:39:52.465 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:52 smithi154 podman[117952]: 2024-09-13 14:39:52.270409477 +0000 UTC m=+1.657523144 container cleanup 4a2892f34e4ff808ae647a2f7cc2ee0274e84b5d9a2d203bebee4dd3747ff93d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=reef-2c01458, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-13T14:39:52.465 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:52 smithi154 bash[117952]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0 2024-09-13T14:39:52.720 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:52 smithi154 podman[117969]: 2024-09-13 14:39:52.464068522 +0000 UTC m=+0.626388783 container remove 4a2892f34e4ff808ae647a2f7cc2ee0274e84b5d9a2d203bebee4dd3747ff93d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True) 2024-09-13T14:39:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:52 smithi172 ceph-mon[100696]: pgmap v172: 73 pgs: 73 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:52 smithi172 ceph-mon[100696]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:39:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:52 smithi172 ceph-mon[100696]: osdmap e90: 8 total, 7 up, 8 in 2024-09-13T14:39:53.209 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:52 smithi154 podman[118055]: 2024-09-13 14:39:52.789693885 +0000 UTC m=+0.015999094 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:39:53.209 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:52 smithi154 podman[118055]: 2024-09-13 14:39:52.89825116 +0000 UTC m=+0.124556368 container create b1de06561ee0ba9b7edbe199ee760848b009785026a13ef5ffc3b20c8465bc0e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-13T14:39:53.209 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:53 smithi154 podman[118055]: 2024-09-13 14:39:53.125823051 +0000 UTC m=+0.352128255 container init b1de06561ee0ba9b7edbe199ee760848b009785026a13ef5ffc3b20c8465bc0e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:39:53.209 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:53 smithi154 podman[118055]: 2024-09-13 14:39:53.129426399 +0000 UTC m=+0.355731601 container start b1de06561ee0ba9b7edbe199ee760848b009785026a13ef5ffc3b20c8465bc0e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-13T14:39:53.209 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:53 smithi154 podman[118055]: 2024-09-13 14:39:53.208929837 +0000 UTC m=+0.435235054 container attach b1de06561ee0ba9b7edbe199ee760848b009785026a13ef5ffc3b20c8465bc0e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-13T14:39:53.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:53 smithi154 ceph-mon[102314]: osdmap e91: 8 total, 7 up, 8 in 2024-09-13T14:39:53.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:53 smithi154 ceph-mon[107038]: osdmap e91: 8 total, 7 up, 8 in 2024-09-13T14:39:53.470 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:53 smithi154 podman[118055]: 2024-09-13 14:39:53.314871428 +0000 UTC m=+0.541176629 container died b1de06561ee0ba9b7edbe199ee760848b009785026a13ef5ffc3b20c8465bc0e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-13T14:39:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:53 smithi172 ceph-mon[100696]: osdmap e91: 8 total, 7 up, 8 in 2024-09-13T14:39:54.340 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 podman[118078]: 2024-09-13 14:39:54.038500786 +0000 UTC m=+0.717659080 container remove b1de06561ee0ba9b7edbe199ee760848b009785026a13ef5ffc3b20c8465bc0e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:39:54.340 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-13T14:39:54.340 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 systemd[1]: Stopped Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:39:54.341 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0.service: Consumed 6.484s CPU time. 2024-09-13T14:39:54.341 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 systemd[1]: Starting Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:39:54.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:54 smithi154 ceph-mon[102314]: pgmap v175: 73 pgs: 11 stale+active+clean, 62 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:54.626 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:54 smithi154 ceph-mon[107038]: pgmap v175: 73 pgs: 11 stale+active+clean, 62 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:54.626 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 podman[118179]: 2024-09-13 14:39:54.40642386 +0000 UTC m=+0.018768929 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:39:54.627 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 podman[118179]: 2024-09-13 14:39:54.506606572 +0000 UTC m=+0.118951622 container create 1f9bace6d72f800ef8477b07711eda249d918ae0b9bb982980df56d05012041f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T14:39:54.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:54 smithi172 ceph-mon[100696]: pgmap v175: 73 pgs: 11 stale+active+clean, 62 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:39:54.896 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 podman[118179]: 2024-09-13 14:39:54.759314559 +0000 UTC m=+0.371659611 container init 1f9bace6d72f800ef8477b07711eda249d918ae0b9bb982980df56d05012041f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=) 2024-09-13T14:39:54.896 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 podman[118179]: 2024-09-13 14:39:54.762812949 +0000 UTC m=+0.375158005 container start 1f9bace6d72f800ef8477b07711eda249d918ae0b9bb982980df56d05012041f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:39:54.896 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 podman[118179]: 2024-09-13 14:39:54.842577728 +0000 UTC m=+0.454922791 container attach 1f9bace6d72f800ef8477b07711eda249d918ae0b9bb982980df56d05012041f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T14:39:55.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 bash[118179]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.220 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 bash[118179]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 bash[118179]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 bash[118179]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:54 smithi154 bash[118179]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 bash[118179]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:39:55.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 bash[118179]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:39:55.705 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:39:55.705 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 bash[118179]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 bash[118179]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 bash[118179]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 bash[118179]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[118202]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:39:55.706 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 bash[118179]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:39:55.707 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:55 smithi154 podman[118179]: 2024-09-13 14:39:55.33913821 +0000 UTC m=+0.951483261 container died 1f9bace6d72f800ef8477b07711eda249d918ae0b9bb982980df56d05012041f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-13T14:39:55.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:55 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:39:56.345 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 podman[118320]: 2024-09-13 14:39:56.008066075 +0000 UTC m=+0.657430134 container remove 1f9bace6d72f800ef8477b07711eda249d918ae0b9bb982980df56d05012041f (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T14:39:56.345 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 podman[118377]: 2024-09-13 14:39:56.122068555 +0000 UTC m=+0.016309425 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:39:56.345 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 podman[118377]: 2024-09-13 14:39:56.222683659 +0000 UTC m=+0.116924558 container create 870726eb983e56dcecaaad94b84d13c5b8f07556c2bcf309b0595ae22567fca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-13T14:39:56.600 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:56 smithi154 ceph-mon[102314]: pgmap v176: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:39:56.601 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:56 smithi154 ceph-mon[107038]: pgmap v176: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:39:56.601 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 podman[118377]: 2024-09-13 14:39:56.483606953 +0000 UTC m=+0.377847810 container init 870726eb983e56dcecaaad94b84d13c5b8f07556c2bcf309b0595ae22567fca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-13T14:39:56.601 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 podman[118377]: 2024-09-13 14:39:56.487112123 +0000 UTC m=+0.381352981 container start 870726eb983e56dcecaaad94b84d13c5b8f07556c2bcf309b0595ae22567fca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T14:39:56.601 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 ceph-osd[118396]: -- 172.21.15.154:0/2510911917 <== mon.2 v2:172.21.15.154:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eaaaaf1a40 con 0x55eaab8be000 2024-09-13T14:39:56.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:56 smithi172 ceph-mon[100696]: pgmap v176: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:39:56.970 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 bash[118377]: 870726eb983e56dcecaaad94b84d13c5b8f07556c2bcf309b0595ae22567fca7 2024-09-13T14:39:56.970 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:56 smithi154 systemd[1]: Started Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:39:57.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:39:58.470 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:39:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[118392]: 2024-09-13T14:39:58.143+0000 7fc4327ea640 -1 Falling back to public interface 2024-09-13T14:39:58.952 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:58 smithi154 ceph-mon[102314]: pgmap v177: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:39:58.952 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:58.952 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:39:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:58.953 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:58 smithi154 ceph-mon[107038]: pgmap v177: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:39:58.953 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:58.953 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:39:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:39:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:58 smithi172 ceph-mon[100696]: pgmap v177: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:39:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:39:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:39:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pgmap v178: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: [WRN] OSD_DOWN: 1 osds down 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: osd.0 (root=default,host=smithi154) is down 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: [WRN] PG_DEGRADED: Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded 2024-09-13T14:40:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 3.b is active+undersized+degraded, acting [3,4] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 3.f is active+undersized+degraded, acting [7,4] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pgmap v178: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: [WRN] OSD_DOWN: 1 osds down 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: osd.0 (root=default,host=smithi154) is down 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: [WRN] PG_DEGRADED: Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-13T14:40:00.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-09-13T14:40:00.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 3.b is active+undersized+degraded, acting [3,4] 2024-09-13T14:40:00.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 3.f is active+undersized+degraded, acting [7,4] 2024-09-13T14:40:00.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-09-13T14:40:00.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-13T14:40:00.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:00 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pgmap v178: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: [WRN] OSD_DOWN: 1 osds down 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: osd.0 (root=default,host=smithi154) is down 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: [WRN] PG_DEGRADED: Degraded data redundancy: 14/84 objects degraded (16.667%), 9 pgs degraded 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 2.1 is active+undersized+degraded, acting [2,3] 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-09-13T14:40:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-13T14:40:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-09-13T14:40:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 3.b is active+undersized+degraded, acting [3,4] 2024-09-13T14:40:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 3.f is active+undersized+degraded, acting [7,4] 2024-09-13T14:40:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-09-13T14:40:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-13T14:40:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:00 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:01.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:01 smithi154 ceph-mon[102314]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:40:01.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:40:01] "GET /metrics HTTP/1.1" 200 33953 "" "Prometheus/2.51.0" 2024-09-13T14:40:01.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:01 smithi154 ceph-mon[107038]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:40:01.971 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:40:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[118392]: 2024-09-13T14:40:01.651+0000 7fc4327ea640 -1 osd.0 89 log_to_monitors true 2024-09-13T14:40:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:01 smithi172 ceph-mon[100696]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:40:02.282 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[118392]: 2024-09-13T14:40:02.147+0000 7fc42658c640 -1 osd.0 89 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:40:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[102314]: pgmap v179: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[102314]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:40:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[102314]: osdmap e92: 8 total, 7 up, 8 in 2024-09-13T14:40:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[102314]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:40:02.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[102314]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' 2024-09-13T14:40:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[107038]: pgmap v179: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[107038]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:40:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[107038]: osdmap e92: 8 total, 7 up, 8 in 2024-09-13T14:40:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[107038]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:40:02.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:02 smithi154 ceph-mon[107038]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' 2024-09-13T14:40:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:02 smithi172 ceph-mon[100696]: pgmap v179: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 264 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:02 smithi172 ceph-mon[100696]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:40:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:02 smithi172 ceph-mon[100696]: osdmap e92: 8 total, 7 up, 8 in 2024-09-13T14:40:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:02 smithi172 ceph-mon[100696]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:40:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:02 smithi172 ceph-mon[100696]: from='osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643]' entity='osd.0' 2024-09-13T14:40:03.865 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[102314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:40:03.866 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[102314]: osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643] boot 2024-09-13T14:40:03.866 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[102314]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:40:03.866 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:40:03.866 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[107038]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:40:03.866 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[107038]: osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643] boot 2024-09-13T14:40:03.866 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[107038]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:40:03.866 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:03 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:40:03.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:03 smithi172 ceph-mon[100696]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:40:03.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:03 smithi172 ceph-mon[100696]: osd.0 [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643] boot 2024-09-13T14:40:03.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:03 smithi172 ceph-mon[100696]: osdmap e93: 8 total, 8 up, 8 in 2024-09-13T14:40:03.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:03 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:40:04.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:40:04.031+0000 7f9d7ab79640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: pgmap v181: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[102314]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: pgmap v181: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:04.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:04.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:40:04.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:04 smithi154 ceph-mon[107038]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:40:04.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: pgmap v181: 73 pgs: 17 active+undersized, 9 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 14/84 objects degraded (16.667%) 2024-09-13T14:40:04.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:40:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:04 smithi172 ceph-mon[100696]: osdmap e94: 8 total, 8 up, 8 in 2024-09-13T14:40:05.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:05 smithi154 ceph-mon[102314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:40:05.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:05 smithi154 ceph-mon[102314]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-13T14:40:05.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:05 smithi154 ceph-mon[107038]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:40:05.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:05 smithi154 ceph-mon[107038]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-13T14:40:05.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:05 smithi172 ceph-mon[100696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:40:05.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:05 smithi172 ceph-mon[100696]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-13T14:40:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:06 smithi154 ceph-mon[102314]: pgmap v184: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:40:06.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:06 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:06.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:06 smithi154 ceph-mon[107038]: pgmap v184: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:40:06.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:06 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:06.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:06 smithi172 ceph-mon[100696]: pgmap v184: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:40:06.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:06 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:08.121 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:07 smithi172 ceph-mon[100696]: pgmap v185: 73 pgs: 15 peering, 58 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:08.121 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:07 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-09-13T14:40:08.121 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:07 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:40:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:07 smithi154 ceph-mon[102314]: pgmap v185: 73 pgs: 15 peering, 58 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:07 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-09-13T14:40:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:07 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:40:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:07 smithi154 ceph-mon[107038]: pgmap v185: 73 pgs: 15 peering, 58 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:07 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-09-13T14:40:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:07 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:40:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:10 smithi154 ceph-mon[102314]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:10.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:10 smithi154 ceph-mon[107038]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:10 smithi172 ceph-mon[100696]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:40:11] "GET /metrics HTTP/1.1" 200 33953 "" "Prometheus/2.51.0" 2024-09-13T14:40:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:12 smithi154 ceph-mon[102314]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:12.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:12 smithi154 ceph-mon[107038]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:12 smithi172 ceph-mon[100696]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:13 smithi154 ceph-mon[102314]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:13 smithi154 ceph-mon[107038]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:13 smithi172 ceph-mon[100696]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:15 smithi154 ceph-mon[102314]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:16.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:15 smithi154 ceph-mon[107038]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:16.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:15 smithi172 ceph-mon[100696]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:18.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:17 smithi154 ceph-mon[102314]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:18.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:17 smithi154 ceph-mon[107038]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:18.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:17 smithi172 ceph-mon[100696]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:19.439 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:19.439 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:19.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:18 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:19.538 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (7m) 20s ago 13m 18.9M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (7m) 43s ago 12m 86.7M - c8b91775d855 4bed28596716 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (11m) 20s ago 11m 20.7M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (11m) 43s ago 11m 18.1M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (8m) 43s ago 17m 455M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (8m) 20s ago 18m 518M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (6m) 20s ago 18m 51.1M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 c1bb51eb0c90 2024-09-13T14:40:20.115 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (3m) 43s ago 17m 39.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 50c3380c21bb 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (5m) 20s ago 17m 40.8M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 fcefc4676e76 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (8m) 20s ago 12m 11.6M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (8m) 43s ago 12m 12.0M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (23s) 20s ago 16m 11.3M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 870726eb983e 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (16m) 20s ago 16m 62.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e e8c2b8e5f61b 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (15m) 20s ago 15m 58.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (15m) 20s ago 15m 54.6M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (15m) 43s ago 15m 57.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (14m) 43s ago 14m 57.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (14m) 43s ago 14m 56.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (14m) 43s ago 13m 55.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:40:20.116 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (7m) 43s ago 13m 47.7M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:40:20.459 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:40:20.459 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:40:20.459 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:40:20.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:20.459 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:40:20.459 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:40:20.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 7, 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 1 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 9, 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 6 2024-09-13T14:40:20.460 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:40:20.461 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:40:20.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[102314]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:20.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:40:20.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[102314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:40:20.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[102314]: Upgrade: osd.1 is safe to restart 2024-09-13T14:40:20.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[102314]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:20.461 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[107038]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:20.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:40:20.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[107038]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:40:20.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[107038]: Upgrade: osd.1 is safe to restart 2024-09-13T14:40:20.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:19 smithi154 ceph-mon[107038]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:19 smithi172 ceph-mon[100696]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:19 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-13T14:40:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:19 smithi172 ceph-mon[100696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-13T14:40:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:19 smithi172 ceph-mon[100696]: Upgrade: osd.1 is safe to restart 2024-09-13T14:40:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:19 smithi172 ceph-mon[100696]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (3 remaining).", 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "1/8 daemons upgraded", 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:40:20.772 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:40:21.025 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:21.025 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:21.025 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1172790177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: Upgrade: Updating osd.1 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[102314]: Deploying daemon osd.1 on smithi154 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1172790177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: Upgrade: Updating osd.1 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:21.026 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:40:21.027 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:21.027 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:20 smithi154 ceph-mon[107038]: Deploying daemon osd.1 on smithi154 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1172790177' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: Upgrade: Updating osd.1 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:20 smithi172 ceph-mon[100696]: Deploying daemon osd.1 on smithi154 2024-09-13T14:40:21.930 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:40:21] "GET /metrics HTTP/1.1" 200 33950 "" "Prometheus/2.51.0" 2024-09-13T14:40:22.856 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:22 smithi154 ceph-mon[102314]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:22.856 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:22 smithi154 ceph-mon[102314]: from='client.44258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:22.856 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:22 smithi154 ceph-mon[107038]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:22.856 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:22 smithi154 ceph-mon[107038]: from='client.44258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:22 smithi172 ceph-mon[100696]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:22.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:22 smithi172 ceph-mon[100696]: from='client.44258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:23.760 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:23 smithi154 systemd[1]: Stopping Ceph osd.1 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:40:24.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:23 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[48623]: 2024-09-13T14:40:23.787+0000 7f8d660dd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:40:24.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:23 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[48623]: 2024-09-13T14:40:23.787+0000 7f8d660dd640 -1 osd.1 94 *** Got signal Terminated *** 2024-09-13T14:40:24.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:23 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[48623]: 2024-09-13T14:40:23.787+0000 7f8d660dd640 -1 osd.1 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:40:24.948 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:24 smithi154 ceph-mon[102314]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:24.948 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:24 smithi154 ceph-mon[102314]: osd.1 marked itself down and dead 2024-09-13T14:40:24.949 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:24 smithi154 ceph-mon[107038]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:24.949 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:24 smithi154 ceph-mon[107038]: osd.1 marked itself down and dead 2024-09-13T14:40:24.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:24 smithi172 ceph-mon[100696]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:24 smithi172 ceph-mon[100696]: osd.1 marked itself down and dead 2024-09-13T14:40:25.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:24 smithi154 podman[122675]: 2024-09-13 14:40:24.947955078 +0000 UTC m=+1.288143931 container stop e8c2b8e5f61b77dfae16eb91bbf68f34adbe21863c46f1920e95f08fc938fb19 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=reef-2c01458, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-13T14:40:25.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:24 smithi154 podman[122675]: 2024-09-13 14:40:24.950378647 +0000 UTC m=+1.290567515 container died e8c2b8e5f61b77dfae16eb91bbf68f34adbe21863c46f1920e95f08fc938fb19 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-13T14:40:25.539 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:25 smithi154 podman[122675]: 2024-09-13 14:40:25.318153004 +0000 UTC m=+1.658341856 container cleanup e8c2b8e5f61b77dfae16eb91bbf68f34adbe21863c46f1920e95f08fc938fb19 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T14:40:25.539 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:25 smithi154 bash[122675]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1 2024-09-13T14:40:25.937 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:25 smithi154 podman[122702]: 2024-09-13 14:40:25.611942438 +0000 UTC m=+0.656535346 container remove e8c2b8e5f61b77dfae16eb91bbf68f34adbe21863c46f1920e95f08fc938fb19 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=reef-2c01458, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:40:25.938 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:25 smithi154 ceph-mon[102314]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:40:25.938 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:25 smithi154 ceph-mon[102314]: osdmap e95: 8 total, 7 up, 8 in 2024-09-13T14:40:25.938 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:25 smithi154 ceph-mon[102314]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:40:25.938 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:25 smithi154 ceph-mon[107038]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:40:25.938 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:25 smithi154 ceph-mon[107038]: osdmap e95: 8 total, 7 up, 8 in 2024-09-13T14:40:25.938 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:25 smithi154 ceph-mon[107038]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:40:25.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:25 smithi172 ceph-mon[100696]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:40:25.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:25 smithi172 ceph-mon[100696]: osdmap e95: 8 total, 7 up, 8 in 2024-09-13T14:40:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:25 smithi172 ceph-mon[100696]: osdmap e96: 8 total, 7 up, 8 in 2024-09-13T14:40:26.191 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:25 smithi154 podman[122792]: 2024-09-13 14:40:25.840201453 +0000 UTC m=+0.016088656 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:40:26.191 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:25 smithi154 podman[122792]: 2024-09-13 14:40:25.937668829 +0000 UTC m=+0.113556027 container create 9fa02262339c41b06f1aed954ee03a6cfc581c186347b007cf2dc0461742a5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-13T14:40:26.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 podman[122792]: 2024-09-13 14:40:26.190543771 +0000 UTC m=+0.366430979 container init 9fa02262339c41b06f1aed954ee03a6cfc581c186347b007cf2dc0461742a5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-13T14:40:26.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 podman[122792]: 2024-09-13 14:40:26.19419712 +0000 UTC m=+0.370084320 container start 9fa02262339c41b06f1aed954ee03a6cfc581c186347b007cf2dc0461742a5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T14:40:26.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 podman[122792]: 2024-09-13 14:40:26.315418731 +0000 UTC m=+0.491305946 container attach 9fa02262339c41b06f1aed954ee03a6cfc581c186347b007cf2dc0461742a5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T14:40:26.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 podman[122792]: 2024-09-13 14:40:26.38502801 +0000 UTC m=+0.560915214 container died 9fa02262339c41b06f1aed954ee03a6cfc581c186347b007cf2dc0461742a5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-13T14:40:26.920 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:26 smithi154 ceph-mon[102314]: pgmap v195: 73 pgs: 9 stale+active+clean, 64 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:26.920 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:26 smithi154 ceph-mon[107038]: pgmap v195: 73 pgs: 9 stale+active+clean, 64 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:26.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:26 smithi172 ceph-mon[100696]: pgmap v195: 73 pgs: 9 stale+active+clean, 64 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:27.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 podman[122816]: 2024-09-13 14:40:26.919617489 +0000 UTC m=+0.528707363 container remove 9fa02262339c41b06f1aed954ee03a6cfc581c186347b007cf2dc0461742a5b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T14:40:27.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-13T14:40:27.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 systemd[1]: Stopped Ceph osd.1 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:40:27.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:26 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.1.service: Consumed 7.369s CPU time. 2024-09-13T14:40:27.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 systemd[1]: Starting Ceph osd.1 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:40:27.632 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:27.633 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 podman[122918]: 2024-09-13 14:40:27.255138181 +0000 UTC m=+0.016590353 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:40:27.633 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 podman[122918]: 2024-09-13 14:40:27.362691792 +0000 UTC m=+0.124143951 container create e8b03fe18ae6b8f1058e37ad6946c55eef2a15a9eb10cbbab0688ebcdbf25c79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:40:27.633 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:27.970 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 podman[122918]: 2024-09-13 14:40:27.632184785 +0000 UTC m=+0.393636945 container init e8b03fe18ae6b8f1058e37ad6946c55eef2a15a9eb10cbbab0688ebcdbf25c79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-13T14:40:27.970 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 podman[122918]: 2024-09-13 14:40:27.635763944 +0000 UTC m=+0.397216106 container start e8b03fe18ae6b8f1058e37ad6946c55eef2a15a9eb10cbbab0688ebcdbf25c79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 podman[122918]: 2024-09-13 14:40:27.707073323 +0000 UTC m=+0.468525486 container attach e8b03fe18ae6b8f1058e37ad6946c55eef2a15a9eb10cbbab0688ebcdbf25c79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 bash[122918]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 bash[122918]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 bash[122918]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 bash[122918]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 bash[122918]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:40:27.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 bash[122918]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:40:27.972 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:40:27.972 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:27 smithi154 bash[122918]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-09-13T14:40:27.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:27 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:28.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:40:28.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 bash[122918]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:40:28.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:40:28.470 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 bash[122918]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-09-13T14:40:28.471 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:40:28.471 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 bash[122918]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-13T14:40:28.471 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:40:28.471 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 bash[122918]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T14:40:28.471 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate[122933]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:40:28.471 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 bash[122918]: --> ceph-volume lvm activate successful for osd ID: 1 2024-09-13T14:40:28.471 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 podman[122918]: 2024-09-13 14:40:28.182429448 +0000 UTC m=+0.943881625 container died e8b03fe18ae6b8f1058e37ad6946c55eef2a15a9eb10cbbab0688ebcdbf25c79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T14:40:28.813 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-mon[102314]: pgmap v197: 73 pgs: 2 active+undersized, 8 stale+active+clean, 2 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:40:28.814 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:28.814 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:28.814 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-mon[107038]: pgmap v197: 73 pgs: 2 active+undersized, 8 stale+active+clean, 2 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:40:28.814 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:28.814 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:28.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:28 smithi172 ceph-mon[100696]: pgmap v197: 73 pgs: 2 active+undersized, 8 stale+active+clean, 2 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:40:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:29.131 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:28 smithi154 podman[123040]: 2024-09-13 14:40:28.813293709 +0000 UTC m=+0.624592370 container remove e8b03fe18ae6b8f1058e37ad6946c55eef2a15a9eb10cbbab0688ebcdbf25c79 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T14:40:29.131 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:29 smithi154 podman[123099]: 2024-09-13 14:40:28.925118354 +0000 UTC m=+0.016135260 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:40:29.131 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:29 smithi154 podman[123099]: 2024-09-13 14:40:29.030895607 +0000 UTC m=+0.121912511 container create 9199417c7bce908d26996813f14ad9f327e5cb6ecbf495f04ade49c5f6e39a88 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-13T14:40:29.720 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:29 smithi154 podman[123099]: 2024-09-13 14:40:29.25820641 +0000 UTC m=+0.349223313 container init 9199417c7bce908d26996813f14ad9f327e5cb6ecbf495f04ade49c5f6e39a88 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-13T14:40:29.720 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:29 smithi154 podman[123099]: 2024-09-13 14:40:29.261566017 +0000 UTC m=+0.352582921 container start 9199417c7bce908d26996813f14ad9f327e5cb6ecbf495f04ade49c5f6e39a88 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T14:40:29.720 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:29 smithi154 ceph-osd[123118]: -- 172.21.15.154:0/2658912876 <== mon.0 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b37460fa40 con 0x55b3753dc400 2024-09-13T14:40:29.720 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:29 smithi154 bash[123099]: 9199417c7bce908d26996813f14ad9f327e5cb6ecbf495f04ade49c5f6e39a88 2024-09-13T14:40:29.720 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:29 smithi154 systemd[1]: Started Ceph osd.1 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:40:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[102314]: pgmap v198: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:30.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[107038]: pgmap v198: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:30.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:30.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:30 smithi172 ceph-mon[100696]: pgmap v198: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:31.220 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:30 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T14:40:30.940+0000 7f4858414640 -1 Falling back to public interface 2024-09-13T14:40:31.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:40:31] "GET /metrics HTTP/1.1" 200 33951 "" "Prometheus/2.51.0" 2024-09-13T14:40:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:32 smithi154 ceph-mon[102314]: pgmap v199: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:32.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:32 smithi154 ceph-mon[107038]: pgmap v199: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:32.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:32 smithi172 ceph-mon[100696]: pgmap v199: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:34.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[102314]: pgmap v200: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:34.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.162 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.162 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[107038]: pgmap v200: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:34.162 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.162 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.162 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.162 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:33 smithi172 ceph-mon[100696]: pgmap v200: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:34.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:34.784 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:34 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T14:40:34.500+0000 7f4858414640 -1 osd.1 94 log_to_monitors true 2024-09-13T14:40:35.117 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:34 smithi154 ceph-mon[102314]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:40:35.117 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:34 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:35.117 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:34 smithi154 ceph-mon[107038]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:40:35.117 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:34 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:35.117 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:40:34 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T14:40:34.877+0000 7f48501be640 -1 osd.1 94 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:40:35.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:34 smithi172 ceph-mon[100696]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-13T14:40:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:34 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:40:36.142 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:35 smithi172 ceph-mon[100696]: pgmap v201: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:36.142 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:35 smithi172 ceph-mon[100696]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:40:36.142 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:35 smithi172 ceph-mon[100696]: osdmap e97: 8 total, 7 up, 8 in 2024-09-13T14:40:36.142 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:35 smithi172 ceph-mon[100696]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:40:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[102314]: pgmap v201: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[102314]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:40:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[102314]: osdmap e97: 8 total, 7 up, 8 in 2024-09-13T14:40:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[102314]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:40:36.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[107038]: pgmap v201: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:40:36.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[107038]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T14:40:36.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[107038]: osdmap e97: 8 total, 7 up, 8 in 2024-09-13T14:40:36.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:35 smithi154 ceph-mon[107038]: from='osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:40:37.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:40:36.897+0000 7f9d7ab79640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:40:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:40:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834] boot 2024-09-13T14:40:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:40:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:40:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834] boot 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:37.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: osd.1 [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834] boot 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: osdmap e98: 8 total, 8 up, 8 in 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:40:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: pgmap v204: 73 pgs: 3 peering, 25 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:40:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:40:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[102314]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: pgmap v204: 73 pgs: 3 peering, 25 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:40:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:38.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:40:38.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:40:38.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:37 smithi154 ceph-mon[107038]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:40:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: pgmap v204: 73 pgs: 3 peering, 25 active+undersized, 4 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:40:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: osdmap e99: 8 total, 8 up, 8 in 2024-09-13T14:40:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:40:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:38.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:40:38.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:40:38.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:37 smithi172 ceph-mon[100696]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-09-13T14:40:39.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:39 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:39.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:39 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:39.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:39 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:40.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:40 smithi154 ceph-mon[102314]: pgmap v206: 73 pgs: 3 peering, 8 active+undersized, 1 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:40:40.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:40:40.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:40 smithi154 ceph-mon[107038]: pgmap v206: 73 pgs: 3 peering, 8 active+undersized, 1 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:40:40.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:40:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:40 smithi172 ceph-mon[100696]: pgmap v206: 73 pgs: 3 peering, 8 active+undersized, 1 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T14:40:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded (PG_DEGRADED) 2024-09-13T14:40:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:41 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-13T14:40:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:41 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:40:41.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:41 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-13T14:40:41.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:41 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:40:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:41 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded) 2024-09-13T14:40:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:41 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:40:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:42 smithi154 ceph-mon[102314]: pgmap v207: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:40:41] "GET /metrics HTTP/1.1" 200 33951 "" "Prometheus/2.51.0" 2024-09-13T14:40:42.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:42 smithi154 ceph-mon[107038]: pgmap v207: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:42 smithi172 ceph-mon[100696]: pgmap v207: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:43 smithi154 ceph-mon[102314]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:43 smithi154 ceph-mon[107038]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:43 smithi172 ceph-mon[100696]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:45 smithi154 ceph-mon[102314]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:45 smithi154 ceph-mon[107038]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:45 smithi172 ceph-mon[100696]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:47 smithi154 ceph-mon[102314]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:47 smithi154 ceph-mon[107038]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:47 smithi172 ceph-mon[100696]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:49 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:49.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:49 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:49.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:49 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:50 smithi154 ceph-mon[102314]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:50 smithi154 ceph-mon[107038]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:50 smithi172 ceph-mon[100696]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:51.091 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:40:51.646 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:40:51.646 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (7m) 18s ago 13m 18.9M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:40:51.646 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (7m) 74s ago 13m 86.7M - c8b91775d855 4bed28596716 2024-09-13T14:40:51.646 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (12m) 18s ago 12m 20.9M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:40:51.646 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (12m) 74s ago 12m 18.1M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:40:51.646 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (8m) 74s ago 17m 455M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (9m) 18s ago 19m 519M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (6m) 18s ago 19m 55.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 c1bb51eb0c90 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (4m) 74s ago 17m 39.4M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 50c3380c21bb 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (6m) 18s ago 17m 46.2M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 fcefc4676e76 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (8m) 18s ago 13m 11.5M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (8m) 74s ago 13m 12.0M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (55s) 18s ago 17m 45.9M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 870726eb983e 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (22s) 18s ago 16m 14.5M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 9199417c7bce 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (16m) 18s ago 16m 58.7M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 3e23ed3b9f8b 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (16m) 18s ago 16m 54.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (15m) 74s ago 15m 57.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (15m) 74s ago 15m 57.0M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:40:51.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (14m) 74s ago 14m 56.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:40:51.648 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (14m) 74s ago 14m 55.1M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:40:51.648 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (8m) 74s ago 14m 47.7M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:40:51.969 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:40:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:40:51] "GET /metrics HTTP/1.1" 200 33948 "" "Prometheus/2.51.0" 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 6, 2024-09-13T14:40:51.988 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 8, 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 7 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:40:51.989 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:40:52.292 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:40:52.292 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:40:52.292 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:40:52.293 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (2 remaining).", 2024-09-13T14:40:52.293 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:40:52.293 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "2/8 daemons upgraded", 2024-09-13T14:40:52.293 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:40:52.293 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:40:52.293 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:40:52.294 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[102314]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:52.294 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[102314]: from='client.44270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.294 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[102314]: from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.294 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[102314]: from='client.44282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.294 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:40:52.294 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1527074354' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[107038]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[107038]: from='client.44270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[107038]: from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[107038]: from='client.44282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:40:52.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:52 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1527074354' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:52 smithi172 ceph-mon[100696]: pgmap v212: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:52 smithi172 ceph-mon[100696]: from='client.44270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:52 smithi172 ceph-mon[100696]: from='client.44276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:52 smithi172 ceph-mon[100696]: from='client.44282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-13T14:40:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:52 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1527074354' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:40:53.164 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:53 smithi154 ceph-mon[102314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:40:53.164 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:53 smithi154 ceph-mon[102314]: Upgrade: osd.2 is safe to restart 2024-09-13T14:40:53.165 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:53 smithi154 ceph-mon[102314]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:53.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:53 smithi154 ceph-mon[107038]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:40:53.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:53 smithi154 ceph-mon[107038]: Upgrade: osd.2 is safe to restart 2024-09-13T14:40:53.166 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:53 smithi154 ceph-mon[107038]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:53.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:53 smithi172 ceph-mon[100696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-13T14:40:53.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:53 smithi172 ceph-mon[100696]: Upgrade: osd.2 is safe to restart 2024-09-13T14:40:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:53 smithi172 ceph-mon[100696]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:40:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[107038]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[107038]: Upgrade: Updating osd.2 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[107038]: Deploying daemon osd.2 on smithi154 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[102314]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[102314]: Upgrade: Updating osd.2 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:54 smithi154 ceph-mon[102314]: Deploying daemon osd.2 on smithi154 2024-09-13T14:40:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:54 smithi172 ceph-mon[100696]: pgmap v213: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:54 smithi172 ceph-mon[100696]: Upgrade: Updating osd.2 2024-09-13T14:40:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-13T14:40:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:40:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:54 smithi172 ceph-mon[100696]: Deploying daemon osd.2 on smithi154 2024-09-13T14:40:56.199 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:56 smithi154 systemd[1]: Stopping Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:40:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:56 smithi154 ceph-mon[102314]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:56.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:56 smithi154 ceph-mon[107038]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:56.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[54339]: 2024-09-13T14:40:56.307+0000 7f9e07d77640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:40:56.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[54339]: 2024-09-13T14:40:56.307+0000 7f9e07d77640 -1 osd.2 99 *** Got signal Terminated *** 2024-09-13T14:40:56.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[54339]: 2024-09-13T14:40:56.307+0000 7f9e07d77640 -1 osd.2 99 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:40:56.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:56 smithi172 ceph-mon[100696]: pgmap v214: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:57.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:57 smithi154 ceph-mon[102314]: osd.2 marked itself down and dead 2024-09-13T14:40:57.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:57 smithi154 ceph-mon[107038]: osd.2 marked itself down and dead 2024-09-13T14:40:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:57 smithi172 ceph-mon[100696]: osd.2 marked itself down and dead 2024-09-13T14:40:57.970 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:57 smithi154 podman[127679]: 2024-09-13 14:40:57.69767113 +0000 UTC m=+1.484908418 container died 3e23ed3b9f8b167235963a0306e2ecbb8c7d904434f0207283f47053709a712b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-13T14:40:58.368 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[102314]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[102314]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[102314]: osdmap e100: 8 total, 7 up, 8 in 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[107038]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[107038]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[107038]: osdmap e100: 8 total, 7 up, 8 in 2024-09-13T14:40:58.369 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:58.370 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:58.370 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:58 smithi154 podman[127679]: 2024-09-13 14:40:58.083114634 +0000 UTC m=+1.870351921 container cleanup 3e23ed3b9f8b167235963a0306e2ecbb8c7d904434f0207283f47053709a712b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-13T14:40:58.370 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:58 smithi154 bash[127679]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2 2024-09-13T14:40:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:58 smithi172 ceph-mon[100696]: pgmap v215: 73 pgs: 73 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:40:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:58 smithi172 ceph-mon[100696]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:40:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:58 smithi172 ceph-mon[100696]: osdmap e100: 8 total, 7 up, 8 in 2024-09-13T14:40:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:40:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:40:58.661 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:58 smithi154 podman[127694]: 2024-09-13 14:40:58.368338765 +0000 UTC m=+0.666631264 container remove 3e23ed3b9f8b167235963a0306e2ecbb8c7d904434f0207283f47053709a712b (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:40:58.914 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:58 smithi154 podman[127785]: 2024-09-13 14:40:58.660941842 +0000 UTC m=+0.096054623 container create ac06e7243c32afafac78ac6aae76467a35449f06c713d76590876228346ab81b (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2) 2024-09-13T14:40:58.915 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:58 smithi154 podman[127785]: 2024-09-13 14:40:58.580727149 +0000 UTC m=+0.015839934 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:40:59.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:58 smithi154 podman[127785]: 2024-09-13 14:40:58.913773726 +0000 UTC m=+0.348886515 container init ac06e7243c32afafac78ac6aae76467a35449f06c713d76590876228346ab81b (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-13T14:40:59.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:58 smithi154 podman[127785]: 2024-09-13 14:40:58.917754998 +0000 UTC m=+0.352867783 container start ac06e7243c32afafac78ac6aae76467a35449f06c713d76590876228346ab81b (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD) 2024-09-13T14:40:59.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:59 smithi154 podman[127785]: 2024-09-13 14:40:59.030524992 +0000 UTC m=+0.465637795 container attach ac06e7243c32afafac78ac6aae76467a35449f06c713d76590876228346ab81b (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T14:40:59.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:59 smithi154 podman[127785]: 2024-09-13 14:40:59.107014747 +0000 UTC m=+0.542127531 container died ac06e7243c32afafac78ac6aae76467a35449f06c713d76590876228346ab81b (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux ) 2024-09-13T14:40:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:40:59 smithi172 ceph-mon[100696]: osdmap e101: 8 total, 7 up, 8 in 2024-09-13T14:40:59.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:40:59 smithi154 ceph-mon[102314]: osdmap e101: 8 total, 7 up, 8 in 2024-09-13T14:40:59.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:40:59 smithi154 ceph-mon[107038]: osdmap e101: 8 total, 7 up, 8 in 2024-09-13T14:41:00.169 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:59 smithi154 podman[127809]: 2024-09-13 14:40:59.734419278 +0000 UTC m=+0.621914930 container remove ac06e7243c32afafac78ac6aae76467a35449f06c713d76590876228346ab81b (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:41:00.170 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:59 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-13T14:41:00.170 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:59 smithi154 systemd[1]: Stopped Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:41:00.170 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:59 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2.service: Consumed 6.489s CPU time. 2024-09-13T14:41:00.170 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:40:59 smithi154 systemd[1]: Starting Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:41:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-mon[102314]: pgmap v218: 73 pgs: 9 active+undersized, 1 stale+active+clean, 3 active+undersized+degraded, 60 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:00.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-mon[107038]: pgmap v218: 73 pgs: 9 active+undersized, 1 stale+active+clean, 3 active+undersized+degraded, 60 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:00.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:00.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 podman[127907]: 2024-09-13 14:41:00.169100766 +0000 UTC m=+0.113038851 container create 29fa647b20b57584c528c75c8deec645c0825dbadfafe90a5068a2cae76bb94a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T14:41:00.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 podman[127907]: 2024-09-13 14:41:00.072585775 +0000 UTC m=+0.016523864 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:41:00.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 podman[127907]: 2024-09-13 14:41:00.380026493 +0000 UTC m=+0.323964571 container init 29fa647b20b57584c528c75c8deec645c0825dbadfafe90a5068a2cae76bb94a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-13T14:41:00.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 podman[127907]: 2024-09-13 14:41:00.383484081 +0000 UTC m=+0.327422157 container start 29fa647b20b57584c528c75c8deec645c0825dbadfafe90a5068a2cae76bb94a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-13T14:41:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:00 smithi172 ceph-mon[100696]: pgmap v218: 73 pgs: 9 active+undersized, 1 stale+active+clean, 3 active+undersized+degraded, 60 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:00 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:00.911 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 podman[127907]: 2024-09-13 14:41:00.483789199 +0000 UTC m=+0.427727290 container attach 29fa647b20b57584c528c75c8deec645c0825dbadfafe90a5068a2cae76bb94a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T14:41:00.911 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.911 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.911 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:41:00.912 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:41:00.913 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[127922]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 bash[127907]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:41:01.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:00 smithi154 podman[127907]: 2024-09-13 14:41:00.939776273 +0000 UTC m=+0.883714362 container died 29fa647b20b57584c528c75c8deec645c0825dbadfafe90a5068a2cae76bb94a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-13T14:41:01.874 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:01 smithi154 podman[128038]: 2024-09-13 14:41:01.540705771 +0000 UTC m=+0.589862541 container remove 29fa647b20b57584c528c75c8deec645c0825dbadfafe90a5068a2cae76bb94a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T14:41:01.874 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:01 smithi154 podman[128096]: 2024-09-13 14:41:01.649570121 +0000 UTC m=+0.016405935 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:41:01.874 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:01 smithi154 podman[128096]: 2024-09-13 14:41:01.776759086 +0000 UTC m=+0.143594896 container create d44a72dae8d347fd550049824aefe5882b028cac4d099c64f42a5a09b98e4b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-13T14:41:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:41:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:41:01] "GET /metrics HTTP/1.1" 200 33946 "" "Prometheus/2.51.0" 2024-09-13T14:41:02.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:01 smithi154 podman[128096]: 2024-09-13 14:41:01.979212727 +0000 UTC m=+0.346048537 container init d44a72dae8d347fd550049824aefe5882b028cac4d099c64f42a5a09b98e4b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T14:41:02.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:01 smithi154 podman[128096]: 2024-09-13 14:41:01.982429629 +0000 UTC m=+0.349265440 container start d44a72dae8d347fd550049824aefe5882b028cac4d099c64f42a5a09b98e4b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:41:02.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-osd[128115]: -- 172.21.15.154:0/2045123616 <== mon.0 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55963d343a40 con 0x55963e110400 2024-09-13T14:41:02.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:02 smithi154 bash[128096]: d44a72dae8d347fd550049824aefe5882b028cac4d099c64f42a5a09b98e4b78 2024-09-13T14:41:02.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:02 smithi154 systemd[1]: Started Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:41:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:02 smithi172 ceph-mon[100696]: pgmap v219: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:02.717 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[102314]: pgmap v219: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[107038]: pgmap v219: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:02.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:04.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:03 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[128111]: 2024-09-13T14:41:03.878+0000 7fd9e4c1f640 -1 Falling back to public interface 2024-09-13T14:41:04.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:04 smithi172 ceph-mon[100696]: pgmap v220: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:04.523 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:04 smithi154 ceph-mon[102314]: pgmap v220: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:04.523 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:04 smithi154 ceph-mon[107038]: pgmap v220: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:05.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:05.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:05.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:06.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:06 smithi172 ceph-mon[100696]: pgmap v221: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:06.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.893 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.893 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.906 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[102314]: pgmap v221: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[107038]: pgmap v221: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.907 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.908 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:06.908 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:07.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:07 smithi154 ceph-mon[102314]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:41:07.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:07 smithi154 ceph-mon[107038]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:41:07.721 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[128111]: 2024-09-13T14:41:07.428+0000 7fd9e4c1f640 -1 osd.2 99 log_to_monitors true 2024-09-13T14:41:07.721 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:41:07 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[128111]: 2024-09-13T14:41:07.655+0000 7fd9d89c1640 -1 osd.2 99 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:41:07.904 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:07 smithi172 ceph-mon[100696]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T14:41:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[102314]: pgmap v222: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[102314]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:41:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[102314]: osdmap e102: 8 total, 7 up, 8 in 2024-09-13T14:41:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[102314]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:08.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[107038]: pgmap v222: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:08.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[107038]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:41:08.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[107038]: osdmap e102: 8 total, 7 up, 8 in 2024-09-13T14:41:08.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:08 smithi154 ceph-mon[107038]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:08.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:08 smithi172 ceph-mon[100696]: pgmap v222: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 258 MiB used, 715 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-09-13T14:41:08.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:08 smithi172 ceph-mon[100696]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T14:41:08.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:08 smithi172 ceph-mon[100696]: osdmap e102: 8 total, 7 up, 8 in 2024-09-13T14:41:08.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:08 smithi172 ceph-mon[100696]: from='osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:09.654 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: 2024-09-13T14:41:09.543+0000 7f9d7ab79640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:41:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:41:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431] boot 2024-09-13T14:41:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:41:09.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431] boot 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:41:09.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:41:09.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: osd.2 [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431] boot 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: osdmap e103: 8 total, 8 up, 8 in 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:09.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:09.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:41:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[102314]: pgmap v225: 73 pgs: 10 peering, 7 active+undersized, 2 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:41:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[102314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:41:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[102314]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:41:10.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[102314]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:41:10.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:10.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[107038]: pgmap v225: 73 pgs: 10 peering, 7 active+undersized, 2 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:41:10.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[107038]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:41:10.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[107038]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:41:10.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[107038]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:41:10.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:10 smithi172 ceph-mon[100696]: pgmap v225: 73 pgs: 10 peering, 7 active+undersized, 2 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:41:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:10 smithi172 ceph-mon[100696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:41:10.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:10 smithi172 ceph-mon[100696]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-13T14:41:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:10 smithi172 ceph-mon[100696]: osdmap e104: 8 total, 8 up, 8 in 2024-09-13T14:41:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:41:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:41:11] "GET /metrics HTTP/1.1" 200 33946 "" "Prometheus/2.51.0" 2024-09-13T14:41:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:12 smithi154 ceph-mon[102314]: pgmap v227: 73 pgs: 10 peering, 3 active+undersized, 2 active+undersized+degraded, 58 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:41:12.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:12 smithi154 ceph-mon[107038]: pgmap v227: 73 pgs: 10 peering, 3 active+undersized, 2 active+undersized+degraded, 58 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:41:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:12 smithi172 ceph-mon[100696]: pgmap v227: 73 pgs: 10 peering, 3 active+undersized, 2 active+undersized+degraded, 58 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T14:41:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[102314]: pgmap v228: 73 pgs: 10 peering, 63 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-09-13T14:41:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:41:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[107038]: pgmap v228: 73 pgs: 10 peering, 63 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-09-13T14:41:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:13 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:41:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:13 smithi172 ceph-mon[100696]: pgmap v228: 73 pgs: 10 peering, 63 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:13 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-09-13T14:41:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:13 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:41:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:15 smithi154 ceph-mon[102314]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:16.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:15 smithi154 ceph-mon[107038]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:16.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:15 smithi172 ceph-mon[100696]: pgmap v229: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:18.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:17 smithi154 ceph-mon[102314]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:18.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:17 smithi154 ceph-mon[107038]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:18.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:17 smithi172 ceph-mon[100696]: pgmap v230: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:20.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:19 smithi154 ceph-mon[102314]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:20.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:19 smithi154 ceph-mon[107038]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:20.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:19 smithi172 ceph-mon[100696]: pgmap v231: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:21 smithi154 ceph-mon[102314]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:41:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:41:21] "GET /metrics HTTP/1.1" 200 33946 "" "Prometheus/2.51.0" 2024-09-13T14:41:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:21 smithi154 ceph-mon[107038]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:22.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:21 smithi172 ceph-mon[100696]: pgmap v232: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:22.604 INFO:teuthology.orchestra.run.smithi154.stdout:true 2024-09-13T14:41:23.179 INFO:teuthology.orchestra.run.smithi154.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:alertmanager.a smithi154 *:9093,9094 running (8m) 17s ago 14m 19.2M - 0.27.0 11f11916f8cd 4971176d5c4c 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:grafana.a smithi172 *:3000 running (8m) 17s ago 13m 83.7M - c8b91775d855 4bed28596716 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi154.daekzd smithi154 running (13m) 17s ago 13m 21.2M - 18.2.4-843-g2c01458a c129f5a2fd4e 47ae04911537 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:mds.foo.smithi172.pmmovd smithi172 running (12m) 17s ago 12m 18.5M - 18.2.4-843-g2c01458a c129f5a2fd4e 6b309841f1ab 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.x smithi172 *:8443,9283,8765 running (9m) 17s ago 18m 456M - 19.3.0-4697-gd58f7306 fec07016a2b0 b3bb4b638482 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:mgr.y smithi154 *:8443,9283,8765 running (9m) 17s ago 19m 520M - 19.3.0-4697-gd58f7306 fec07016a2b0 fc4464b991bd 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a smithi154 running (7m) 17s ago 19m 55.6M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 c1bb51eb0c90 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b smithi172 running (5m) 17s ago 18m 42.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 50c3380c21bb 2024-09-13T14:41:23.180 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c smithi154 running (6m) 17s ago 18m 44.7M 2048M 19.3.0-4697-gd58f7306 fec07016a2b0 fcefc4676e76 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.a smithi154 *:9100 running (9m) 17s ago 13m 11.9M - 1.7.0 72c9c2088986 c2e00912b58c 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:node-exporter.b smithi172 *:9100 running (9m) 17s ago 13m 12.4M - 1.7.0 72c9c2088986 60a8496c91ef 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 smithi154 running (87s) 17s ago 17m 47.3M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 870726eb983e 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 smithi154 running (54s) 17s ago 17m 43.8M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 9199417c7bce 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 smithi154 running (21s) 17s ago 16m 12.2M 2480M 19.3.0-4697-gd58f7306 fec07016a2b0 d44a72dae8d3 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 smithi154 running (16m) 17s ago 16m 55.3M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 692e375368d3 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 smithi172 running (16m) 17s ago 16m 59.2M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e 7490623a5a63 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 smithi172 running (15m) 17s ago 15m 57.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e bd1089408fdb 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 smithi172 running (15m) 17s ago 15m 56.8M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e ac940ce031a6 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 smithi172 running (15m) 17s ago 15m 56.9M 2480M 18.2.4-843-g2c01458a c129f5a2fd4e aa72c07d8bc1 2024-09-13T14:41:23.181 INFO:teuthology.orchestra.run.smithi154.stdout:prometheus.a smithi172 *:9095 running (8m) 17s ago 14m 47.8M - 2.51.0 1d3b7f56885b 12b1d8bc5be4 2024-09-13T14:41:23.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:22 smithi154 ceph-mon[102314]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:23.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:22 smithi154 ceph-mon[107038]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:23.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:22 smithi172 ceph-mon[100696]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:23.531 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:41:23.531 INFO:teuthology.orchestra.run.smithi154.stdout: "mon": { 2024-09-13T14:41:23.531 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:41:23.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:41:23.531 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr": { 2024-09-13T14:41:23.531 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 2 2024-09-13T14:41:23.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "osd": { 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 5, 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 3 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 2 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "overall": { 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 18.2.4-843-g2c01458a (2c01458affee2b2435cd231b6e392435c1c21661) reef (stable)": 7, 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev)": 8 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:41:23.532 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout: "in_progress": true, 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 4 daemons (1 remaining).", 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout: "services_complete": [], 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout: "progress": "3/8 daemons upgraded", 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout: "message": "Currently upgrading osd daemons", 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout: "is_paused": false 2024-09-13T14:41:23.828 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[102314]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[102314]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[102314]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4166428706' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[107038]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[107038]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:24.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[107038]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4166428706' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:24.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:24 smithi172 ceph-mon[100696]: pgmap v233: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:24 smithi172 ceph-mon[100696]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:24 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:24 smithi172 ceph-mon[100696]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4166428706' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:25.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[107038]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:25.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:41:25.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[107038]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:41:25.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[107038]: Upgrade: osd.3 is safe to restart 2024-09-13T14:41:25.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[102314]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:25.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:41:25.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[102314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:41:25.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:25 smithi154 ceph-mon[102314]: Upgrade: osd.3 is safe to restart 2024-09-13T14:41:25.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:25 smithi172 ceph-mon[100696]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-13T14:41:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:25 smithi172 ceph-mon[100696]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-13T14:41:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:25 smithi172 ceph-mon[100696]: Upgrade: osd.3 is safe to restart 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[102314]: pgmap v234: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[107038]: pgmap v234: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:41:26.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:26 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:26 smithi172 ceph-mon[100696]: pgmap v234: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:26 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:26 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-13T14:41:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:26 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:27 smithi154 ceph-mon[102314]: Upgrade: Updating osd.3 2024-09-13T14:41:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:27 smithi154 ceph-mon[102314]: Deploying daemon osd.3 on smithi154 2024-09-13T14:41:27.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:27 smithi154 ceph-mon[107038]: Upgrade: Updating osd.3 2024-09-13T14:41:27.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:27 smithi154 ceph-mon[107038]: Deploying daemon osd.3 on smithi154 2024-09-13T14:41:27.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:27 smithi172 ceph-mon[100696]: Upgrade: Updating osd.3 2024-09-13T14:41:27.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:27 smithi172 ceph-mon[100696]: Deploying daemon osd.3 on smithi154 2024-09-13T14:41:28.146 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:28 smithi154 ceph-mon[102314]: pgmap v235: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:28.147 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:28.147 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:28.147 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:28 smithi154 ceph-mon[107038]: pgmap v235: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:28.147 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:28.148 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:28 smithi172 ceph-mon[100696]: pgmap v235: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:29.022 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:28 smithi154 systemd[1]: Stopping Ceph osd.3 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:41:29.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:29 smithi154 ceph-mon[102314]: osd.3 marked itself down and dead 2024-09-13T14:41:29.429 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:29 smithi154 ceph-mon[107038]: osd.3 marked itself down and dead 2024-09-13T14:41:29.430 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:29 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[60271]: 2024-09-13T14:41:29.019+0000 7f7ef02b7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:41:29.430 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:29 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[60271]: 2024-09-13T14:41:29.019+0000 7f7ef02b7640 -1 osd.3 104 *** Got signal Terminated *** 2024-09-13T14:41:29.430 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:29 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[60271]: 2024-09-13T14:41:29.019+0000 7f7ef02b7640 -1 osd.3 104 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:41:29.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:29 smithi172 ceph-mon[100696]: osd.3 marked itself down and dead 2024-09-13T14:41:29.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:29 smithi154 podman[132241]: 2024-09-13 14:41:29.429307951 +0000 UTC m=+0.508245426 container died 692e375368d33615e7483e4c506b2bd6fb5af4a077b44eba52df6ac4bb9979b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T14:41:30.038 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:29 smithi154 podman[132241]: 2024-09-13 14:41:29.838042722 +0000 UTC m=+0.916980196 container cleanup 692e375368d33615e7483e4c506b2bd6fb5af4a077b44eba52df6ac4bb9979b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:41:30.038 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:29 smithi154 bash[132241]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3 2024-09-13T14:41:30.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:30 smithi154 ceph-mon[102314]: pgmap v236: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:30.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:30 smithi154 ceph-mon[102314]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:41:30.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:30 smithi154 ceph-mon[102314]: osdmap e105: 8 total, 7 up, 8 in 2024-09-13T14:41:30.433 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:30 smithi154 podman[132256]: 2024-09-13 14:41:30.039503849 +0000 UTC m=+0.607247895 container remove 692e375368d33615e7483e4c506b2bd6fb5af4a077b44eba52df6ac4bb9979b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-13T14:41:30.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:30 smithi154 ceph-mon[107038]: pgmap v236: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:30.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:30 smithi154 ceph-mon[107038]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:41:30.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:30 smithi154 ceph-mon[107038]: osdmap e105: 8 total, 7 up, 8 in 2024-09-13T14:41:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:30 smithi172 ceph-mon[100696]: pgmap v236: 73 pgs: 73 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:30 smithi172 ceph-mon[100696]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:41:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:30 smithi172 ceph-mon[100696]: osdmap e105: 8 total, 7 up, 8 in 2024-09-13T14:41:30.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:30 smithi154 podman[132346]: 2024-09-13 14:41:30.432352413 +0000 UTC m=+0.092406314 container create b1393b9ce4fade47dcdf0b9a46254f6ac4dd720f85178bae3e4fdc94828903dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T14:41:30.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:30 smithi154 podman[132346]: 2024-09-13 14:41:30.356576032 +0000 UTC m=+0.016629944 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:41:30.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:30 smithi154 podman[132346]: 2024-09-13 14:41:30.626501971 +0000 UTC m=+0.286555872 container init b1393b9ce4fade47dcdf0b9a46254f6ac4dd720f85178bae3e4fdc94828903dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-13T14:41:30.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:30 smithi154 podman[132346]: 2024-09-13 14:41:30.629805273 +0000 UTC m=+0.289859186 container start b1393b9ce4fade47dcdf0b9a46254f6ac4dd720f85178bae3e4fdc94828903dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-13T14:41:31.048 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:30 smithi154 podman[132346]: 2024-09-13 14:41:30.726506028 +0000 UTC m=+0.386559934 container attach b1393b9ce4fade47dcdf0b9a46254f6ac4dd720f85178bae3e4fdc94828903dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-13T14:41:31.048 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:30 smithi154 podman[132346]: 2024-09-13 14:41:30.814620281 +0000 UTC m=+0.474674202 container died b1393b9ce4fade47dcdf0b9a46254f6ac4dd720f85178bae3e4fdc94828903dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:41:31.372 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:31 smithi154 ceph-mon[102314]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T14:41:31.372 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:31 smithi154 ceph-mon[107038]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T14:41:31.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:31 smithi172 ceph-mon[100696]: osdmap e106: 8 total, 7 up, 8 in 2024-09-13T14:41:31.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:31 smithi154 podman[132369]: 2024-09-13 14:41:31.371967774 +0000 UTC m=+0.551153415 container remove b1393b9ce4fade47dcdf0b9a46254f6ac4dd720f85178bae3e4fdc94828903dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-13T14:41:31.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:31 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-13T14:41:31.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:31 smithi154 systemd[1]: Stopped Ceph osd.3 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:41:31.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:31 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.3.service: Consumed 6.192s CPU time. 2024-09-13T14:41:31.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:31 smithi154 systemd[1]: Starting Ceph osd.3 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:41:32.051 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-mon[102314]: pgmap v239: 73 pgs: 7 peering, 9 stale+active+clean, 57 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:32.051 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-mon[107038]: pgmap v239: 73 pgs: 7 peering, 9 stale+active+clean, 57 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:32.052 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:31 smithi154 podman[132471]: 2024-09-13 14:41:31.702790448 +0000 UTC m=+0.016163452 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:41:32.052 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:31 smithi154 podman[132471]: 2024-09-13 14:41:31.807007163 +0000 UTC m=+0.120380153 container create 26ccfd214fbaea03bfed70cb69e1a9fc928fa4c5a2ab7f4dcd619f7cd0b6ece7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-13T14:41:32.052 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:41:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:41:31] "GET /metrics HTTP/1.1" 200 33945 "" "Prometheus/2.51.0" 2024-09-13T14:41:32.470 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 podman[132471]: 2024-09-13 14:41:32.051221626 +0000 UTC m=+0.364594617 container init 26ccfd214fbaea03bfed70cb69e1a9fc928fa4c5a2ab7f4dcd619f7cd0b6ece7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 podman[132471]: 2024-09-13 14:41:32.054674617 +0000 UTC m=+0.368047608 container start 26ccfd214fbaea03bfed70cb69e1a9fc928fa4c5a2ab7f4dcd619f7cd0b6ece7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 podman[132471]: 2024-09-13 14:41:32.142992349 +0000 UTC m=+0.456365339 container attach 26ccfd214fbaea03bfed70cb69e1a9fc928fa4c5a2ab7f4dcd619f7cd0b6ece7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.471 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.472 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:41:32.472 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:41:32.472 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:41:32.472 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:41:32.472 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-13T14:41:32.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:32 smithi172 ceph-mon[100696]: pgmap v239: 73 pgs: 7 peering, 9 stale+active+clean, 57 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:32.965 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:41:32.965 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:41:32.966 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:41:32.966 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-13T14:41:32.966 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:41:32.966 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-13T14:41:32.967 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:41:32.967 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T14:41:32.967 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate[132486]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:41:32.967 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 bash[132471]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-13T14:41:32.967 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:32 smithi154 podman[132471]: 2024-09-13 14:41:32.604584466 +0000 UTC m=+0.917957460 container died 26ccfd214fbaea03bfed70cb69e1a9fc928fa4c5a2ab7f4dcd619f7cd0b6ece7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-09-13T14:41:33.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:33 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:33.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:33 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:33.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:33 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:33.642 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 podman[132597]: 2024-09-13 14:41:33.33977751 +0000 UTC m=+0.727929445 container remove 26ccfd214fbaea03bfed70cb69e1a9fc928fa4c5a2ab7f4dcd619f7cd0b6ece7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-activate, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:41:33.642 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 podman[132658]: 2024-09-13 14:41:33.532406261 +0000 UTC m=+0.099566808 container create df625f79bb4780e5a25547e8f451b439c6252d53276264f09cd1b28f154201d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True) 2024-09-13T14:41:33.642 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 podman[132658]: 2024-09-13 14:41:33.449203624 +0000 UTC m=+0.016364136 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:41:33.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 podman[132658]: 2024-09-13 14:41:33.768109582 +0000 UTC m=+0.335270089 container init df625f79bb4780e5a25547e8f451b439c6252d53276264f09cd1b28f154201d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-13T14:41:33.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 podman[132658]: 2024-09-13 14:41:33.771510997 +0000 UTC m=+0.338671503 container start df625f79bb4780e5a25547e8f451b439c6252d53276264f09cd1b28f154201d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T14:41:33.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 ceph-osd[132677]: -- 172.21.15.154:0/3139486883 <== mon.0 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ab69c59a40 con 0x55ab6aa26000 2024-09-13T14:41:33.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 bash[132658]: df625f79bb4780e5a25547e8f451b439c6252d53276264f09cd1b28f154201d9 2024-09-13T14:41:33.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:33 smithi154 systemd[1]: Started Ceph osd.3 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:41:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[107038]: pgmap v240: 73 pgs: 8 active+undersized, 7 peering, 6 stale+active+clean, 2 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:41:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:34.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[102314]: pgmap v240: 73 pgs: 8 active+undersized, 7 peering, 6 stale+active+clean, 2 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:41:34.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:34.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:34 smithi172 ceph-mon[100696]: pgmap v240: 73 pgs: 8 active+undersized, 7 peering, 6 stale+active+clean, 2 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-09-13T14:41:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:35.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T14:41:35.397+0000 7fb5fb45b640 -1 Falling back to public interface 2024-09-13T14:41:36.969 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:36 smithi154 ceph-mon[102314]: pgmap v241: 73 pgs: 20 active+undersized, 7 peering, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:36.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:36 smithi154 ceph-mon[107038]: pgmap v241: 73 pgs: 20 active+undersized, 7 peering, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:36.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:36 smithi172 ceph-mon[100696]: pgmap v241: 73 pgs: 20 active+undersized, 7 peering, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[102314]: pgmap v242: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[107038]: pgmap v242: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:38 smithi172 ceph-mon[100696]: pgmap v242: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:39.133 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:38 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T14:41:38.968+0000 7fb5fb45b640 -1 osd.3 104 log_to_monitors true 2024-09-13T14:41:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:39 smithi154 ceph-mon[102314]: from='osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:41:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:39 smithi154 ceph-mon[102314]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:41:39.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:39 smithi154 ceph-mon[107038]: from='osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:41:39.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:39 smithi154 ceph-mon[107038]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:41:39.721 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 14:41:39 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T14:41:39.494+0000 7fb5ef1fd640 -1 osd.3 104 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:41:39.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:39 smithi172 ceph-mon[100696]: from='osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:41:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:39 smithi172 ceph-mon[100696]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-13T14:41:40.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:40 smithi172 ceph-mon[100696]: pgmap v243: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:40 smithi172 ceph-mon[100696]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:41:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:40 smithi172 ceph-mon[100696]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T14:41:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:40 smithi172 ceph-mon[100696]: from='osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:40 smithi172 ceph-mon[100696]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[102314]: pgmap v243: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[102314]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:41:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[102314]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T14:41:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[102314]: from='osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[102314]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[107038]: pgmap v243: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 255 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[107038]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T14:41:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[107038]: osdmap e107: 8 total, 7 up, 8 in 2024-09-13T14:41:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[107038]: from='osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[107038]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:41:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:41.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:41:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861] boot 2024-09-13T14:41:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: osdmap e108: 8 total, 8 up, 8 in 2024-09-13T14:41:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:41:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:41.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:41.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:41.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:41:41.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:41:41.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:41.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:41:41.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:41:41.725 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:41:41.725 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861] boot 2024-09-13T14:41:41.725 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: osdmap e108: 8 total, 8 up, 8 in 2024-09-13T14:41:41.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:41:41.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:41.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:41.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.727 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.728 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.728 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:41:41.728 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:41:41.728 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:41.728 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:41:41.728 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:41:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T14:41:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: osd.3 [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861] boot 2024-09-13T14:41:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: osdmap e108: 8 total, 8 up, 8 in 2024-09-13T14:41:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-13T14:41:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:41.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:41.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:41.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-13T14:41:41.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mgr"} : dispatch 2024-09-13T14:41:41.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-13T14:41:41.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:41.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: osdmap e109: 8 total, 8 up, 8 in 2024-09-13T14:41:41.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:41 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-13T14:41:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:41:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:41:41] "GET /metrics HTTP/1.1" 200 33945 "" "Prometheus/2.51.0" 2024-09-13T14:41:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: pgmap v246: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: Upgrade: Finalizing container_image settings 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: Upgrade: Complete! 2024-09-13T14:41:42.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: pgmap v246: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: Upgrade: Finalizing container_image settings 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: Upgrade: Complete! 2024-09-13T14:41:42.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:42.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:42.976 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: pgmap v246: 73 pgs: 27 active+undersized, 4 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: Upgrade: Setting container_image for all nvmeof 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: Upgrade: Finalizing container_image settings 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.crash"} : dispatch 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "osd"} : dispatch 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mds"} : dispatch 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rgw"} : dispatch 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"} : dispatch 2024-09-13T14:41:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.iscsi"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nfs"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon"} : dispatch 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: Upgrade: Complete! 2024-09-13T14:41:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"} : dispatch 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:41:42.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[102314]: pgmap v248: 73 pgs: 21 active+undersized, 3 active+undersized+degraded, 49 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[107038]: pgmap v248: 73 pgs: 21 active+undersized, 3 active+undersized+degraded, 49 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:43 smithi172 ceph-mon[100696]: pgmap v248: 73 pgs: 21 active+undersized, 3 active+undersized+degraded, 49 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-09-13T14:41:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:45.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:45.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:41:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:45 smithi154 ceph-mon[102314]: pgmap v249: 73 pgs: 3 active+undersized, 70 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:45 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded) 2024-09-13T14:41:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:45 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:41:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:45 smithi154 ceph-mon[107038]: pgmap v249: 73 pgs: 3 active+undersized, 70 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:45 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded) 2024-09-13T14:41:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:45 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:41:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:45 smithi172 ceph-mon[100696]: pgmap v249: 73 pgs: 3 active+undersized, 70 active+clean; 4.6 MiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:45 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded) 2024-09-13T14:41:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:45 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:41:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:47 smithi154 ceph-mon[102314]: pgmap v250: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:47 smithi154 ceph-mon[107038]: pgmap v250: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:47 smithi172 ceph-mon[100696]: pgmap v250: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:49 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:49.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:49 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:49.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:49 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:50 smithi154 ceph-mon[102314]: pgmap v251: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:50 smithi154 ceph-mon[107038]: pgmap v251: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:50 smithi172 ceph-mon[100696]: pgmap v251: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:52 smithi154 ceph-mon[102314]: pgmap v252: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:52.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:41:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:41:51] "GET /metrics HTTP/1.1" 200 33799 "" "Prometheus/2.51.0" 2024-09-13T14:41:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:52 smithi154 ceph-mon[107038]: pgmap v252: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:52 smithi172 ceph-mon[100696]: pgmap v252: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:54.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:54 smithi154 ceph-mon[102314]: pgmap v253: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:54.465 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:54 smithi154 ceph-mon[107038]: pgmap v253: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:54.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:54 smithi172 ceph-mon[100696]: pgmap v253: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:54.926 DEBUG:teuthology.parallel:result is None 2024-09-13T14:41:54.926 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:41:54.944 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-13T14:41:54.944 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-13T14:41:54.960 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T14:41:54.961 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T14:41:54.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T14:41:55.168 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:55 smithi154 ceph-mon[102314]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:55 smithi154 ceph-mon[107038]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:55.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:55 smithi172 ceph-mon[100696]: from='client.44342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:41:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:56 smithi154 ceph-mon[102314]: pgmap v254: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:56.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:56 smithi154 ceph-mon[107038]: pgmap v254: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:56.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:56 smithi172 ceph-mon[100696]: pgmap v254: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:56.858 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:41:57.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3165985549' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:41:57.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3165985549' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:41:57.467 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":39,"flags":0,"active_gid":25736,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3795893612},{"type":"v1","addr":"172.21.15.154:6801","nonce":3795893612}]},"active_addr":"172.21.15.154:6801/3795893612","active_change":"2024-09-13T14:34:12.642035+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25357,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":89,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2935684456}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2241985076}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":3152436209}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":3802125794}]}]} 2024-09-13T14:41:57.474 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T14:41:57.474 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T14:41:57.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:41:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3165985549' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-13T14:41:57.674 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:41:58.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:58 smithi154 ceph-mon[102314]: pgmap v255: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:58.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:58.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:41:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:58.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:58 smithi154 ceph-mon[107038]: pgmap v255: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:58.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:58.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:41:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:58 smithi172 ceph-mon[100696]: pgmap v255: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:41:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:41:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:41:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:41:59.311 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:41:59.311 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":109,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T14:41:41.473972+0000","last_up_change":"2024-09-13T14:41:40.472228+0000","last_in_change":"2024-09-13T14:26:09.317846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:28:12.645576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:28:13.527124+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":89,"up_from":93,"up_thru":108,"down_at":90,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6803","nonce":3729051643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6805","nonce":3729051643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6809","nonce":3729051643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6807","nonce":3729051643}]},"public_addr":"172.21.15.154:6803/3729051643","cluster_addr":"172.21.15.154:6805/3729051643","heartbeat_back_addr":"172.21.15.154:6809/3729051643","heartbeat_front_addr":"172.21.15.154:6807/3729051643","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":94,"up_from":98,"up_thru":108,"down_at":95,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6811","nonce":148745834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6813","nonce":148745834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6817","nonce":148745834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6815","nonce":148745834}]},"public_addr":"172.21.15.154:6811/148745834","cluster_addr":"172.21.15.154:6813/148745834","heartbeat_back_addr":"172.21.15.154:6817/148745834","heartbeat_front_addr":"172.21.15.154:6815/148745834","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":99,"up_from":103,"up_thru":108,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6819","nonce":2711669431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6821","nonce":2711669431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6825","nonce":2711669431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6823","nonce":2711669431}]},"public_addr":"172.21.15.154:6819/2711669431","cluster_addr":"172.21.15.154:6821/2711669431","heartbeat_back_addr":"172.21.15.154:6825/2711669431","heartbeat_front_addr":"172.21.15.154:6823/2711669431","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":104,"up_from":108,"up_thru":108,"down_at":105,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6827","nonce":381867861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6829","nonce":381867861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6833","nonce":381867861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6831","nonce":381867861}]},"public_addr":"172.21.15.154:6827/381867861","cluster_addr":"172.21.15.154:6829/381867861","heartbeat_back_addr":"172.21.15.154:6833/381867861","heartbeat_front_addr":"172.21.15.154:6831/381867861","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":89},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":94},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":99},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":104},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/3255496958":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:0/4103428848":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/2090383084":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/1584830159":"2024-09-14T14:34:12.641827+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/18164567":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/3191030600":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:6835/3129749965":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/213038927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3680637464":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/3924873105":"2024-09-14T14:31:04.278426+0000","172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/2189531829":"2024-09-14T14:34:12.641827+0000","172.21.15.172:0/2607163146":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1379294727":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/2289953927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/3573827628":"2024-09-14T14:31:33.289382+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.172:6834/3129749965":"2024-09-14T14:31:33.289382+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:41:59.857 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T14:41:59.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T14:42:00.055 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:00.176 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:00 smithi154 ceph-mon[102314]: pgmap v256: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:00.176 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1414003815' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:42:00.177 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:00 smithi154 ceph-mon[107038]: pgmap v256: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:00.177 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1414003815' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:42:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:00 smithi172 ceph-mon[100696]: pgmap v256: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1414003815' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:42:01.781 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:42:01.782 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":109,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T14:41:41.473972+0000","last_up_change":"2024-09-13T14:41:40.472228+0000","last_in_change":"2024-09-13T14:26:09.317846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:28:12.645576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:28:13.527124+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":89,"up_from":93,"up_thru":108,"down_at":90,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6803","nonce":3729051643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6805","nonce":3729051643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6809","nonce":3729051643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3729051643},{"type":"v1","addr":"172.21.15.154:6807","nonce":3729051643}]},"public_addr":"172.21.15.154:6803/3729051643","cluster_addr":"172.21.15.154:6805/3729051643","heartbeat_back_addr":"172.21.15.154:6809/3729051643","heartbeat_front_addr":"172.21.15.154:6807/3729051643","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":94,"up_from":98,"up_thru":108,"down_at":95,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6811","nonce":148745834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6813","nonce":148745834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6817","nonce":148745834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6815","nonce":148745834}]},"public_addr":"172.21.15.154:6811/148745834","cluster_addr":"172.21.15.154:6813/148745834","heartbeat_back_addr":"172.21.15.154:6817/148745834","heartbeat_front_addr":"172.21.15.154:6815/148745834","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":99,"up_from":103,"up_thru":108,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6819","nonce":2711669431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6821","nonce":2711669431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6825","nonce":2711669431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2711669431},{"type":"v1","addr":"172.21.15.154:6823","nonce":2711669431}]},"public_addr":"172.21.15.154:6819/2711669431","cluster_addr":"172.21.15.154:6821/2711669431","heartbeat_back_addr":"172.21.15.154:6825/2711669431","heartbeat_front_addr":"172.21.15.154:6823/2711669431","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":104,"up_from":108,"up_thru":108,"down_at":105,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6827","nonce":381867861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6829","nonce":381867861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6833","nonce":381867861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6831","nonce":381867861}]},"public_addr":"172.21.15.154:6827/381867861","cluster_addr":"172.21.15.154:6829/381867861","heartbeat_back_addr":"172.21.15.154:6833/381867861","heartbeat_front_addr":"172.21.15.154:6831/381867861","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":89},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":94},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":99},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":104},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/3255496958":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:0/4103428848":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/2090383084":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/1584830159":"2024-09-14T14:34:12.641827+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/18164567":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/3191030600":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:6835/3129749965":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/213038927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3680637464":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/3924873105":"2024-09-14T14:31:04.278426+0000","172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/2189531829":"2024-09-14T14:34:12.641827+0000","172.21.15.172:0/2607163146":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1379294727":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/2289953927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/3573827628":"2024-09-14T14:31:33.289382+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.172:6834/3129749965":"2024-09-14T14:31:33.289382+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T14:42:02.068 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:42:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:42:01] "GET /metrics HTTP/1.1" 200 33945 "" "Prometheus/2.51.0" 2024-09-13T14:42:02.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:02 smithi154 ceph-mon[102314]: pgmap v257: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:02.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/982230267' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:42:02.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:02 smithi154 ceph-mon[107038]: pgmap v257: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:02.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/982230267' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:42:02.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T14:42:02.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T14:42:02.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T14:42:02.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T14:42:02.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T14:42:02.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T14:42:02.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T14:42:02.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T14:42:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:02 smithi172 ceph-mon[100696]: pgmap v257: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/982230267' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T14:42:02.726 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.840 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.841 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.841 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.841 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.847 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.848 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:02.848 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:04.080 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:04 smithi154 ceph-mon[102314]: pgmap v258: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:04.081 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:04 smithi154 ceph-mon[107038]: pgmap v258: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:04.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:04 smithi172 ceph-mon[100696]: pgmap v258: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:06.373 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:06 smithi154 ceph-mon[102314]: pgmap v259: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:06.374 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:06 smithi154 ceph-mon[107038]: pgmap v259: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:06.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:06 smithi172 ceph-mon[100696]: pgmap v259: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:08 smithi154 ceph-mon[102314]: pgmap v260: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:08 smithi154 ceph-mon[107038]: pgmap v260: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:08.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:08 smithi172 ceph-mon[100696]: pgmap v260: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:10.292 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:10 smithi154 ceph-mon[102314]: pgmap v261: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:10.293 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:10 smithi154 ceph-mon[107038]: pgmap v261: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:10 smithi172 ceph-mon[100696]: pgmap v261: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:11.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:42:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:42:11] "GET /metrics HTTP/1.1" 200 33945 "" "Prometheus/2.51.0" 2024-09-13T14:42:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:12 smithi154 ceph-mon[102314]: pgmap v262: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:12.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:12 smithi154 ceph-mon[107038]: pgmap v262: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:12.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:12 smithi172 ceph-mon[100696]: pgmap v262: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:13.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:13.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:13.632 INFO:teuthology.orchestra.run.smithi154.stdout:399431958556 2024-09-13T14:42:13.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T14:42:14.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:14 smithi154 ceph-mon[102314]: pgmap v263: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:14.141 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:14 smithi154 ceph-mon[107038]: pgmap v263: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:14.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:14 smithi172 ceph-mon[100696]: pgmap v263: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:14.884 INFO:teuthology.orchestra.run.smithi154.stdout:231928234177 2024-09-13T14:42:14.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T14:42:15.388 INFO:teuthology.orchestra.run.smithi154.stdout:163208757450 2024-09-13T14:42:15.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T14:42:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:16 smithi154 ceph-mon[102314]: pgmap v264: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:16.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:16 smithi154 ceph-mon[107038]: pgmap v264: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:16.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:16 smithi172 ceph-mon[100696]: pgmap v264: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:16.706 INFO:teuthology.orchestra.run.smithi154.stdout:442381631502 2024-09-13T14:42:16.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T14:42:18.270 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:18 smithi154 ceph-mon[102314]: pgmap v265: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:18.271 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:18 smithi154 ceph-mon[107038]: pgmap v265: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:18.292 INFO:teuthology.orchestra.run.smithi154.stdout:463856467977 2024-09-13T14:42:18.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T14:42:18.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:18 smithi172 ceph-mon[100696]: pgmap v265: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:18.720 INFO:teuthology.orchestra.run.smithi154.stdout:197568495814 2024-09-13T14:42:18.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T14:42:19.121 INFO:teuthology.orchestra.run.smithi154.stdout:128849019089 2024-09-13T14:42:19.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T14:42:19.586 INFO:teuthology.orchestra.run.smithi154.stdout:420906795030 2024-09-13T14:42:19.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T14:42:19.719 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:19.724 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:19.725 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:19.725 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:19.730 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:19.744 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:19.781 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:20.151 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:20 smithi154 ceph-mon[102314]: pgmap v266: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:20.152 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:20 smithi154 ceph-mon[107038]: pgmap v266: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:20 smithi172 ceph-mon[100696]: pgmap v266: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:22.146 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:22 smithi154 ceph-mon[102314]: pgmap v267: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:22.146 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:22 smithi154 ceph-mon[107038]: pgmap v267: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:22.147 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:42:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:42:21] "GET /metrics HTTP/1.1" 200 33946 "" "Prometheus/2.51.0" 2024-09-13T14:42:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:22 smithi172 ceph-mon[100696]: pgmap v267: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:24 smithi154 ceph-mon[102314]: pgmap v268: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:24 smithi154 ceph-mon[107038]: pgmap v268: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:24.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:24 smithi172 ceph-mon[100696]: pgmap v268: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:26.359 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:26 smithi154 ceph-mon[102314]: pgmap v269: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:26.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2438138623' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:42:26.360 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:26 smithi154 ceph-mon[107038]: pgmap v269: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:26.360 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2438138623' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:42:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:26 smithi172 ceph-mon[100696]: pgmap v269: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2438138623' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T14:42:26.624 INFO:teuthology.orchestra.run.smithi154.stdout:231928234180 2024-09-13T14:42:27.149 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2915332337' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:42:27.150 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2915332337' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:42:27.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2915332337' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T14:42:27.584 INFO:teuthology.orchestra.run.smithi154.stdout:163208757453 2024-09-13T14:42:28.177 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757450 got 163208757453 for osd.5 2024-09-13T14:42:28.177 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:28 smithi154 ceph-mon[102314]: pgmap v270: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:28 smithi154 ceph-mon[107038]: pgmap v270: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:28 smithi172 ceph-mon[100696]: pgmap v270: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:28.768 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234177 got 231928234180 for osd.7 2024-09-13T14:42:28.769 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:29.197 INFO:teuthology.orchestra.run.smithi154.stdout:197568495817 2024-09-13T14:42:29.808 INFO:teuthology.orchestra.run.smithi154.stdout:442381631506 2024-09-13T14:42:30.390 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:30 smithi154 ceph-mon[102314]: pgmap v271: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:30.391 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:30 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/972491168' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:42:30.391 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:30 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/924674687' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:42:30.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:30 smithi154 ceph-mon[107038]: pgmap v271: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:30.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:30 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/972491168' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:42:30.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:30 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/924674687' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:42:30.392 INFO:teuthology.orchestra.run.smithi154.stdout:463856467979 2024-09-13T14:42:30.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:30 smithi172 ceph-mon[100696]: pgmap v271: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:30.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:30 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/972491168' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T14:42:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:30 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/924674687' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T14:42:30.980 INFO:teuthology.orchestra.run.smithi154.stdout:128849019092 2024-09-13T14:42:31.390 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:31 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3396878925' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:42:31.390 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:31 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1734294971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:42:31.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:31 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3396878925' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:42:31.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:31 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1734294971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:42:31.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:31 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3396878925' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T14:42:31.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:31 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1734294971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T14:42:31.716 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019089 got 128849019092 for osd.4 2024-09-13T14:42:31.716 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:31.949 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:42:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:42:31] "GET /metrics HTTP/1.1" 200 33943 "" "Prometheus/2.51.0" 2024-09-13T14:42:32.302 INFO:teuthology.orchestra.run.smithi154.stdout:399431958559 2024-09-13T14:42:32.302 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:32 smithi154 ceph-mon[102314]: pgmap v272: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:32.303 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:32 smithi154 ceph-mon[107038]: pgmap v272: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:32.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:32 smithi172 ceph-mon[100696]: pgmap v272: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:32.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 442381631502 got 442381631506 for osd.2 2024-09-13T14:42:32.835 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:33.419 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3554295483' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:42:33.419 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3554295483' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:42:33.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 399431958556 got 399431958559 for osd.0 2024-09-13T14:42:33.441 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:33.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3554295483' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T14:42:33.496 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:33.844 INFO:tasks.cephadm.ceph_manager.ceph:need seq 463856467977 got 463856467979 for osd.3 2024-09-13T14:42:33.845 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:34.318 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:34 smithi154 ceph-mon[102314]: pgmap v273: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:34.318 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:34 smithi154 ceph-mon[107038]: pgmap v273: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:34.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:34 smithi172 ceph-mon[100696]: pgmap v273: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:34.815 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495814 got 197568495817 for osd.6 2024-09-13T14:42:34.815 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:35.920 INFO:teuthology.orchestra.run.smithi154.stdout:420906795033 2024-09-13T14:42:36.134 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:36 smithi154 ceph-mon[102314]: pgmap v274: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:36.134 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/288683232' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:42:36.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:36 smithi154 ceph-mon[107038]: pgmap v274: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:36.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/288683232' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:42:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:36 smithi172 ceph-mon[100696]: pgmap v274: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/288683232' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T14:42:36.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 420906795030 got 420906795033 for osd.1 2024-09-13T14:42:36.552 DEBUG:teuthology.parallel:result is None 2024-09-13T14:42:36.552 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:42:36.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:36.752 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:38.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:38 smithi154 ceph-mon[102314]: pgmap v275: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:38.185 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:38 smithi154 ceph-mon[107038]: pgmap v275: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:38.448 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:42:38.449 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:42:38.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:38 smithi172 ceph-mon[100696]: pgmap v275: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:38.976 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-13T14:42:36.774335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":262208,"kb_used_data":23496,"kb_used_omap":34,"kb_used_meta":238557,"kb_avail":749699008,"statfs":{"total":767960285184,"available":767691784192,"internally_reserved":0,"allocated":24059904,"data_stored":19336071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34821,"internal_metadata":244283387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003069"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488438+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:41:41.488438+0000","last_peered":"2024-09-13T14:41:41.488438+0000","last_clean":"2024-09-13T14:41:41.488438+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:41:41.488438+0000","last_undegraded":"2024-09-13T14:41:41.488438+0000","last_fullsized":"2024-09-13T14:41:41.488438+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481357+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:41:41.481357+0000","last_peered":"2024-09-13T14:41:41.481357+0000","last_clean":"2024-09-13T14:41:41.481357+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:41:41.481357+0000","last_undegraded":"2024-09-13T14:41:41.481357+0000","last_fullsized":"2024-09-13T14:41:41.481357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493742+0000","last_change":"2024-09-13T14:41:41.493742+0000","last_active":"2024-09-13T14:41:41.493742+0000","last_peered":"2024-09-13T14:41:41.493742+0000","last_clean":"2024-09-13T14:41:41.493742+0000","last_became_active":"2024-09-13T14:41:41.493388+0000","last_became_peered":"2024-09-13T14:41:41.493388+0000","last_unstale":"2024-09-13T14:41:41.493742+0000","last_undegraded":"2024-09-13T14:41:41.493742+0000","last_fullsized":"2024-09-13T14:41:41.493742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:46.304592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496213+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:41:41.496213+0000","last_peered":"2024-09-13T14:41:41.496213+0000","last_clean":"2024-09-13T14:41:41.496213+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:41:41.496213+0000","last_undegraded":"2024-09-13T14:41:41.496213+0000","last_fullsized":"2024-09-13T14:41:41.496213+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490816+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:41:41.490816+0000","last_peered":"2024-09-13T14:41:41.490816+0000","last_clean":"2024-09-13T14:41:41.490816+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:41:41.490816+0000","last_undegraded":"2024-09-13T14:41:41.490816+0000","last_fullsized":"2024-09-13T14:41:41.490816+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":56,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482849+0000","last_change":"2024-09-13T14:28:14.292467+0000","last_active":"2024-09-13T14:41:41.482849+0000","last_peered":"2024-09-13T14:41:41.482849+0000","last_clean":"2024-09-13T14:41:41.482849+0000","last_became_active":"2024-09-13T14:28:14.292208+0000","last_became_peered":"2024-09-13T14:28:14.292208+0000","last_unstale":"2024-09-13T14:41:41.482849+0000","last_undegraded":"2024-09-13T14:41:41.482849+0000","last_fullsized":"2024-09-13T14:41:41.482849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:51.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486054+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:41:41.486054+0000","last_peered":"2024-09-13T14:41:41.486054+0000","last_clean":"2024-09-13T14:41:41.486054+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:41:41.486054+0000","last_undegraded":"2024-09-13T14:41:41.486054+0000","last_fullsized":"2024-09-13T14:41:41.486054+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479873+0000","last_change":"2024-09-13T14:40:04.161607+0000","last_active":"2024-09-13T14:41:41.479873+0000","last_peered":"2024-09-13T14:41:41.479873+0000","last_clean":"2024-09-13T14:41:41.479873+0000","last_became_active":"2024-09-13T14:40:04.161226+0000","last_became_peered":"2024-09-13T14:40:04.161226+0000","last_unstale":"2024-09-13T14:41:41.479873+0000","last_undegraded":"2024-09-13T14:41:41.479873+0000","last_fullsized":"2024-09-13T14:41:41.479873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:21.273027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484875+0000","last_change":"2024-09-13T14:40:04.164112+0000","last_active":"2024-09-13T14:41:41.484875+0000","last_peered":"2024-09-13T14:41:41.484875+0000","last_clean":"2024-09-13T14:41:41.484875+0000","last_became_active":"2024-09-13T14:40:04.163753+0000","last_became_peered":"2024-09-13T14:40:04.163753+0000","last_unstale":"2024-09-13T14:41:41.484875+0000","last_undegraded":"2024-09-13T14:41:41.484875+0000","last_fullsized":"2024-09-13T14:41:41.484875+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.685538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":55,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477581+0000","last_change":"2024-09-13T14:28:14.409515+0000","last_active":"2024-09-13T14:41:40.477581+0000","last_peered":"2024-09-13T14:41:40.477581+0000","last_clean":"2024-09-13T14:41:40.477581+0000","last_became_active":"2024-09-13T14:28:14.409202+0000","last_became_peered":"2024-09-13T14:28:14.409202+0000","last_unstale":"2024-09-13T14:41:40.477581+0000","last_undegraded":"2024-09-13T14:41:40.477581+0000","last_fullsized":"2024-09-13T14:41:40.477581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:24.530144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484456+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:41:41.484456+0000","last_peered":"2024-09-13T14:41:41.484456+0000","last_clean":"2024-09-13T14:41:41.484456+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:41:41.484456+0000","last_undegraded":"2024-09-13T14:41:41.484456+0000","last_fullsized":"2024-09-13T14:41:41.484456+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493612+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:41:41.493612+0000","last_peered":"2024-09-13T14:41:41.493612+0000","last_clean":"2024-09-13T14:41:41.493612+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:41:41.493612+0000","last_undegraded":"2024-09-13T14:41:41.493612+0000","last_fullsized":"2024-09-13T14:41:41.493612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484512+0000","last_change":"2024-09-13T14:28:14.289966+0000","last_active":"2024-09-13T14:41:41.484512+0000","last_peered":"2024-09-13T14:41:41.484512+0000","last_clean":"2024-09-13T14:41:41.484512+0000","last_became_active":"2024-09-13T14:28:14.289682+0000","last_became_peered":"2024-09-13T14:28:14.289682+0000","last_unstale":"2024-09-13T14:41:41.484512+0000","last_undegraded":"2024-09-13T14:41:41.484512+0000","last_fullsized":"2024-09-13T14:41:41.484512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:37.096298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":69,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483095+0000","last_change":"2024-09-13T14:40:36.880233+0000","last_active":"2024-09-13T14:41:40.483095+0000","last_peered":"2024-09-13T14:41:40.483095+0000","last_clean":"2024-09-13T14:41:40.483095+0000","last_became_active":"2024-09-13T14:40:36.879886+0000","last_became_peered":"2024-09-13T14:40:36.879886+0000","last_unstale":"2024-09-13T14:41:40.483095+0000","last_undegraded":"2024-09-13T14:41:40.483095+0000","last_fullsized":"2024-09-13T14:41:40.483095+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:49.693282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491428+0000","last_change":"2024-09-13T14:41:41.490296+0000","last_active":"2024-09-13T14:41:41.491428+0000","last_peered":"2024-09-13T14:41:41.491428+0000","last_clean":"2024-09-13T14:41:41.491428+0000","last_became_active":"2024-09-13T14:41:41.489937+0000","last_became_peered":"2024-09-13T14:41:41.489937+0000","last_unstale":"2024-09-13T14:41:41.491428+0000","last_undegraded":"2024-09-13T14:41:41.491428+0000","last_fullsized":"2024-09-13T14:41:41.491428+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:30.748843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482550+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:41:41.482550+0000","last_peered":"2024-09-13T14:41:41.482550+0000","last_clean":"2024-09-13T14:41:41.482550+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:41:41.482550+0000","last_undegraded":"2024-09-13T14:41:41.482550+0000","last_fullsized":"2024-09-13T14:41:41.482550+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491300+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:41:41.491300+0000","last_peered":"2024-09-13T14:41:41.491300+0000","last_clean":"2024-09-13T14:41:41.491300+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:41:41.491300+0000","last_undegraded":"2024-09-13T14:41:41.491300+0000","last_fullsized":"2024-09-13T14:41:41.491300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491488+0000","last_change":"2024-09-13T14:41:41.491050+0000","last_active":"2024-09-13T14:41:41.491488+0000","last_peered":"2024-09-13T14:41:41.491488+0000","last_clean":"2024-09-13T14:41:41.491488+0000","last_became_active":"2024-09-13T14:41:41.490645+0000","last_became_peered":"2024-09-13T14:41:41.490645+0000","last_unstale":"2024-09-13T14:41:41.491488+0000","last_undegraded":"2024-09-13T14:41:41.491488+0000","last_fullsized":"2024-09-13T14:41:41.491488+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:01.779411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483326+0000","last_change":"2024-09-13T14:40:36.879533+0000","last_active":"2024-09-13T14:41:40.483326+0000","last_peered":"2024-09-13T14:41:40.483326+0000","last_clean":"2024-09-13T14:41:40.483326+0000","last_became_active":"2024-09-13T14:40:36.879212+0000","last_became_peered":"2024-09-13T14:40:36.879212+0000","last_unstale":"2024-09-13T14:41:40.483326+0000","last_undegraded":"2024-09-13T14:41:40.483326+0000","last_fullsized":"2024-09-13T14:41:40.483326+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:03.087074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481810+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:41:41.481810+0000","last_peered":"2024-09-13T14:41:41.481810+0000","last_clean":"2024-09-13T14:41:41.481810+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:41:41.481810+0000","last_undegraded":"2024-09-13T14:41:41.481810+0000","last_fullsized":"2024-09-13T14:41:41.481810+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481078+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:41:40.481078+0000","last_peered":"2024-09-13T14:41:40.481078+0000","last_clean":"2024-09-13T14:41:40.481078+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:41:40.481078+0000","last_undegraded":"2024-09-13T14:41:40.481078+0000","last_fullsized":"2024-09-13T14:41:40.481078+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481692+0000","last_change":"2024-09-13T14:40:36.875852+0000","last_active":"2024-09-13T14:41:41.481692+0000","last_peered":"2024-09-13T14:41:41.481692+0000","last_clean":"2024-09-13T14:41:41.481692+0000","last_became_active":"2024-09-13T14:40:36.875486+0000","last_became_peered":"2024-09-13T14:40:36.875486+0000","last_unstale":"2024-09-13T14:41:41.481692+0000","last_undegraded":"2024-09-13T14:41:41.481692+0000","last_fullsized":"2024-09-13T14:41:41.481692+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:18.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488653+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:41:41.488653+0000","last_peered":"2024-09-13T14:41:41.488653+0000","last_clean":"2024-09-13T14:41:41.488653+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:41:41.488653+0000","last_undegraded":"2024-09-13T14:41:41.488653+0000","last_fullsized":"2024-09-13T14:41:41.488653+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488397+0000","last_change":"2024-09-13T14:41:41.488308+0000","last_active":"2024-09-13T14:41:41.488397+0000","last_peered":"2024-09-13T14:41:41.488397+0000","last_clean":"2024-09-13T14:41:41.488397+0000","last_became_active":"2024-09-13T14:41:41.487374+0000","last_became_peered":"2024-09-13T14:41:41.487374+0000","last_unstale":"2024-09-13T14:41:41.488397+0000","last_undegraded":"2024-09-13T14:41:41.488397+0000","last_fullsized":"2024-09-13T14:41:41.488397+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:51.242762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":401,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.484511+0000","last_change":"2024-09-13T14:40:04.164078+0000","last_active":"2024-09-13T14:41:40.484511+0000","last_peered":"2024-09-13T14:41:40.484511+0000","last_clean":"2024-09-13T14:41:40.484511+0000","last_became_active":"2024-09-13T14:40:04.163685+0000","last_became_peered":"2024-09-13T14:40:04.163685+0000","last_unstale":"2024-09-13T14:41:40.484511+0000","last_undegraded":"2024-09-13T14:41:40.484511+0000","last_fullsized":"2024-09-13T14:41:40.484511+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:40.476401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":601,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478473+0000","last_change":"2024-09-13T14:41:09.647983+0000","last_active":"2024-09-13T14:41:40.478473+0000","last_peered":"2024-09-13T14:41:40.478473+0000","last_clean":"2024-09-13T14:41:40.478473+0000","last_became_active":"2024-09-13T14:41:09.647486+0000","last_became_peered":"2024-09-13T14:41:09.647486+0000","last_unstale":"2024-09-13T14:41:40.478473+0000","last_undegraded":"2024-09-13T14:41:40.478473+0000","last_fullsized":"2024-09-13T14:41:40.478473+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:42.155770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.489718+0000","last_change":"2024-09-13T14:41:41.489630+0000","last_active":"2024-09-13T14:41:41.489718+0000","last_peered":"2024-09-13T14:41:41.489718+0000","last_clean":"2024-09-13T14:41:41.489718+0000","last_became_active":"2024-09-13T14:41:41.488964+0000","last_became_peered":"2024-09-13T14:41:41.488964+0000","last_unstale":"2024-09-13T14:41:41.489718+0000","last_undegraded":"2024-09-13T14:41:41.489718+0000","last_fullsized":"2024-09-13T14:41:41.489718+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:18.124795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483900+0000","last_change":"2024-09-13T14:28:14.409690+0000","last_active":"2024-09-13T14:41:40.483900+0000","last_peered":"2024-09-13T14:41:40.483900+0000","last_clean":"2024-09-13T14:41:40.483900+0000","last_became_active":"2024-09-13T14:28:14.409379+0000","last_became_peered":"2024-09-13T14:28:14.409379+0000","last_unstale":"2024-09-13T14:41:40.483900+0000","last_undegraded":"2024-09-13T14:41:40.483900+0000","last_fullsized":"2024-09-13T14:41:40.483900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:15.731901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":260,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.484376+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:41:40.484376+0000","last_peered":"2024-09-13T14:41:40.484376+0000","last_clean":"2024-09-13T14:41:40.484376+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:41:40.484376+0000","last_undegraded":"2024-09-13T14:41:40.484376+0000","last_fullsized":"2024-09-13T14:41:40.484376+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496253+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:41:41.496253+0000","last_peered":"2024-09-13T14:41:41.496253+0000","last_clean":"2024-09-13T14:41:41.496253+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:41:41.496253+0000","last_undegraded":"2024-09-13T14:41:41.496253+0000","last_fullsized":"2024-09-13T14:41:41.496253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482067+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:41:41.482067+0000","last_peered":"2024-09-13T14:41:41.482067+0000","last_clean":"2024-09-13T14:41:41.482067+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:41:41.482067+0000","last_undegraded":"2024-09-13T14:41:41.482067+0000","last_fullsized":"2024-09-13T14:41:41.482067+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":627,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492696+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:41:41.492696+0000","last_peered":"2024-09-13T14:41:41.492696+0000","last_clean":"2024-09-13T14:41:41.492696+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:41:41.492696+0000","last_undegraded":"2024-09-13T14:41:41.492696+0000","last_fullsized":"2024-09-13T14:41:41.492696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476974+0000","last_change":"2024-09-13T14:41:09.650296+0000","last_active":"2024-09-13T14:41:41.476974+0000","last_peered":"2024-09-13T14:41:41.476974+0000","last_clean":"2024-09-13T14:41:41.476974+0000","last_became_active":"2024-09-13T14:41:09.650016+0000","last_became_peered":"2024-09-13T14:41:09.650016+0000","last_unstale":"2024-09-13T14:41:41.476974+0000","last_undegraded":"2024-09-13T14:41:41.476974+0000","last_fullsized":"2024-09-13T14:41:41.476974+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:29.994396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479484+0000","last_change":"2024-09-13T14:41:09.655336+0000","last_active":"2024-09-13T14:41:41.479484+0000","last_peered":"2024-09-13T14:41:41.479484+0000","last_clean":"2024-09-13T14:41:41.479484+0000","last_became_active":"2024-09-13T14:41:09.654964+0000","last_became_peered":"2024-09-13T14:41:09.654964+0000","last_unstale":"2024-09-13T14:41:41.479484+0000","last_undegraded":"2024-09-13T14:41:41.479484+0000","last_fullsized":"2024-09-13T14:41:41.479484+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:40.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":724,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477655+0000","last_change":"2024-09-13T14:40:36.879942+0000","last_active":"2024-09-13T14:41:40.477655+0000","last_peered":"2024-09-13T14:41:40.477655+0000","last_clean":"2024-09-13T14:41:40.477655+0000","last_became_active":"2024-09-13T14:40:36.879489+0000","last_became_peered":"2024-09-13T14:40:36.879489+0000","last_unstale":"2024-09-13T14:41:40.477655+0000","last_undegraded":"2024-09-13T14:41:40.477655+0000","last_fullsized":"2024-09-13T14:41:40.477655+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:51.584189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":986,"num_read_kb":2939,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481639+0000","last_change":"2024-09-13T14:40:04.165067+0000","last_active":"2024-09-13T14:41:40.481639+0000","last_peered":"2024-09-13T14:41:40.481639+0000","last_clean":"2024-09-13T14:41:40.481639+0000","last_became_active":"2024-09-13T14:40:04.164682+0000","last_became_peered":"2024-09-13T14:40:04.164682+0000","last_unstale":"2024-09-13T14:41:40.481639+0000","last_undegraded":"2024-09-13T14:41:40.481639+0000","last_fullsized":"2024-09-13T14:41:40.481639+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:56.250533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":65,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.483912+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:41:41.483912+0000","last_peered":"2024-09-13T14:41:41.483912+0000","last_clean":"2024-09-13T14:41:41.483912+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:41:41.483912+0000","last_undegraded":"2024-09-13T14:41:41.483912+0000","last_fullsized":"2024-09-13T14:41:41.483912+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.494473+0000","last_change":"2024-09-13T14:41:41.494473+0000","last_active":"2024-09-13T14:41:41.494473+0000","last_peered":"2024-09-13T14:41:41.494473+0000","last_clean":"2024-09-13T14:41:41.494473+0000","last_became_active":"2024-09-13T14:41:41.493836+0000","last_became_peered":"2024-09-13T14:41:41.493836+0000","last_unstale":"2024-09-13T14:41:41.494473+0000","last_undegraded":"2024-09-13T14:41:41.494473+0000","last_fullsized":"2024-09-13T14:41:41.494473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.352976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":593,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.477346+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:41:41.477346+0000","last_peered":"2024-09-13T14:41:41.477346+0000","last_clean":"2024-09-13T14:41:41.477346+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:41:41.477346+0000","last_undegraded":"2024-09-13T14:41:41.477346+0000","last_fullsized":"2024-09-13T14:41:41.477346+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":43,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476760+0000","last_change":"2024-09-13T14:40:36.878867+0000","last_active":"2024-09-13T14:41:41.476760+0000","last_peered":"2024-09-13T14:41:41.476760+0000","last_clean":"2024-09-13T14:41:41.476760+0000","last_became_active":"2024-09-13T14:40:36.873722+0000","last_became_peered":"2024-09-13T14:40:36.873722+0000","last_unstale":"2024-09-13T14:41:41.476760+0000","last_undegraded":"2024-09-13T14:41:41.476760+0000","last_fullsized":"2024-09-13T14:41:41.476760+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:36.046735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476866+0000","last_change":"2024-09-13T14:41:09.651306+0000","last_active":"2024-09-13T14:41:41.476866+0000","last_peered":"2024-09-13T14:41:41.476866+0000","last_clean":"2024-09-13T14:41:41.476866+0000","last_became_active":"2024-09-13T14:41:09.650597+0000","last_became_peered":"2024-09-13T14:41:09.650597+0000","last_unstale":"2024-09-13T14:41:41.476866+0000","last_undegraded":"2024-09-13T14:41:41.476866+0000","last_fullsized":"2024-09-13T14:41:41.476866+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:02.341051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":421,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.485681+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:41:41.485681+0000","last_peered":"2024-09-13T14:41:41.485681+0000","last_clean":"2024-09-13T14:41:41.485681+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:41:41.485681+0000","last_undegraded":"2024-09-13T14:41:41.485681+0000","last_fullsized":"2024-09-13T14:41:41.485681+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482473+0000","last_change":"2024-09-13T14:40:04.163073+0000","last_active":"2024-09-13T14:41:41.482473+0000","last_peered":"2024-09-13T14:41:41.482473+0000","last_clean":"2024-09-13T14:41:41.482473+0000","last_became_active":"2024-09-13T14:40:04.162461+0000","last_became_peered":"2024-09-13T14:40:04.162461+0000","last_unstale":"2024-09-13T14:41:41.482473+0000","last_undegraded":"2024-09-13T14:41:41.482473+0000","last_fullsized":"2024-09-13T14:41:41.482473+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:28.279878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493597+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:41:41.493597+0000","last_peered":"2024-09-13T14:41:41.493597+0000","last_clean":"2024-09-13T14:41:41.493597+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:41:41.493597+0000","last_undegraded":"2024-09-13T14:41:41.493597+0000","last_fullsized":"2024-09-13T14:41:41.493597+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":477,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476810+0000","last_change":"2024-09-13T14:40:36.875554+0000","last_active":"2024-09-13T14:41:41.476810+0000","last_peered":"2024-09-13T14:41:41.476810+0000","last_clean":"2024-09-13T14:41:41.476810+0000","last_became_active":"2024-09-13T14:40:36.874924+0000","last_became_peered":"2024-09-13T14:40:36.874924+0000","last_unstale":"2024-09-13T14:41:41.476810+0000","last_undegraded":"2024-09-13T14:41:41.476810+0000","last_fullsized":"2024-09-13T14:41:41.476810+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:11.594853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486798+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:41:41.486798+0000","last_peered":"2024-09-13T14:41:41.486798+0000","last_clean":"2024-09-13T14:41:41.486798+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:41:41.486798+0000","last_undegraded":"2024-09-13T14:41:41.486798+0000","last_fullsized":"2024-09-13T14:41:41.486798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481946+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:41:41.481946+0000","last_peered":"2024-09-13T14:41:41.481946+0000","last_clean":"2024-09-13T14:41:41.481946+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:41:41.481946+0000","last_undegraded":"2024-09-13T14:41:41.481946+0000","last_fullsized":"2024-09-13T14:41:41.481946+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":675,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480619+0000","last_change":"2024-09-13T14:41:09.650053+0000","last_active":"2024-09-13T14:41:41.480619+0000","last_peered":"2024-09-13T14:41:41.480619+0000","last_clean":"2024-09-13T14:41:41.480619+0000","last_became_active":"2024-09-13T14:41:09.649530+0000","last_became_peered":"2024-09-13T14:41:09.649530+0000","last_unstale":"2024-09-13T14:41:41.480619+0000","last_undegraded":"2024-09-13T14:41:41.480619+0000","last_fullsized":"2024-09-13T14:41:41.480619+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:39.045795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.480192+0000","last_change":"2024-09-13T14:28:14.320314+0000","last_active":"2024-09-13T14:41:40.480192+0000","last_peered":"2024-09-13T14:41:40.480192+0000","last_clean":"2024-09-13T14:41:40.480192+0000","last_became_active":"2024-09-13T14:28:14.319934+0000","last_became_peered":"2024-09-13T14:28:14.319934+0000","last_unstale":"2024-09-13T14:41:40.480192+0000","last_undegraded":"2024-09-13T14:41:40.480192+0000","last_fullsized":"2024-09-13T14:41:40.480192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:21.514353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480732+0000","last_change":"2024-09-13T14:40:36.870296+0000","last_active":"2024-09-13T14:41:41.480732+0000","last_peered":"2024-09-13T14:41:41.480732+0000","last_clean":"2024-09-13T14:41:41.480732+0000","last_became_active":"2024-09-13T14:40:36.869969+0000","last_became_peered":"2024-09-13T14:40:36.869969+0000","last_unstale":"2024-09-13T14:41:41.480732+0000","last_undegraded":"2024-09-13T14:41:41.480732+0000","last_fullsized":"2024-09-13T14:41:41.480732+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:57.021118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481406+0000","last_change":"2024-09-13T14:28:14.293329+0000","last_active":"2024-09-13T14:41:41.481406+0000","last_peered":"2024-09-13T14:41:41.481406+0000","last_clean":"2024-09-13T14:41:41.481406+0000","last_became_active":"2024-09-13T14:28:14.293075+0000","last_became_peered":"2024-09-13T14:28:14.293075+0000","last_unstale":"2024-09-13T14:41:41.481406+0000","last_undegraded":"2024-09-13T14:41:41.481406+0000","last_fullsized":"2024-09-13T14:41:41.481406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:20.310761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476555+0000","last_change":"2024-09-13T14:41:09.650180+0000","last_active":"2024-09-13T14:41:41.476555+0000","last_peered":"2024-09-13T14:41:41.476555+0000","last_clean":"2024-09-13T14:41:41.476555+0000","last_became_active":"2024-09-13T14:41:09.649908+0000","last_became_peered":"2024-09-13T14:41:09.649908+0000","last_unstale":"2024-09-13T14:41:41.476555+0000","last_undegraded":"2024-09-13T14:41:41.476555+0000","last_fullsized":"2024-09-13T14:41:41.476555+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:08.029267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.489532+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:41:41.489532+0000","last_peered":"2024-09-13T14:41:41.489532+0000","last_clean":"2024-09-13T14:41:41.489532+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:41:41.489532+0000","last_undegraded":"2024-09-13T14:41:41.489532+0000","last_fullsized":"2024-09-13T14:41:41.489532+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477969+0000","last_change":"2024-09-13T14:40:36.879188+0000","last_active":"2024-09-13T14:41:40.477969+0000","last_peered":"2024-09-13T14:41:40.477969+0000","last_clean":"2024-09-13T14:41:40.477969+0000","last_became_active":"2024-09-13T14:40:36.878788+0000","last_became_peered":"2024-09-13T14:40:36.878788+0000","last_unstale":"2024-09-13T14:41:40.477969+0000","last_undegraded":"2024-09-13T14:41:40.477969+0000","last_fullsized":"2024-09-13T14:41:40.477969+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:11.826299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484913+0000","last_change":"2024-09-13T14:41:41.484817+0000","last_active":"2024-09-13T14:41:41.484913+0000","last_peered":"2024-09-13T14:41:41.484913+0000","last_clean":"2024-09-13T14:41:41.484913+0000","last_became_active":"2024-09-13T14:41:41.484505+0000","last_became_peered":"2024-09-13T14:41:41.484505+0000","last_unstale":"2024-09-13T14:41:41.484913+0000","last_undegraded":"2024-09-13T14:41:41.484913+0000","last_fullsized":"2024-09-13T14:41:41.484913+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:49.618334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496242+0000","last_change":"2024-09-13T14:41:41.491895+0000","last_active":"2024-09-13T14:41:41.496242+0000","last_peered":"2024-09-13T14:41:41.496242+0000","last_clean":"2024-09-13T14:41:41.496242+0000","last_became_active":"2024-09-13T14:41:41.491517+0000","last_became_peered":"2024-09-13T14:41:41.491517+0000","last_unstale":"2024-09-13T14:41:41.496242+0000","last_undegraded":"2024-09-13T14:41:41.496242+0000","last_fullsized":"2024-09-13T14:41:41.496242+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:08.848118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490024+0000","last_change":"2024-09-13T14:41:41.489947+0000","last_active":"2024-09-13T14:41:41.490024+0000","last_peered":"2024-09-13T14:41:41.490024+0000","last_clean":"2024-09-13T14:41:41.490024+0000","last_became_active":"2024-09-13T14:41:41.487936+0000","last_became_peered":"2024-09-13T14:41:41.487936+0000","last_unstale":"2024-09-13T14:41:41.490024+0000","last_undegraded":"2024-09-13T14:41:41.490024+0000","last_fullsized":"2024-09-13T14:41:41.490024+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":69,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481742+0000","last_change":"2024-09-13T14:40:36.875850+0000","last_active":"2024-09-13T14:41:40.481742+0000","last_peered":"2024-09-13T14:41:40.481742+0000","last_clean":"2024-09-13T14:41:40.481742+0000","last_became_active":"2024-09-13T14:40:36.875550+0000","last_became_peered":"2024-09-13T14:40:36.875550+0000","last_unstale":"2024-09-13T14:41:40.481742+0000","last_undegraded":"2024-09-13T14:41:40.481742+0000","last_fullsized":"2024-09-13T14:41:40.481742+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:57.875071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492293+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:41:41.492293+0000","last_peered":"2024-09-13T14:41:41.492293+0000","last_clean":"2024-09-13T14:41:41.492293+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:41:41.492293+0000","last_undegraded":"2024-09-13T14:41:41.492293+0000","last_fullsized":"2024-09-13T14:41:41.492293+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490839+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:41:41.490839+0000","last_peered":"2024-09-13T14:41:41.490839+0000","last_clean":"2024-09-13T14:41:41.490839+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:41:41.490839+0000","last_undegraded":"2024-09-13T14:41:41.490839+0000","last_fullsized":"2024-09-13T14:41:41.490839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486110+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:41:41.486110+0000","last_peered":"2024-09-13T14:41:41.486110+0000","last_clean":"2024-09-13T14:41:41.486110+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:41:41.486110+0000","last_undegraded":"2024-09-13T14:41:41.486110+0000","last_fullsized":"2024-09-13T14:41:41.486110+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493149+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:41:41.493149+0000","last_peered":"2024-09-13T14:41:41.493149+0000","last_clean":"2024-09-13T14:41:41.493149+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:41:41.493149+0000","last_undegraded":"2024-09-13T14:41:41.493149+0000","last_fullsized":"2024-09-13T14:41:41.493149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.479047+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:41:40.479047+0000","last_peered":"2024-09-13T14:41:40.479047+0000","last_clean":"2024-09-13T14:41:40.479047+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:41:40.479047+0000","last_undegraded":"2024-09-13T14:41:40.479047+0000","last_fullsized":"2024-09-13T14:41:40.479047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490987+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:41:41.490987+0000","last_peered":"2024-09-13T14:41:41.490987+0000","last_clean":"2024-09-13T14:41:41.490987+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:41:41.490987+0000","last_undegraded":"2024-09-13T14:41:41.490987+0000","last_fullsized":"2024-09-13T14:41:41.490987+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482609+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:41:41.482609+0000","last_peered":"2024-09-13T14:41:41.482609+0000","last_clean":"2024-09-13T14:41:41.482609+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:41:41.482609+0000","last_undegraded":"2024-09-13T14:41:41.482609+0000","last_fullsized":"2024-09-13T14:41:41.482609+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490102+0000","last_change":"2024-09-13T14:41:41.490102+0000","last_active":"2024-09-13T14:41:41.490102+0000","last_peered":"2024-09-13T14:41:41.490102+0000","last_clean":"2024-09-13T14:41:41.490102+0000","last_became_active":"2024-09-13T14:41:41.489822+0000","last_became_peered":"2024-09-13T14:41:41.489822+0000","last_unstale":"2024-09-13T14:41:41.490102+0000","last_undegraded":"2024-09-13T14:41:41.490102+0000","last_fullsized":"2024-09-13T14:41:41.490102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:28.447149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477814+0000","last_change":"2024-09-13T14:40:36.879235+0000","last_active":"2024-09-13T14:41:40.477814+0000","last_peered":"2024-09-13T14:41:40.477814+0000","last_clean":"2024-09-13T14:41:40.477814+0000","last_became_active":"2024-09-13T14:40:36.878874+0000","last_became_peered":"2024-09-13T14:40:36.878874+0000","last_unstale":"2024-09-13T14:41:40.477814+0000","last_undegraded":"2024-09-13T14:41:40.477814+0000","last_fullsized":"2024-09-13T14:41:40.477814+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:07.605097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490555+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:41:41.490555+0000","last_peered":"2024-09-13T14:41:41.490555+0000","last_clean":"2024-09-13T14:41:41.490555+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:41:41.490555+0000","last_undegraded":"2024-09-13T14:41:41.490555+0000","last_fullsized":"2024-09-13T14:41:41.490555+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":57,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478200+0000","last_change":"2024-09-13T14:28:14.297761+0000","last_active":"2024-09-13T14:41:40.478200+0000","last_peered":"2024-09-13T14:41:40.478200+0000","last_clean":"2024-09-13T14:41:40.478200+0000","last_became_active":"2024-09-13T14:28:14.297478+0000","last_became_peered":"2024-09-13T14:28:14.297478+0000","last_unstale":"2024-09-13T14:41:40.478200+0000","last_undegraded":"2024-09-13T14:41:40.478200+0000","last_fullsized":"2024-09-13T14:41:40.478200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:48.450673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488366+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:41:41.488366+0000","last_peered":"2024-09-13T14:41:41.488366+0000","last_clean":"2024-09-13T14:41:41.488366+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:41:41.488366+0000","last_undegraded":"2024-09-13T14:41:41.488366+0000","last_fullsized":"2024-09-13T14:41:41.488366+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486185+0000","last_change":"2024-09-13T14:41:41.486093+0000","last_active":"2024-09-13T14:41:41.486185+0000","last_peered":"2024-09-13T14:41:41.486185+0000","last_clean":"2024-09-13T14:41:41.486185+0000","last_became_active":"2024-09-13T14:41:41.485759+0000","last_became_peered":"2024-09-13T14:41:41.485759+0000","last_unstale":"2024-09-13T14:41:41.486185+0000","last_undegraded":"2024-09-13T14:41:41.486185+0000","last_fullsized":"2024-09-13T14:41:41.486185+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:05.424465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478481+0000","last_change":"2024-09-13T14:40:36.876656+0000","last_active":"2024-09-13T14:41:40.478481+0000","last_peered":"2024-09-13T14:41:40.478481+0000","last_clean":"2024-09-13T14:41:40.478481+0000","last_became_active":"2024-09-13T14:40:36.876361+0000","last_became_peered":"2024-09-13T14:40:36.876361+0000","last_unstale":"2024-09-13T14:41:40.478481+0000","last_undegraded":"2024-09-13T14:41:40.478481+0000","last_fullsized":"2024-09-13T14:41:40.478481+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:51.843071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482257+0000","last_change":"2024-09-13T14:41:09.653285+0000","last_active":"2024-09-13T14:41:41.482257+0000","last_peered":"2024-09-13T14:41:41.482257+0000","last_clean":"2024-09-13T14:41:41.482257+0000","last_became_active":"2024-09-13T14:41:09.652737+0000","last_became_peered":"2024-09-13T14:41:09.652737+0000","last_unstale":"2024-09-13T14:41:41.482257+0000","last_undegraded":"2024-09-13T14:41:41.482257+0000","last_fullsized":"2024-09-13T14:41:41.482257+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:13.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234182,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32892,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712260,"statfs":{"total":95995035648,"available":95961354240,"internally_reserved":0,"allocated":1818624,"data_stored":1215821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495818,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1806336,"data_stored":1215421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757455,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1208320,"data_stored":621210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019093,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1224704,"data_stored":625714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856467981,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32748,"kb_used_data":5272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93712404,"statfs":{"total":95995035648,"available":95961501696,"internally_reserved":0,"allocated":5398528,"data_stored":4815218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":28109988},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631507,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32808,"kb_used_data":5276,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712344,"statfs":{"total":95995035648,"available":95961440256,"internally_reserved":0,"allocated":5402624,"data_stored":4815744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":28172255},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795034,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1176,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1204224,"data_stored":620843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":28172914},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958561,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37548,"kb_used_data":5856,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93707604,"statfs":{"total":95995035648,"available":95956586496,"internally_reserved":0,"allocated":5996544,"data_stored":5406100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":32432615},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:38.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:39.181 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:39.302 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:39 smithi154 ceph-mon[102314]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:39.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:39 smithi154 ceph-mon[107038]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:39.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:39 smithi172 ceph-mon[100696]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:40 smithi154 ceph-mon[102314]: pgmap v276: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:40.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:40 smithi154 ceph-mon[107038]: pgmap v276: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:40 smithi172 ceph-mon[100696]: pgmap v276: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:41.037 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:42:41.038 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:42:41.619 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-13T14:42:40.775087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":262208,"kb_used_data":23496,"kb_used_omap":34,"kb_used_meta":238557,"kb_avail":749699008,"statfs":{"total":767960285184,"available":767691784192,"internally_reserved":0,"allocated":24059904,"data_stored":19336071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34821,"internal_metadata":244283387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002947"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488438+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:41:41.488438+0000","last_peered":"2024-09-13T14:41:41.488438+0000","last_clean":"2024-09-13T14:41:41.488438+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:41:41.488438+0000","last_undegraded":"2024-09-13T14:41:41.488438+0000","last_fullsized":"2024-09-13T14:41:41.488438+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481357+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:41:41.481357+0000","last_peered":"2024-09-13T14:41:41.481357+0000","last_clean":"2024-09-13T14:41:41.481357+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:41:41.481357+0000","last_undegraded":"2024-09-13T14:41:41.481357+0000","last_fullsized":"2024-09-13T14:41:41.481357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493742+0000","last_change":"2024-09-13T14:41:41.493742+0000","last_active":"2024-09-13T14:41:41.493742+0000","last_peered":"2024-09-13T14:41:41.493742+0000","last_clean":"2024-09-13T14:41:41.493742+0000","last_became_active":"2024-09-13T14:41:41.493388+0000","last_became_peered":"2024-09-13T14:41:41.493388+0000","last_unstale":"2024-09-13T14:41:41.493742+0000","last_undegraded":"2024-09-13T14:41:41.493742+0000","last_fullsized":"2024-09-13T14:41:41.493742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:46.304592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496213+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:41:41.496213+0000","last_peered":"2024-09-13T14:41:41.496213+0000","last_clean":"2024-09-13T14:41:41.496213+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:41:41.496213+0000","last_undegraded":"2024-09-13T14:41:41.496213+0000","last_fullsized":"2024-09-13T14:41:41.496213+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490816+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:41:41.490816+0000","last_peered":"2024-09-13T14:41:41.490816+0000","last_clean":"2024-09-13T14:41:41.490816+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:41:41.490816+0000","last_undegraded":"2024-09-13T14:41:41.490816+0000","last_fullsized":"2024-09-13T14:41:41.490816+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":56,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482849+0000","last_change":"2024-09-13T14:28:14.292467+0000","last_active":"2024-09-13T14:41:41.482849+0000","last_peered":"2024-09-13T14:41:41.482849+0000","last_clean":"2024-09-13T14:41:41.482849+0000","last_became_active":"2024-09-13T14:28:14.292208+0000","last_became_peered":"2024-09-13T14:28:14.292208+0000","last_unstale":"2024-09-13T14:41:41.482849+0000","last_undegraded":"2024-09-13T14:41:41.482849+0000","last_fullsized":"2024-09-13T14:41:41.482849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:51.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486054+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:41:41.486054+0000","last_peered":"2024-09-13T14:41:41.486054+0000","last_clean":"2024-09-13T14:41:41.486054+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:41:41.486054+0000","last_undegraded":"2024-09-13T14:41:41.486054+0000","last_fullsized":"2024-09-13T14:41:41.486054+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479873+0000","last_change":"2024-09-13T14:40:04.161607+0000","last_active":"2024-09-13T14:41:41.479873+0000","last_peered":"2024-09-13T14:41:41.479873+0000","last_clean":"2024-09-13T14:41:41.479873+0000","last_became_active":"2024-09-13T14:40:04.161226+0000","last_became_peered":"2024-09-13T14:40:04.161226+0000","last_unstale":"2024-09-13T14:41:41.479873+0000","last_undegraded":"2024-09-13T14:41:41.479873+0000","last_fullsized":"2024-09-13T14:41:41.479873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:21.273027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484875+0000","last_change":"2024-09-13T14:40:04.164112+0000","last_active":"2024-09-13T14:41:41.484875+0000","last_peered":"2024-09-13T14:41:41.484875+0000","last_clean":"2024-09-13T14:41:41.484875+0000","last_became_active":"2024-09-13T14:40:04.163753+0000","last_became_peered":"2024-09-13T14:40:04.163753+0000","last_unstale":"2024-09-13T14:41:41.484875+0000","last_undegraded":"2024-09-13T14:41:41.484875+0000","last_fullsized":"2024-09-13T14:41:41.484875+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.685538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":55,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477581+0000","last_change":"2024-09-13T14:28:14.409515+0000","last_active":"2024-09-13T14:41:40.477581+0000","last_peered":"2024-09-13T14:41:40.477581+0000","last_clean":"2024-09-13T14:41:40.477581+0000","last_became_active":"2024-09-13T14:28:14.409202+0000","last_became_peered":"2024-09-13T14:28:14.409202+0000","last_unstale":"2024-09-13T14:41:40.477581+0000","last_undegraded":"2024-09-13T14:41:40.477581+0000","last_fullsized":"2024-09-13T14:41:40.477581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:24.530144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484456+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:41:41.484456+0000","last_peered":"2024-09-13T14:41:41.484456+0000","last_clean":"2024-09-13T14:41:41.484456+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:41:41.484456+0000","last_undegraded":"2024-09-13T14:41:41.484456+0000","last_fullsized":"2024-09-13T14:41:41.484456+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493612+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:41:41.493612+0000","last_peered":"2024-09-13T14:41:41.493612+0000","last_clean":"2024-09-13T14:41:41.493612+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:41:41.493612+0000","last_undegraded":"2024-09-13T14:41:41.493612+0000","last_fullsized":"2024-09-13T14:41:41.493612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484512+0000","last_change":"2024-09-13T14:28:14.289966+0000","last_active":"2024-09-13T14:41:41.484512+0000","last_peered":"2024-09-13T14:41:41.484512+0000","last_clean":"2024-09-13T14:41:41.484512+0000","last_became_active":"2024-09-13T14:28:14.289682+0000","last_became_peered":"2024-09-13T14:28:14.289682+0000","last_unstale":"2024-09-13T14:41:41.484512+0000","last_undegraded":"2024-09-13T14:41:41.484512+0000","last_fullsized":"2024-09-13T14:41:41.484512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:37.096298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":69,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483095+0000","last_change":"2024-09-13T14:40:36.880233+0000","last_active":"2024-09-13T14:41:40.483095+0000","last_peered":"2024-09-13T14:41:40.483095+0000","last_clean":"2024-09-13T14:41:40.483095+0000","last_became_active":"2024-09-13T14:40:36.879886+0000","last_became_peered":"2024-09-13T14:40:36.879886+0000","last_unstale":"2024-09-13T14:41:40.483095+0000","last_undegraded":"2024-09-13T14:41:40.483095+0000","last_fullsized":"2024-09-13T14:41:40.483095+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:49.693282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491428+0000","last_change":"2024-09-13T14:41:41.490296+0000","last_active":"2024-09-13T14:41:41.491428+0000","last_peered":"2024-09-13T14:41:41.491428+0000","last_clean":"2024-09-13T14:41:41.491428+0000","last_became_active":"2024-09-13T14:41:41.489937+0000","last_became_peered":"2024-09-13T14:41:41.489937+0000","last_unstale":"2024-09-13T14:41:41.491428+0000","last_undegraded":"2024-09-13T14:41:41.491428+0000","last_fullsized":"2024-09-13T14:41:41.491428+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:30.748843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482550+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:41:41.482550+0000","last_peered":"2024-09-13T14:41:41.482550+0000","last_clean":"2024-09-13T14:41:41.482550+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:41:41.482550+0000","last_undegraded":"2024-09-13T14:41:41.482550+0000","last_fullsized":"2024-09-13T14:41:41.482550+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491300+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:41:41.491300+0000","last_peered":"2024-09-13T14:41:41.491300+0000","last_clean":"2024-09-13T14:41:41.491300+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:41:41.491300+0000","last_undegraded":"2024-09-13T14:41:41.491300+0000","last_fullsized":"2024-09-13T14:41:41.491300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491488+0000","last_change":"2024-09-13T14:41:41.491050+0000","last_active":"2024-09-13T14:41:41.491488+0000","last_peered":"2024-09-13T14:41:41.491488+0000","last_clean":"2024-09-13T14:41:41.491488+0000","last_became_active":"2024-09-13T14:41:41.490645+0000","last_became_peered":"2024-09-13T14:41:41.490645+0000","last_unstale":"2024-09-13T14:41:41.491488+0000","last_undegraded":"2024-09-13T14:41:41.491488+0000","last_fullsized":"2024-09-13T14:41:41.491488+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:01.779411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483326+0000","last_change":"2024-09-13T14:40:36.879533+0000","last_active":"2024-09-13T14:41:40.483326+0000","last_peered":"2024-09-13T14:41:40.483326+0000","last_clean":"2024-09-13T14:41:40.483326+0000","last_became_active":"2024-09-13T14:40:36.879212+0000","last_became_peered":"2024-09-13T14:40:36.879212+0000","last_unstale":"2024-09-13T14:41:40.483326+0000","last_undegraded":"2024-09-13T14:41:40.483326+0000","last_fullsized":"2024-09-13T14:41:40.483326+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:03.087074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481810+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:41:41.481810+0000","last_peered":"2024-09-13T14:41:41.481810+0000","last_clean":"2024-09-13T14:41:41.481810+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:41:41.481810+0000","last_undegraded":"2024-09-13T14:41:41.481810+0000","last_fullsized":"2024-09-13T14:41:41.481810+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481078+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:41:40.481078+0000","last_peered":"2024-09-13T14:41:40.481078+0000","last_clean":"2024-09-13T14:41:40.481078+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:41:40.481078+0000","last_undegraded":"2024-09-13T14:41:40.481078+0000","last_fullsized":"2024-09-13T14:41:40.481078+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481692+0000","last_change":"2024-09-13T14:40:36.875852+0000","last_active":"2024-09-13T14:41:41.481692+0000","last_peered":"2024-09-13T14:41:41.481692+0000","last_clean":"2024-09-13T14:41:41.481692+0000","last_became_active":"2024-09-13T14:40:36.875486+0000","last_became_peered":"2024-09-13T14:40:36.875486+0000","last_unstale":"2024-09-13T14:41:41.481692+0000","last_undegraded":"2024-09-13T14:41:41.481692+0000","last_fullsized":"2024-09-13T14:41:41.481692+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:18.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488653+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:41:41.488653+0000","last_peered":"2024-09-13T14:41:41.488653+0000","last_clean":"2024-09-13T14:41:41.488653+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:41:41.488653+0000","last_undegraded":"2024-09-13T14:41:41.488653+0000","last_fullsized":"2024-09-13T14:41:41.488653+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488397+0000","last_change":"2024-09-13T14:41:41.488308+0000","last_active":"2024-09-13T14:41:41.488397+0000","last_peered":"2024-09-13T14:41:41.488397+0000","last_clean":"2024-09-13T14:41:41.488397+0000","last_became_active":"2024-09-13T14:41:41.487374+0000","last_became_peered":"2024-09-13T14:41:41.487374+0000","last_unstale":"2024-09-13T14:41:41.488397+0000","last_undegraded":"2024-09-13T14:41:41.488397+0000","last_fullsized":"2024-09-13T14:41:41.488397+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:51.242762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":402,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.483583+0000","last_change":"2024-09-13T14:40:04.164078+0000","last_active":"2024-09-13T14:41:41.483583+0000","last_peered":"2024-09-13T14:41:41.483583+0000","last_clean":"2024-09-13T14:41:41.483583+0000","last_became_active":"2024-09-13T14:40:04.163685+0000","last_became_peered":"2024-09-13T14:40:04.163685+0000","last_unstale":"2024-09-13T14:41:41.483583+0000","last_undegraded":"2024-09-13T14:41:41.483583+0000","last_fullsized":"2024-09-13T14:41:41.483583+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:40.476401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":601,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478473+0000","last_change":"2024-09-13T14:41:09.647983+0000","last_active":"2024-09-13T14:41:40.478473+0000","last_peered":"2024-09-13T14:41:40.478473+0000","last_clean":"2024-09-13T14:41:40.478473+0000","last_became_active":"2024-09-13T14:41:09.647486+0000","last_became_peered":"2024-09-13T14:41:09.647486+0000","last_unstale":"2024-09-13T14:41:40.478473+0000","last_undegraded":"2024-09-13T14:41:40.478473+0000","last_fullsized":"2024-09-13T14:41:40.478473+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:42.155770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.489718+0000","last_change":"2024-09-13T14:41:41.489630+0000","last_active":"2024-09-13T14:41:41.489718+0000","last_peered":"2024-09-13T14:41:41.489718+0000","last_clean":"2024-09-13T14:41:41.489718+0000","last_became_active":"2024-09-13T14:41:41.488964+0000","last_became_peered":"2024-09-13T14:41:41.488964+0000","last_unstale":"2024-09-13T14:41:41.489718+0000","last_undegraded":"2024-09-13T14:41:41.489718+0000","last_fullsized":"2024-09-13T14:41:41.489718+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:18.124795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483900+0000","last_change":"2024-09-13T14:28:14.409690+0000","last_active":"2024-09-13T14:41:40.483900+0000","last_peered":"2024-09-13T14:41:40.483900+0000","last_clean":"2024-09-13T14:41:40.483900+0000","last_became_active":"2024-09-13T14:28:14.409379+0000","last_became_peered":"2024-09-13T14:28:14.409379+0000","last_unstale":"2024-09-13T14:41:40.483900+0000","last_undegraded":"2024-09-13T14:41:40.483900+0000","last_fullsized":"2024-09-13T14:41:40.483900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:15.731901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":260,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.484376+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:41:40.484376+0000","last_peered":"2024-09-13T14:41:40.484376+0000","last_clean":"2024-09-13T14:41:40.484376+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:41:40.484376+0000","last_undegraded":"2024-09-13T14:41:40.484376+0000","last_fullsized":"2024-09-13T14:41:40.484376+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496253+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:41:41.496253+0000","last_peered":"2024-09-13T14:41:41.496253+0000","last_clean":"2024-09-13T14:41:41.496253+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:41:41.496253+0000","last_undegraded":"2024-09-13T14:41:41.496253+0000","last_fullsized":"2024-09-13T14:41:41.496253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482067+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:41:41.482067+0000","last_peered":"2024-09-13T14:41:41.482067+0000","last_clean":"2024-09-13T14:41:41.482067+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:41:41.482067+0000","last_undegraded":"2024-09-13T14:41:41.482067+0000","last_fullsized":"2024-09-13T14:41:41.482067+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":627,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492696+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:41:41.492696+0000","last_peered":"2024-09-13T14:41:41.492696+0000","last_clean":"2024-09-13T14:41:41.492696+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:41:41.492696+0000","last_undegraded":"2024-09-13T14:41:41.492696+0000","last_fullsized":"2024-09-13T14:41:41.492696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476974+0000","last_change":"2024-09-13T14:41:09.650296+0000","last_active":"2024-09-13T14:41:41.476974+0000","last_peered":"2024-09-13T14:41:41.476974+0000","last_clean":"2024-09-13T14:41:41.476974+0000","last_became_active":"2024-09-13T14:41:09.650016+0000","last_became_peered":"2024-09-13T14:41:09.650016+0000","last_unstale":"2024-09-13T14:41:41.476974+0000","last_undegraded":"2024-09-13T14:41:41.476974+0000","last_fullsized":"2024-09-13T14:41:41.476974+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:29.994396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479484+0000","last_change":"2024-09-13T14:41:09.655336+0000","last_active":"2024-09-13T14:41:41.479484+0000","last_peered":"2024-09-13T14:41:41.479484+0000","last_clean":"2024-09-13T14:41:41.479484+0000","last_became_active":"2024-09-13T14:41:09.654964+0000","last_became_peered":"2024-09-13T14:41:09.654964+0000","last_unstale":"2024-09-13T14:41:41.479484+0000","last_undegraded":"2024-09-13T14:41:41.479484+0000","last_fullsized":"2024-09-13T14:41:41.479484+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:40.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":724,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477655+0000","last_change":"2024-09-13T14:40:36.879942+0000","last_active":"2024-09-13T14:41:40.477655+0000","last_peered":"2024-09-13T14:41:40.477655+0000","last_clean":"2024-09-13T14:41:40.477655+0000","last_became_active":"2024-09-13T14:40:36.879489+0000","last_became_peered":"2024-09-13T14:40:36.879489+0000","last_unstale":"2024-09-13T14:41:40.477655+0000","last_undegraded":"2024-09-13T14:41:40.477655+0000","last_fullsized":"2024-09-13T14:41:40.477655+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:51.584189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":986,"num_read_kb":2939,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481639+0000","last_change":"2024-09-13T14:40:04.165067+0000","last_active":"2024-09-13T14:41:40.481639+0000","last_peered":"2024-09-13T14:41:40.481639+0000","last_clean":"2024-09-13T14:41:40.481639+0000","last_became_active":"2024-09-13T14:40:04.164682+0000","last_became_peered":"2024-09-13T14:40:04.164682+0000","last_unstale":"2024-09-13T14:41:40.481639+0000","last_undegraded":"2024-09-13T14:41:40.481639+0000","last_fullsized":"2024-09-13T14:41:40.481639+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:56.250533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484161+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:41:41.484161+0000","last_peered":"2024-09-13T14:41:41.484161+0000","last_clean":"2024-09-13T14:41:41.484161+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:41:41.484161+0000","last_undegraded":"2024-09-13T14:41:41.484161+0000","last_fullsized":"2024-09-13T14:41:41.484161+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.494473+0000","last_change":"2024-09-13T14:41:41.494473+0000","last_active":"2024-09-13T14:41:41.494473+0000","last_peered":"2024-09-13T14:41:41.494473+0000","last_clean":"2024-09-13T14:41:41.494473+0000","last_became_active":"2024-09-13T14:41:41.493836+0000","last_became_peered":"2024-09-13T14:41:41.493836+0000","last_unstale":"2024-09-13T14:41:41.494473+0000","last_undegraded":"2024-09-13T14:41:41.494473+0000","last_fullsized":"2024-09-13T14:41:41.494473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.352976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":594,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.477391+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:41:41.477391+0000","last_peered":"2024-09-13T14:41:41.477391+0000","last_clean":"2024-09-13T14:41:41.477391+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:41:41.477391+0000","last_undegraded":"2024-09-13T14:41:41.477391+0000","last_fullsized":"2024-09-13T14:41:41.477391+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476798+0000","last_change":"2024-09-13T14:40:36.878867+0000","last_active":"2024-09-13T14:41:41.476798+0000","last_peered":"2024-09-13T14:41:41.476798+0000","last_clean":"2024-09-13T14:41:41.476798+0000","last_became_active":"2024-09-13T14:40:36.873722+0000","last_became_peered":"2024-09-13T14:40:36.873722+0000","last_unstale":"2024-09-13T14:41:41.476798+0000","last_undegraded":"2024-09-13T14:41:41.476798+0000","last_fullsized":"2024-09-13T14:41:41.476798+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:36.046735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.477716+0000","last_change":"2024-09-13T14:41:09.651306+0000","last_active":"2024-09-13T14:41:41.477716+0000","last_peered":"2024-09-13T14:41:41.477716+0000","last_clean":"2024-09-13T14:41:41.477716+0000","last_became_active":"2024-09-13T14:41:09.650597+0000","last_became_peered":"2024-09-13T14:41:09.650597+0000","last_unstale":"2024-09-13T14:41:41.477716+0000","last_undegraded":"2024-09-13T14:41:41.477716+0000","last_fullsized":"2024-09-13T14:41:41.477716+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:02.341051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":421,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.485681+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:41:41.485681+0000","last_peered":"2024-09-13T14:41:41.485681+0000","last_clean":"2024-09-13T14:41:41.485681+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:41:41.485681+0000","last_undegraded":"2024-09-13T14:41:41.485681+0000","last_fullsized":"2024-09-13T14:41:41.485681+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482473+0000","last_change":"2024-09-13T14:40:04.163073+0000","last_active":"2024-09-13T14:41:41.482473+0000","last_peered":"2024-09-13T14:41:41.482473+0000","last_clean":"2024-09-13T14:41:41.482473+0000","last_became_active":"2024-09-13T14:40:04.162461+0000","last_became_peered":"2024-09-13T14:40:04.162461+0000","last_unstale":"2024-09-13T14:41:41.482473+0000","last_undegraded":"2024-09-13T14:41:41.482473+0000","last_fullsized":"2024-09-13T14:41:41.482473+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:28.279878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493597+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:41:41.493597+0000","last_peered":"2024-09-13T14:41:41.493597+0000","last_clean":"2024-09-13T14:41:41.493597+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:41:41.493597+0000","last_undegraded":"2024-09-13T14:41:41.493597+0000","last_fullsized":"2024-09-13T14:41:41.493597+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":477,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476810+0000","last_change":"2024-09-13T14:40:36.875554+0000","last_active":"2024-09-13T14:41:41.476810+0000","last_peered":"2024-09-13T14:41:41.476810+0000","last_clean":"2024-09-13T14:41:41.476810+0000","last_became_active":"2024-09-13T14:40:36.874924+0000","last_became_peered":"2024-09-13T14:40:36.874924+0000","last_unstale":"2024-09-13T14:41:41.476810+0000","last_undegraded":"2024-09-13T14:41:41.476810+0000","last_fullsized":"2024-09-13T14:41:41.476810+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:11.594853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486798+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:41:41.486798+0000","last_peered":"2024-09-13T14:41:41.486798+0000","last_clean":"2024-09-13T14:41:41.486798+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:41:41.486798+0000","last_undegraded":"2024-09-13T14:41:41.486798+0000","last_fullsized":"2024-09-13T14:41:41.486798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481946+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:41:41.481946+0000","last_peered":"2024-09-13T14:41:41.481946+0000","last_clean":"2024-09-13T14:41:41.481946+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:41:41.481946+0000","last_undegraded":"2024-09-13T14:41:41.481946+0000","last_fullsized":"2024-09-13T14:41:41.481946+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":675,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480619+0000","last_change":"2024-09-13T14:41:09.650053+0000","last_active":"2024-09-13T14:41:41.480619+0000","last_peered":"2024-09-13T14:41:41.480619+0000","last_clean":"2024-09-13T14:41:41.480619+0000","last_became_active":"2024-09-13T14:41:09.649530+0000","last_became_peered":"2024-09-13T14:41:09.649530+0000","last_unstale":"2024-09-13T14:41:41.480619+0000","last_undegraded":"2024-09-13T14:41:41.480619+0000","last_fullsized":"2024-09-13T14:41:41.480619+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:39.045795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.480192+0000","last_change":"2024-09-13T14:28:14.320314+0000","last_active":"2024-09-13T14:41:40.480192+0000","last_peered":"2024-09-13T14:41:40.480192+0000","last_clean":"2024-09-13T14:41:40.480192+0000","last_became_active":"2024-09-13T14:28:14.319934+0000","last_became_peered":"2024-09-13T14:28:14.319934+0000","last_unstale":"2024-09-13T14:41:40.480192+0000","last_undegraded":"2024-09-13T14:41:40.480192+0000","last_fullsized":"2024-09-13T14:41:40.480192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:21.514353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480732+0000","last_change":"2024-09-13T14:40:36.870296+0000","last_active":"2024-09-13T14:41:41.480732+0000","last_peered":"2024-09-13T14:41:41.480732+0000","last_clean":"2024-09-13T14:41:41.480732+0000","last_became_active":"2024-09-13T14:40:36.869969+0000","last_became_peered":"2024-09-13T14:40:36.869969+0000","last_unstale":"2024-09-13T14:41:41.480732+0000","last_undegraded":"2024-09-13T14:41:41.480732+0000","last_fullsized":"2024-09-13T14:41:41.480732+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:57.021118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481406+0000","last_change":"2024-09-13T14:28:14.293329+0000","last_active":"2024-09-13T14:41:41.481406+0000","last_peered":"2024-09-13T14:41:41.481406+0000","last_clean":"2024-09-13T14:41:41.481406+0000","last_became_active":"2024-09-13T14:28:14.293075+0000","last_became_peered":"2024-09-13T14:28:14.293075+0000","last_unstale":"2024-09-13T14:41:41.481406+0000","last_undegraded":"2024-09-13T14:41:41.481406+0000","last_fullsized":"2024-09-13T14:41:41.481406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:20.310761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476555+0000","last_change":"2024-09-13T14:41:09.650180+0000","last_active":"2024-09-13T14:41:41.476555+0000","last_peered":"2024-09-13T14:41:41.476555+0000","last_clean":"2024-09-13T14:41:41.476555+0000","last_became_active":"2024-09-13T14:41:09.649908+0000","last_became_peered":"2024-09-13T14:41:09.649908+0000","last_unstale":"2024-09-13T14:41:41.476555+0000","last_undegraded":"2024-09-13T14:41:41.476555+0000","last_fullsized":"2024-09-13T14:41:41.476555+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:08.029267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.489532+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:41:41.489532+0000","last_peered":"2024-09-13T14:41:41.489532+0000","last_clean":"2024-09-13T14:41:41.489532+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:41:41.489532+0000","last_undegraded":"2024-09-13T14:41:41.489532+0000","last_fullsized":"2024-09-13T14:41:41.489532+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477969+0000","last_change":"2024-09-13T14:40:36.879188+0000","last_active":"2024-09-13T14:41:40.477969+0000","last_peered":"2024-09-13T14:41:40.477969+0000","last_clean":"2024-09-13T14:41:40.477969+0000","last_became_active":"2024-09-13T14:40:36.878788+0000","last_became_peered":"2024-09-13T14:40:36.878788+0000","last_unstale":"2024-09-13T14:41:40.477969+0000","last_undegraded":"2024-09-13T14:41:40.477969+0000","last_fullsized":"2024-09-13T14:41:40.477969+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:11.826299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484913+0000","last_change":"2024-09-13T14:41:41.484817+0000","last_active":"2024-09-13T14:41:41.484913+0000","last_peered":"2024-09-13T14:41:41.484913+0000","last_clean":"2024-09-13T14:41:41.484913+0000","last_became_active":"2024-09-13T14:41:41.484505+0000","last_became_peered":"2024-09-13T14:41:41.484505+0000","last_unstale":"2024-09-13T14:41:41.484913+0000","last_undegraded":"2024-09-13T14:41:41.484913+0000","last_fullsized":"2024-09-13T14:41:41.484913+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:49.618334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496242+0000","last_change":"2024-09-13T14:41:41.491895+0000","last_active":"2024-09-13T14:41:41.496242+0000","last_peered":"2024-09-13T14:41:41.496242+0000","last_clean":"2024-09-13T14:41:41.496242+0000","last_became_active":"2024-09-13T14:41:41.491517+0000","last_became_peered":"2024-09-13T14:41:41.491517+0000","last_unstale":"2024-09-13T14:41:41.496242+0000","last_undegraded":"2024-09-13T14:41:41.496242+0000","last_fullsized":"2024-09-13T14:41:41.496242+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:08.848118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490024+0000","last_change":"2024-09-13T14:41:41.489947+0000","last_active":"2024-09-13T14:41:41.490024+0000","last_peered":"2024-09-13T14:41:41.490024+0000","last_clean":"2024-09-13T14:41:41.490024+0000","last_became_active":"2024-09-13T14:41:41.487936+0000","last_became_peered":"2024-09-13T14:41:41.487936+0000","last_unstale":"2024-09-13T14:41:41.490024+0000","last_undegraded":"2024-09-13T14:41:41.490024+0000","last_fullsized":"2024-09-13T14:41:41.490024+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":69,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481742+0000","last_change":"2024-09-13T14:40:36.875850+0000","last_active":"2024-09-13T14:41:40.481742+0000","last_peered":"2024-09-13T14:41:40.481742+0000","last_clean":"2024-09-13T14:41:40.481742+0000","last_became_active":"2024-09-13T14:40:36.875550+0000","last_became_peered":"2024-09-13T14:40:36.875550+0000","last_unstale":"2024-09-13T14:41:40.481742+0000","last_undegraded":"2024-09-13T14:41:40.481742+0000","last_fullsized":"2024-09-13T14:41:40.481742+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:57.875071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492293+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:41:41.492293+0000","last_peered":"2024-09-13T14:41:41.492293+0000","last_clean":"2024-09-13T14:41:41.492293+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:41:41.492293+0000","last_undegraded":"2024-09-13T14:41:41.492293+0000","last_fullsized":"2024-09-13T14:41:41.492293+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490839+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:41:41.490839+0000","last_peered":"2024-09-13T14:41:41.490839+0000","last_clean":"2024-09-13T14:41:41.490839+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:41:41.490839+0000","last_undegraded":"2024-09-13T14:41:41.490839+0000","last_fullsized":"2024-09-13T14:41:41.490839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486110+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:41:41.486110+0000","last_peered":"2024-09-13T14:41:41.486110+0000","last_clean":"2024-09-13T14:41:41.486110+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:41:41.486110+0000","last_undegraded":"2024-09-13T14:41:41.486110+0000","last_fullsized":"2024-09-13T14:41:41.486110+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493149+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:41:41.493149+0000","last_peered":"2024-09-13T14:41:41.493149+0000","last_clean":"2024-09-13T14:41:41.493149+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:41:41.493149+0000","last_undegraded":"2024-09-13T14:41:41.493149+0000","last_fullsized":"2024-09-13T14:41:41.493149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.479047+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:41:40.479047+0000","last_peered":"2024-09-13T14:41:40.479047+0000","last_clean":"2024-09-13T14:41:40.479047+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:41:40.479047+0000","last_undegraded":"2024-09-13T14:41:40.479047+0000","last_fullsized":"2024-09-13T14:41:40.479047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490987+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:41:41.490987+0000","last_peered":"2024-09-13T14:41:41.490987+0000","last_clean":"2024-09-13T14:41:41.490987+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:41:41.490987+0000","last_undegraded":"2024-09-13T14:41:41.490987+0000","last_fullsized":"2024-09-13T14:41:41.490987+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482609+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:41:41.482609+0000","last_peered":"2024-09-13T14:41:41.482609+0000","last_clean":"2024-09-13T14:41:41.482609+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:41:41.482609+0000","last_undegraded":"2024-09-13T14:41:41.482609+0000","last_fullsized":"2024-09-13T14:41:41.482609+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490102+0000","last_change":"2024-09-13T14:41:41.490102+0000","last_active":"2024-09-13T14:41:41.490102+0000","last_peered":"2024-09-13T14:41:41.490102+0000","last_clean":"2024-09-13T14:41:41.490102+0000","last_became_active":"2024-09-13T14:41:41.489822+0000","last_became_peered":"2024-09-13T14:41:41.489822+0000","last_unstale":"2024-09-13T14:41:41.490102+0000","last_undegraded":"2024-09-13T14:41:41.490102+0000","last_fullsized":"2024-09-13T14:41:41.490102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:28.447149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477814+0000","last_change":"2024-09-13T14:40:36.879235+0000","last_active":"2024-09-13T14:41:40.477814+0000","last_peered":"2024-09-13T14:41:40.477814+0000","last_clean":"2024-09-13T14:41:40.477814+0000","last_became_active":"2024-09-13T14:40:36.878874+0000","last_became_peered":"2024-09-13T14:40:36.878874+0000","last_unstale":"2024-09-13T14:41:40.477814+0000","last_undegraded":"2024-09-13T14:41:40.477814+0000","last_fullsized":"2024-09-13T14:41:40.477814+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:07.605097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490555+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:41:41.490555+0000","last_peered":"2024-09-13T14:41:41.490555+0000","last_clean":"2024-09-13T14:41:41.490555+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:41:41.490555+0000","last_undegraded":"2024-09-13T14:41:41.490555+0000","last_fullsized":"2024-09-13T14:41:41.490555+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":57,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478200+0000","last_change":"2024-09-13T14:28:14.297761+0000","last_active":"2024-09-13T14:41:40.478200+0000","last_peered":"2024-09-13T14:41:40.478200+0000","last_clean":"2024-09-13T14:41:40.478200+0000","last_became_active":"2024-09-13T14:28:14.297478+0000","last_became_peered":"2024-09-13T14:28:14.297478+0000","last_unstale":"2024-09-13T14:41:40.478200+0000","last_undegraded":"2024-09-13T14:41:40.478200+0000","last_fullsized":"2024-09-13T14:41:40.478200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:48.450673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488366+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:41:41.488366+0000","last_peered":"2024-09-13T14:41:41.488366+0000","last_clean":"2024-09-13T14:41:41.488366+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:41:41.488366+0000","last_undegraded":"2024-09-13T14:41:41.488366+0000","last_fullsized":"2024-09-13T14:41:41.488366+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486185+0000","last_change":"2024-09-13T14:41:41.486093+0000","last_active":"2024-09-13T14:41:41.486185+0000","last_peered":"2024-09-13T14:41:41.486185+0000","last_clean":"2024-09-13T14:41:41.486185+0000","last_became_active":"2024-09-13T14:41:41.485759+0000","last_became_peered":"2024-09-13T14:41:41.485759+0000","last_unstale":"2024-09-13T14:41:41.486185+0000","last_undegraded":"2024-09-13T14:41:41.486185+0000","last_fullsized":"2024-09-13T14:41:41.486185+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:05.424465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478481+0000","last_change":"2024-09-13T14:40:36.876656+0000","last_active":"2024-09-13T14:41:40.478481+0000","last_peered":"2024-09-13T14:41:40.478481+0000","last_clean":"2024-09-13T14:41:40.478481+0000","last_became_active":"2024-09-13T14:40:36.876361+0000","last_became_peered":"2024-09-13T14:40:36.876361+0000","last_unstale":"2024-09-13T14:41:40.478481+0000","last_undegraded":"2024-09-13T14:41:40.478481+0000","last_fullsized":"2024-09-13T14:41:40.478481+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:51.843071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482257+0000","last_change":"2024-09-13T14:41:09.653285+0000","last_active":"2024-09-13T14:41:41.482257+0000","last_peered":"2024-09-13T14:41:41.482257+0000","last_clean":"2024-09-13T14:41:41.482257+0000","last_became_active":"2024-09-13T14:41:09.652737+0000","last_became_peered":"2024-09-13T14:41:09.652737+0000","last_unstale":"2024-09-13T14:41:41.482257+0000","last_undegraded":"2024-09-13T14:41:41.482257+0000","last_fullsized":"2024-09-13T14:41:41.482257+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:13.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234183,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32892,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712260,"statfs":{"total":95995035648,"available":95961354240,"internally_reserved":0,"allocated":1818624,"data_stored":1215821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495819,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1806336,"data_stored":1215421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757456,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1208320,"data_stored":621210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019094,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1224704,"data_stored":625714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856467982,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32748,"kb_used_data":5272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93712404,"statfs":{"total":95995035648,"available":95961501696,"internally_reserved":0,"allocated":5398528,"data_stored":4815218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":28109988},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631508,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32808,"kb_used_data":5276,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712344,"statfs":{"total":95995035648,"available":95961440256,"internally_reserved":0,"allocated":5402624,"data_stored":4815744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":28172255},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795035,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1176,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1204224,"data_stored":620843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":28172914},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958561,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37548,"kb_used_data":5856,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93707604,"statfs":{"total":95995035648,"available":95956586496,"internally_reserved":0,"allocated":5996544,"data_stored":5406100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":32432615},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:41.623 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:42:41.623 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T14:42:41.624 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T14:42:41.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T14:42:41.830 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:42.148 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:42:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:42:41] "GET /metrics HTTP/1.1" 200 33943 "" "Prometheus/2.51.0" 2024-09-13T14:42:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:42 smithi154 ceph-mon[102314]: pgmap v277: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:42 smithi154 ceph-mon[102314]: from='client.44468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:42 smithi154 ceph-mon[107038]: pgmap v277: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:42 smithi154 ceph-mon[107038]: from='client.44468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:42 smithi172 ceph-mon[100696]: pgmap v277: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:42 smithi172 ceph-mon[100696]: from='client.44468 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:42:43.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:43.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:43.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:43.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:43.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:43.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:43.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:43.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:42:43.572 INFO:teuthology.orchestra.run.smithi154.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T14:42:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:44 smithi154 ceph-mon[102314]: pgmap v278: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3017927388' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:42:44.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:44 smithi154 ceph-mon[107038]: pgmap v278: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:44.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3017927388' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:42:44.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:44 smithi172 ceph-mon[100696]: pgmap v278: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:44.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3017927388' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-13T14:42:44.682 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T14:42:44.682 INFO:teuthology.run_tasks:Running task print... 2024-09-13T14:42:44.691 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-13T14:42:44.691 INFO:teuthology.run_tasks:Running task parallel... 2024-09-13T14:42:44.733 INFO:teuthology.task.parallel:starting parallel... 2024-09-13T14:42:44.733 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-13T14:42:44.733 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-13T14:42:44.735 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:42:44.735 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_AVAILABILITY'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T14:42:44.736 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T14:42:44.736 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T14:42:44.736 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T14:42:44.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:44.976 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:46 smithi154 ceph-mon[102314]: pgmap v279: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:46.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:46.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:46 smithi154 ceph-mon[107038]: pgmap v279: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:46.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:46.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:46.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:46 smithi172 ceph-mon[100696]: pgmap v279: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:47.290 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:42:47.291 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:42:48.015 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-13T14:42:46.776426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":262208,"kb_used_data":23496,"kb_used_omap":34,"kb_used_meta":238557,"kb_avail":749699008,"statfs":{"total":767960285184,"available":767691784192,"internally_reserved":0,"allocated":24059904,"data_stored":19336071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34821,"internal_metadata":244283387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002757"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488438+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:41:41.488438+0000","last_peered":"2024-09-13T14:41:41.488438+0000","last_clean":"2024-09-13T14:41:41.488438+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:41:41.488438+0000","last_undegraded":"2024-09-13T14:41:41.488438+0000","last_fullsized":"2024-09-13T14:41:41.488438+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481357+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:41:41.481357+0000","last_peered":"2024-09-13T14:41:41.481357+0000","last_clean":"2024-09-13T14:41:41.481357+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:41:41.481357+0000","last_undegraded":"2024-09-13T14:41:41.481357+0000","last_fullsized":"2024-09-13T14:41:41.481357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.494559+0000","last_change":"2024-09-13T14:41:41.493742+0000","last_active":"2024-09-13T14:41:41.494559+0000","last_peered":"2024-09-13T14:41:41.494559+0000","last_clean":"2024-09-13T14:41:41.494559+0000","last_became_active":"2024-09-13T14:41:41.493388+0000","last_became_peered":"2024-09-13T14:41:41.493388+0000","last_unstale":"2024-09-13T14:41:41.494559+0000","last_undegraded":"2024-09-13T14:41:41.494559+0000","last_fullsized":"2024-09-13T14:41:41.494559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:46.304592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496213+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:41:41.496213+0000","last_peered":"2024-09-13T14:41:41.496213+0000","last_clean":"2024-09-13T14:41:41.496213+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:41:41.496213+0000","last_undegraded":"2024-09-13T14:41:41.496213+0000","last_fullsized":"2024-09-13T14:41:41.496213+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490816+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:41:41.490816+0000","last_peered":"2024-09-13T14:41:41.490816+0000","last_clean":"2024-09-13T14:41:41.490816+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:41:41.490816+0000","last_undegraded":"2024-09-13T14:41:41.490816+0000","last_fullsized":"2024-09-13T14:41:41.490816+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":56,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482849+0000","last_change":"2024-09-13T14:28:14.292467+0000","last_active":"2024-09-13T14:41:41.482849+0000","last_peered":"2024-09-13T14:41:41.482849+0000","last_clean":"2024-09-13T14:41:41.482849+0000","last_became_active":"2024-09-13T14:28:14.292208+0000","last_became_peered":"2024-09-13T14:28:14.292208+0000","last_unstale":"2024-09-13T14:41:41.482849+0000","last_undegraded":"2024-09-13T14:41:41.482849+0000","last_fullsized":"2024-09-13T14:41:41.482849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:51.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486054+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:41:41.486054+0000","last_peered":"2024-09-13T14:41:41.486054+0000","last_clean":"2024-09-13T14:41:41.486054+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:41:41.486054+0000","last_undegraded":"2024-09-13T14:41:41.486054+0000","last_fullsized":"2024-09-13T14:41:41.486054+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479873+0000","last_change":"2024-09-13T14:40:04.161607+0000","last_active":"2024-09-13T14:41:41.479873+0000","last_peered":"2024-09-13T14:41:41.479873+0000","last_clean":"2024-09-13T14:41:41.479873+0000","last_became_active":"2024-09-13T14:40:04.161226+0000","last_became_peered":"2024-09-13T14:40:04.161226+0000","last_unstale":"2024-09-13T14:41:41.479873+0000","last_undegraded":"2024-09-13T14:41:41.479873+0000","last_fullsized":"2024-09-13T14:41:41.479873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:21.273027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484875+0000","last_change":"2024-09-13T14:40:04.164112+0000","last_active":"2024-09-13T14:41:41.484875+0000","last_peered":"2024-09-13T14:41:41.484875+0000","last_clean":"2024-09-13T14:41:41.484875+0000","last_became_active":"2024-09-13T14:40:04.163753+0000","last_became_peered":"2024-09-13T14:40:04.163753+0000","last_unstale":"2024-09-13T14:41:41.484875+0000","last_undegraded":"2024-09-13T14:41:41.484875+0000","last_fullsized":"2024-09-13T14:41:41.484875+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.685538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":55,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477581+0000","last_change":"2024-09-13T14:28:14.409515+0000","last_active":"2024-09-13T14:41:40.477581+0000","last_peered":"2024-09-13T14:41:40.477581+0000","last_clean":"2024-09-13T14:41:40.477581+0000","last_became_active":"2024-09-13T14:28:14.409202+0000","last_became_peered":"2024-09-13T14:28:14.409202+0000","last_unstale":"2024-09-13T14:41:40.477581+0000","last_undegraded":"2024-09-13T14:41:40.477581+0000","last_fullsized":"2024-09-13T14:41:40.477581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:24.530144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484456+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:41:41.484456+0000","last_peered":"2024-09-13T14:41:41.484456+0000","last_clean":"2024-09-13T14:41:41.484456+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:41:41.484456+0000","last_undegraded":"2024-09-13T14:41:41.484456+0000","last_fullsized":"2024-09-13T14:41:41.484456+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493612+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:41:41.493612+0000","last_peered":"2024-09-13T14:41:41.493612+0000","last_clean":"2024-09-13T14:41:41.493612+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:41:41.493612+0000","last_undegraded":"2024-09-13T14:41:41.493612+0000","last_fullsized":"2024-09-13T14:41:41.493612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484512+0000","last_change":"2024-09-13T14:28:14.289966+0000","last_active":"2024-09-13T14:41:41.484512+0000","last_peered":"2024-09-13T14:41:41.484512+0000","last_clean":"2024-09-13T14:41:41.484512+0000","last_became_active":"2024-09-13T14:28:14.289682+0000","last_became_peered":"2024-09-13T14:28:14.289682+0000","last_unstale":"2024-09-13T14:41:41.484512+0000","last_undegraded":"2024-09-13T14:41:41.484512+0000","last_fullsized":"2024-09-13T14:41:41.484512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:37.096298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":69,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483095+0000","last_change":"2024-09-13T14:40:36.880233+0000","last_active":"2024-09-13T14:41:40.483095+0000","last_peered":"2024-09-13T14:41:40.483095+0000","last_clean":"2024-09-13T14:41:40.483095+0000","last_became_active":"2024-09-13T14:40:36.879886+0000","last_became_peered":"2024-09-13T14:40:36.879886+0000","last_unstale":"2024-09-13T14:41:40.483095+0000","last_undegraded":"2024-09-13T14:41:40.483095+0000","last_fullsized":"2024-09-13T14:41:40.483095+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:49.693282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491428+0000","last_change":"2024-09-13T14:41:41.490296+0000","last_active":"2024-09-13T14:41:41.491428+0000","last_peered":"2024-09-13T14:41:41.491428+0000","last_clean":"2024-09-13T14:41:41.491428+0000","last_became_active":"2024-09-13T14:41:41.489937+0000","last_became_peered":"2024-09-13T14:41:41.489937+0000","last_unstale":"2024-09-13T14:41:41.491428+0000","last_undegraded":"2024-09-13T14:41:41.491428+0000","last_fullsized":"2024-09-13T14:41:41.491428+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:30.748843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482550+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:41:41.482550+0000","last_peered":"2024-09-13T14:41:41.482550+0000","last_clean":"2024-09-13T14:41:41.482550+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:41:41.482550+0000","last_undegraded":"2024-09-13T14:41:41.482550+0000","last_fullsized":"2024-09-13T14:41:41.482550+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491300+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:41:41.491300+0000","last_peered":"2024-09-13T14:41:41.491300+0000","last_clean":"2024-09-13T14:41:41.491300+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:41:41.491300+0000","last_undegraded":"2024-09-13T14:41:41.491300+0000","last_fullsized":"2024-09-13T14:41:41.491300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491488+0000","last_change":"2024-09-13T14:41:41.491050+0000","last_active":"2024-09-13T14:41:41.491488+0000","last_peered":"2024-09-13T14:41:41.491488+0000","last_clean":"2024-09-13T14:41:41.491488+0000","last_became_active":"2024-09-13T14:41:41.490645+0000","last_became_peered":"2024-09-13T14:41:41.490645+0000","last_unstale":"2024-09-13T14:41:41.491488+0000","last_undegraded":"2024-09-13T14:41:41.491488+0000","last_fullsized":"2024-09-13T14:41:41.491488+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:01.779411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483326+0000","last_change":"2024-09-13T14:40:36.879533+0000","last_active":"2024-09-13T14:41:40.483326+0000","last_peered":"2024-09-13T14:41:40.483326+0000","last_clean":"2024-09-13T14:41:40.483326+0000","last_became_active":"2024-09-13T14:40:36.879212+0000","last_became_peered":"2024-09-13T14:40:36.879212+0000","last_unstale":"2024-09-13T14:41:40.483326+0000","last_undegraded":"2024-09-13T14:41:40.483326+0000","last_fullsized":"2024-09-13T14:41:40.483326+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:03.087074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481810+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:41:41.481810+0000","last_peered":"2024-09-13T14:41:41.481810+0000","last_clean":"2024-09-13T14:41:41.481810+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:41:41.481810+0000","last_undegraded":"2024-09-13T14:41:41.481810+0000","last_fullsized":"2024-09-13T14:41:41.481810+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481078+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:41:40.481078+0000","last_peered":"2024-09-13T14:41:40.481078+0000","last_clean":"2024-09-13T14:41:40.481078+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:41:40.481078+0000","last_undegraded":"2024-09-13T14:41:40.481078+0000","last_fullsized":"2024-09-13T14:41:40.481078+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481692+0000","last_change":"2024-09-13T14:40:36.875852+0000","last_active":"2024-09-13T14:41:41.481692+0000","last_peered":"2024-09-13T14:41:41.481692+0000","last_clean":"2024-09-13T14:41:41.481692+0000","last_became_active":"2024-09-13T14:40:36.875486+0000","last_became_peered":"2024-09-13T14:40:36.875486+0000","last_unstale":"2024-09-13T14:41:41.481692+0000","last_undegraded":"2024-09-13T14:41:41.481692+0000","last_fullsized":"2024-09-13T14:41:41.481692+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:18.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488653+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:41:41.488653+0000","last_peered":"2024-09-13T14:41:41.488653+0000","last_clean":"2024-09-13T14:41:41.488653+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:41:41.488653+0000","last_undegraded":"2024-09-13T14:41:41.488653+0000","last_fullsized":"2024-09-13T14:41:41.488653+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488397+0000","last_change":"2024-09-13T14:41:41.488308+0000","last_active":"2024-09-13T14:41:41.488397+0000","last_peered":"2024-09-13T14:41:41.488397+0000","last_clean":"2024-09-13T14:41:41.488397+0000","last_became_active":"2024-09-13T14:41:41.487374+0000","last_became_peered":"2024-09-13T14:41:41.487374+0000","last_unstale":"2024-09-13T14:41:41.488397+0000","last_undegraded":"2024-09-13T14:41:41.488397+0000","last_fullsized":"2024-09-13T14:41:41.488397+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:51.242762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":402,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.483583+0000","last_change":"2024-09-13T14:40:04.164078+0000","last_active":"2024-09-13T14:41:41.483583+0000","last_peered":"2024-09-13T14:41:41.483583+0000","last_clean":"2024-09-13T14:41:41.483583+0000","last_became_active":"2024-09-13T14:40:04.163685+0000","last_became_peered":"2024-09-13T14:40:04.163685+0000","last_unstale":"2024-09-13T14:41:41.483583+0000","last_undegraded":"2024-09-13T14:41:41.483583+0000","last_fullsized":"2024-09-13T14:41:41.483583+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:40.476401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":601,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478473+0000","last_change":"2024-09-13T14:41:09.647983+0000","last_active":"2024-09-13T14:41:40.478473+0000","last_peered":"2024-09-13T14:41:40.478473+0000","last_clean":"2024-09-13T14:41:40.478473+0000","last_became_active":"2024-09-13T14:41:09.647486+0000","last_became_peered":"2024-09-13T14:41:09.647486+0000","last_unstale":"2024-09-13T14:41:40.478473+0000","last_undegraded":"2024-09-13T14:41:40.478473+0000","last_fullsized":"2024-09-13T14:41:40.478473+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:42.155770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.489718+0000","last_change":"2024-09-13T14:41:41.489630+0000","last_active":"2024-09-13T14:41:41.489718+0000","last_peered":"2024-09-13T14:41:41.489718+0000","last_clean":"2024-09-13T14:41:41.489718+0000","last_became_active":"2024-09-13T14:41:41.488964+0000","last_became_peered":"2024-09-13T14:41:41.488964+0000","last_unstale":"2024-09-13T14:41:41.489718+0000","last_undegraded":"2024-09-13T14:41:41.489718+0000","last_fullsized":"2024-09-13T14:41:41.489718+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:18.124795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483900+0000","last_change":"2024-09-13T14:28:14.409690+0000","last_active":"2024-09-13T14:41:40.483900+0000","last_peered":"2024-09-13T14:41:40.483900+0000","last_clean":"2024-09-13T14:41:40.483900+0000","last_became_active":"2024-09-13T14:28:14.409379+0000","last_became_peered":"2024-09-13T14:28:14.409379+0000","last_unstale":"2024-09-13T14:41:40.483900+0000","last_undegraded":"2024-09-13T14:41:40.483900+0000","last_fullsized":"2024-09-13T14:41:40.483900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:15.731901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":260,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.484376+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:41:40.484376+0000","last_peered":"2024-09-13T14:41:40.484376+0000","last_clean":"2024-09-13T14:41:40.484376+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:41:40.484376+0000","last_undegraded":"2024-09-13T14:41:40.484376+0000","last_fullsized":"2024-09-13T14:41:40.484376+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496253+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:41:41.496253+0000","last_peered":"2024-09-13T14:41:41.496253+0000","last_clean":"2024-09-13T14:41:41.496253+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:41:41.496253+0000","last_undegraded":"2024-09-13T14:41:41.496253+0000","last_fullsized":"2024-09-13T14:41:41.496253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482067+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:41:41.482067+0000","last_peered":"2024-09-13T14:41:41.482067+0000","last_clean":"2024-09-13T14:41:41.482067+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:41:41.482067+0000","last_undegraded":"2024-09-13T14:41:41.482067+0000","last_fullsized":"2024-09-13T14:41:41.482067+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":627,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492696+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:41:41.492696+0000","last_peered":"2024-09-13T14:41:41.492696+0000","last_clean":"2024-09-13T14:41:41.492696+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:41:41.492696+0000","last_undegraded":"2024-09-13T14:41:41.492696+0000","last_fullsized":"2024-09-13T14:41:41.492696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476974+0000","last_change":"2024-09-13T14:41:09.650296+0000","last_active":"2024-09-13T14:41:41.476974+0000","last_peered":"2024-09-13T14:41:41.476974+0000","last_clean":"2024-09-13T14:41:41.476974+0000","last_became_active":"2024-09-13T14:41:09.650016+0000","last_became_peered":"2024-09-13T14:41:09.650016+0000","last_unstale":"2024-09-13T14:41:41.476974+0000","last_undegraded":"2024-09-13T14:41:41.476974+0000","last_fullsized":"2024-09-13T14:41:41.476974+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:29.994396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479484+0000","last_change":"2024-09-13T14:41:09.655336+0000","last_active":"2024-09-13T14:41:41.479484+0000","last_peered":"2024-09-13T14:41:41.479484+0000","last_clean":"2024-09-13T14:41:41.479484+0000","last_became_active":"2024-09-13T14:41:09.654964+0000","last_became_peered":"2024-09-13T14:41:09.654964+0000","last_unstale":"2024-09-13T14:41:41.479484+0000","last_undegraded":"2024-09-13T14:41:41.479484+0000","last_fullsized":"2024-09-13T14:41:41.479484+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:40.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":724,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477655+0000","last_change":"2024-09-13T14:40:36.879942+0000","last_active":"2024-09-13T14:41:40.477655+0000","last_peered":"2024-09-13T14:41:40.477655+0000","last_clean":"2024-09-13T14:41:40.477655+0000","last_became_active":"2024-09-13T14:40:36.879489+0000","last_became_peered":"2024-09-13T14:40:36.879489+0000","last_unstale":"2024-09-13T14:41:40.477655+0000","last_undegraded":"2024-09-13T14:41:40.477655+0000","last_fullsized":"2024-09-13T14:41:40.477655+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:51.584189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":986,"num_read_kb":2939,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481639+0000","last_change":"2024-09-13T14:40:04.165067+0000","last_active":"2024-09-13T14:41:40.481639+0000","last_peered":"2024-09-13T14:41:40.481639+0000","last_clean":"2024-09-13T14:41:40.481639+0000","last_became_active":"2024-09-13T14:40:04.164682+0000","last_became_peered":"2024-09-13T14:40:04.164682+0000","last_unstale":"2024-09-13T14:41:40.481639+0000","last_undegraded":"2024-09-13T14:41:40.481639+0000","last_fullsized":"2024-09-13T14:41:40.481639+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:56.250533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484161+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:41:41.484161+0000","last_peered":"2024-09-13T14:41:41.484161+0000","last_clean":"2024-09-13T14:41:41.484161+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:41:41.484161+0000","last_undegraded":"2024-09-13T14:41:41.484161+0000","last_fullsized":"2024-09-13T14:41:41.484161+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.494473+0000","last_change":"2024-09-13T14:41:41.494473+0000","last_active":"2024-09-13T14:41:41.494473+0000","last_peered":"2024-09-13T14:41:41.494473+0000","last_clean":"2024-09-13T14:41:41.494473+0000","last_became_active":"2024-09-13T14:41:41.493836+0000","last_became_peered":"2024-09-13T14:41:41.493836+0000","last_unstale":"2024-09-13T14:41:41.494473+0000","last_undegraded":"2024-09-13T14:41:41.494473+0000","last_fullsized":"2024-09-13T14:41:41.494473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.352976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":594,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.477391+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:41:41.477391+0000","last_peered":"2024-09-13T14:41:41.477391+0000","last_clean":"2024-09-13T14:41:41.477391+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:41:41.477391+0000","last_undegraded":"2024-09-13T14:41:41.477391+0000","last_fullsized":"2024-09-13T14:41:41.477391+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476798+0000","last_change":"2024-09-13T14:40:36.878867+0000","last_active":"2024-09-13T14:41:41.476798+0000","last_peered":"2024-09-13T14:41:41.476798+0000","last_clean":"2024-09-13T14:41:41.476798+0000","last_became_active":"2024-09-13T14:40:36.873722+0000","last_became_peered":"2024-09-13T14:40:36.873722+0000","last_unstale":"2024-09-13T14:41:41.476798+0000","last_undegraded":"2024-09-13T14:41:41.476798+0000","last_fullsized":"2024-09-13T14:41:41.476798+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:36.046735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.477716+0000","last_change":"2024-09-13T14:41:09.651306+0000","last_active":"2024-09-13T14:41:41.477716+0000","last_peered":"2024-09-13T14:41:41.477716+0000","last_clean":"2024-09-13T14:41:41.477716+0000","last_became_active":"2024-09-13T14:41:09.650597+0000","last_became_peered":"2024-09-13T14:41:09.650597+0000","last_unstale":"2024-09-13T14:41:41.477716+0000","last_undegraded":"2024-09-13T14:41:41.477716+0000","last_fullsized":"2024-09-13T14:41:41.477716+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:02.341051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":421,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.485681+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:41:41.485681+0000","last_peered":"2024-09-13T14:41:41.485681+0000","last_clean":"2024-09-13T14:41:41.485681+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:41:41.485681+0000","last_undegraded":"2024-09-13T14:41:41.485681+0000","last_fullsized":"2024-09-13T14:41:41.485681+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482473+0000","last_change":"2024-09-13T14:40:04.163073+0000","last_active":"2024-09-13T14:41:41.482473+0000","last_peered":"2024-09-13T14:41:41.482473+0000","last_clean":"2024-09-13T14:41:41.482473+0000","last_became_active":"2024-09-13T14:40:04.162461+0000","last_became_peered":"2024-09-13T14:40:04.162461+0000","last_unstale":"2024-09-13T14:41:41.482473+0000","last_undegraded":"2024-09-13T14:41:41.482473+0000","last_fullsized":"2024-09-13T14:41:41.482473+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:28.279878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493597+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:41:41.493597+0000","last_peered":"2024-09-13T14:41:41.493597+0000","last_clean":"2024-09-13T14:41:41.493597+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:41:41.493597+0000","last_undegraded":"2024-09-13T14:41:41.493597+0000","last_fullsized":"2024-09-13T14:41:41.493597+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":478,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476856+0000","last_change":"2024-09-13T14:40:36.875554+0000","last_active":"2024-09-13T14:41:41.476856+0000","last_peered":"2024-09-13T14:41:41.476856+0000","last_clean":"2024-09-13T14:41:41.476856+0000","last_became_active":"2024-09-13T14:40:36.874924+0000","last_became_peered":"2024-09-13T14:40:36.874924+0000","last_unstale":"2024-09-13T14:41:41.476856+0000","last_undegraded":"2024-09-13T14:41:41.476856+0000","last_fullsized":"2024-09-13T14:41:41.476856+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:11.594853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486798+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:41:41.486798+0000","last_peered":"2024-09-13T14:41:41.486798+0000","last_clean":"2024-09-13T14:41:41.486798+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:41:41.486798+0000","last_undegraded":"2024-09-13T14:41:41.486798+0000","last_fullsized":"2024-09-13T14:41:41.486798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481946+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:41:41.481946+0000","last_peered":"2024-09-13T14:41:41.481946+0000","last_clean":"2024-09-13T14:41:41.481946+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:41:41.481946+0000","last_undegraded":"2024-09-13T14:41:41.481946+0000","last_fullsized":"2024-09-13T14:41:41.481946+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":675,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480619+0000","last_change":"2024-09-13T14:41:09.650053+0000","last_active":"2024-09-13T14:41:41.480619+0000","last_peered":"2024-09-13T14:41:41.480619+0000","last_clean":"2024-09-13T14:41:41.480619+0000","last_became_active":"2024-09-13T14:41:09.649530+0000","last_became_peered":"2024-09-13T14:41:09.649530+0000","last_unstale":"2024-09-13T14:41:41.480619+0000","last_undegraded":"2024-09-13T14:41:41.480619+0000","last_fullsized":"2024-09-13T14:41:41.480619+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:39.045795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.480192+0000","last_change":"2024-09-13T14:28:14.320314+0000","last_active":"2024-09-13T14:41:40.480192+0000","last_peered":"2024-09-13T14:41:40.480192+0000","last_clean":"2024-09-13T14:41:40.480192+0000","last_became_active":"2024-09-13T14:28:14.319934+0000","last_became_peered":"2024-09-13T14:28:14.319934+0000","last_unstale":"2024-09-13T14:41:40.480192+0000","last_undegraded":"2024-09-13T14:41:40.480192+0000","last_fullsized":"2024-09-13T14:41:40.480192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:21.514353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480732+0000","last_change":"2024-09-13T14:40:36.870296+0000","last_active":"2024-09-13T14:41:41.480732+0000","last_peered":"2024-09-13T14:41:41.480732+0000","last_clean":"2024-09-13T14:41:41.480732+0000","last_became_active":"2024-09-13T14:40:36.869969+0000","last_became_peered":"2024-09-13T14:40:36.869969+0000","last_unstale":"2024-09-13T14:41:41.480732+0000","last_undegraded":"2024-09-13T14:41:41.480732+0000","last_fullsized":"2024-09-13T14:41:41.480732+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:57.021118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481406+0000","last_change":"2024-09-13T14:28:14.293329+0000","last_active":"2024-09-13T14:41:41.481406+0000","last_peered":"2024-09-13T14:41:41.481406+0000","last_clean":"2024-09-13T14:41:41.481406+0000","last_became_active":"2024-09-13T14:28:14.293075+0000","last_became_peered":"2024-09-13T14:28:14.293075+0000","last_unstale":"2024-09-13T14:41:41.481406+0000","last_undegraded":"2024-09-13T14:41:41.481406+0000","last_fullsized":"2024-09-13T14:41:41.481406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:20.310761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476555+0000","last_change":"2024-09-13T14:41:09.650180+0000","last_active":"2024-09-13T14:41:41.476555+0000","last_peered":"2024-09-13T14:41:41.476555+0000","last_clean":"2024-09-13T14:41:41.476555+0000","last_became_active":"2024-09-13T14:41:09.649908+0000","last_became_peered":"2024-09-13T14:41:09.649908+0000","last_unstale":"2024-09-13T14:41:41.476555+0000","last_undegraded":"2024-09-13T14:41:41.476555+0000","last_fullsized":"2024-09-13T14:41:41.476555+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:08.029267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490320+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:41:41.490320+0000","last_peered":"2024-09-13T14:41:41.490320+0000","last_clean":"2024-09-13T14:41:41.490320+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:41:41.490320+0000","last_undegraded":"2024-09-13T14:41:41.490320+0000","last_fullsized":"2024-09-13T14:41:41.490320+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477969+0000","last_change":"2024-09-13T14:40:36.879188+0000","last_active":"2024-09-13T14:41:40.477969+0000","last_peered":"2024-09-13T14:41:40.477969+0000","last_clean":"2024-09-13T14:41:40.477969+0000","last_became_active":"2024-09-13T14:40:36.878788+0000","last_became_peered":"2024-09-13T14:40:36.878788+0000","last_unstale":"2024-09-13T14:41:40.477969+0000","last_undegraded":"2024-09-13T14:41:40.477969+0000","last_fullsized":"2024-09-13T14:41:40.477969+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:11.826299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484913+0000","last_change":"2024-09-13T14:41:41.484817+0000","last_active":"2024-09-13T14:41:41.484913+0000","last_peered":"2024-09-13T14:41:41.484913+0000","last_clean":"2024-09-13T14:41:41.484913+0000","last_became_active":"2024-09-13T14:41:41.484505+0000","last_became_peered":"2024-09-13T14:41:41.484505+0000","last_unstale":"2024-09-13T14:41:41.484913+0000","last_undegraded":"2024-09-13T14:41:41.484913+0000","last_fullsized":"2024-09-13T14:41:41.484913+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:49.618334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496242+0000","last_change":"2024-09-13T14:41:41.491895+0000","last_active":"2024-09-13T14:41:41.496242+0000","last_peered":"2024-09-13T14:41:41.496242+0000","last_clean":"2024-09-13T14:41:41.496242+0000","last_became_active":"2024-09-13T14:41:41.491517+0000","last_became_peered":"2024-09-13T14:41:41.491517+0000","last_unstale":"2024-09-13T14:41:41.496242+0000","last_undegraded":"2024-09-13T14:41:41.496242+0000","last_fullsized":"2024-09-13T14:41:41.496242+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:08.848118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490024+0000","last_change":"2024-09-13T14:41:41.489947+0000","last_active":"2024-09-13T14:41:41.490024+0000","last_peered":"2024-09-13T14:41:41.490024+0000","last_clean":"2024-09-13T14:41:41.490024+0000","last_became_active":"2024-09-13T14:41:41.487936+0000","last_became_peered":"2024-09-13T14:41:41.487936+0000","last_unstale":"2024-09-13T14:41:41.490024+0000","last_undegraded":"2024-09-13T14:41:41.490024+0000","last_fullsized":"2024-09-13T14:41:41.490024+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":69,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481742+0000","last_change":"2024-09-13T14:40:36.875850+0000","last_active":"2024-09-13T14:41:40.481742+0000","last_peered":"2024-09-13T14:41:40.481742+0000","last_clean":"2024-09-13T14:41:40.481742+0000","last_became_active":"2024-09-13T14:40:36.875550+0000","last_became_peered":"2024-09-13T14:40:36.875550+0000","last_unstale":"2024-09-13T14:41:40.481742+0000","last_undegraded":"2024-09-13T14:41:40.481742+0000","last_fullsized":"2024-09-13T14:41:40.481742+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:57.875071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492293+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:41:41.492293+0000","last_peered":"2024-09-13T14:41:41.492293+0000","last_clean":"2024-09-13T14:41:41.492293+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:41:41.492293+0000","last_undegraded":"2024-09-13T14:41:41.492293+0000","last_fullsized":"2024-09-13T14:41:41.492293+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490839+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:41:41.490839+0000","last_peered":"2024-09-13T14:41:41.490839+0000","last_clean":"2024-09-13T14:41:41.490839+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:41:41.490839+0000","last_undegraded":"2024-09-13T14:41:41.490839+0000","last_fullsized":"2024-09-13T14:41:41.490839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486110+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:41:41.486110+0000","last_peered":"2024-09-13T14:41:41.486110+0000","last_clean":"2024-09-13T14:41:41.486110+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:41:41.486110+0000","last_undegraded":"2024-09-13T14:41:41.486110+0000","last_fullsized":"2024-09-13T14:41:41.486110+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493149+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:41:41.493149+0000","last_peered":"2024-09-13T14:41:41.493149+0000","last_clean":"2024-09-13T14:41:41.493149+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:41:41.493149+0000","last_undegraded":"2024-09-13T14:41:41.493149+0000","last_fullsized":"2024-09-13T14:41:41.493149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.479047+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:41:40.479047+0000","last_peered":"2024-09-13T14:41:40.479047+0000","last_clean":"2024-09-13T14:41:40.479047+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:41:40.479047+0000","last_undegraded":"2024-09-13T14:41:40.479047+0000","last_fullsized":"2024-09-13T14:41:40.479047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490987+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:41:41.490987+0000","last_peered":"2024-09-13T14:41:41.490987+0000","last_clean":"2024-09-13T14:41:41.490987+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:41:41.490987+0000","last_undegraded":"2024-09-13T14:41:41.490987+0000","last_fullsized":"2024-09-13T14:41:41.490987+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482609+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:41:41.482609+0000","last_peered":"2024-09-13T14:41:41.482609+0000","last_clean":"2024-09-13T14:41:41.482609+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:41:41.482609+0000","last_undegraded":"2024-09-13T14:41:41.482609+0000","last_fullsized":"2024-09-13T14:41:41.482609+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490102+0000","last_change":"2024-09-13T14:41:41.490102+0000","last_active":"2024-09-13T14:41:41.490102+0000","last_peered":"2024-09-13T14:41:41.490102+0000","last_clean":"2024-09-13T14:41:41.490102+0000","last_became_active":"2024-09-13T14:41:41.489822+0000","last_became_peered":"2024-09-13T14:41:41.489822+0000","last_unstale":"2024-09-13T14:41:41.490102+0000","last_undegraded":"2024-09-13T14:41:41.490102+0000","last_fullsized":"2024-09-13T14:41:41.490102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:28.447149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477814+0000","last_change":"2024-09-13T14:40:36.879235+0000","last_active":"2024-09-13T14:41:40.477814+0000","last_peered":"2024-09-13T14:41:40.477814+0000","last_clean":"2024-09-13T14:41:40.477814+0000","last_became_active":"2024-09-13T14:40:36.878874+0000","last_became_peered":"2024-09-13T14:40:36.878874+0000","last_unstale":"2024-09-13T14:41:40.477814+0000","last_undegraded":"2024-09-13T14:41:40.477814+0000","last_fullsized":"2024-09-13T14:41:40.477814+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:07.605097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490555+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:41:41.490555+0000","last_peered":"2024-09-13T14:41:41.490555+0000","last_clean":"2024-09-13T14:41:41.490555+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:41:41.490555+0000","last_undegraded":"2024-09-13T14:41:41.490555+0000","last_fullsized":"2024-09-13T14:41:41.490555+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":57,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478200+0000","last_change":"2024-09-13T14:28:14.297761+0000","last_active":"2024-09-13T14:41:40.478200+0000","last_peered":"2024-09-13T14:41:40.478200+0000","last_clean":"2024-09-13T14:41:40.478200+0000","last_became_active":"2024-09-13T14:28:14.297478+0000","last_became_peered":"2024-09-13T14:28:14.297478+0000","last_unstale":"2024-09-13T14:41:40.478200+0000","last_undegraded":"2024-09-13T14:41:40.478200+0000","last_fullsized":"2024-09-13T14:41:40.478200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:48.450673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488366+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:41:41.488366+0000","last_peered":"2024-09-13T14:41:41.488366+0000","last_clean":"2024-09-13T14:41:41.488366+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:41:41.488366+0000","last_undegraded":"2024-09-13T14:41:41.488366+0000","last_fullsized":"2024-09-13T14:41:41.488366+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486185+0000","last_change":"2024-09-13T14:41:41.486093+0000","last_active":"2024-09-13T14:41:41.486185+0000","last_peered":"2024-09-13T14:41:41.486185+0000","last_clean":"2024-09-13T14:41:41.486185+0000","last_became_active":"2024-09-13T14:41:41.485759+0000","last_became_peered":"2024-09-13T14:41:41.485759+0000","last_unstale":"2024-09-13T14:41:41.486185+0000","last_undegraded":"2024-09-13T14:41:41.486185+0000","last_fullsized":"2024-09-13T14:41:41.486185+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:05.424465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478481+0000","last_change":"2024-09-13T14:40:36.876656+0000","last_active":"2024-09-13T14:41:40.478481+0000","last_peered":"2024-09-13T14:41:40.478481+0000","last_clean":"2024-09-13T14:41:40.478481+0000","last_became_active":"2024-09-13T14:40:36.876361+0000","last_became_peered":"2024-09-13T14:40:36.876361+0000","last_unstale":"2024-09-13T14:41:40.478481+0000","last_undegraded":"2024-09-13T14:41:40.478481+0000","last_fullsized":"2024-09-13T14:41:40.478481+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:51.843071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482257+0000","last_change":"2024-09-13T14:41:09.653285+0000","last_active":"2024-09-13T14:41:41.482257+0000","last_peered":"2024-09-13T14:41:41.482257+0000","last_clean":"2024-09-13T14:41:41.482257+0000","last_became_active":"2024-09-13T14:41:09.652737+0000","last_became_peered":"2024-09-13T14:41:09.652737+0000","last_unstale":"2024-09-13T14:41:41.482257+0000","last_undegraded":"2024-09-13T14:41:41.482257+0000","last_fullsized":"2024-09-13T14:41:41.482257+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:13.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234184,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32892,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712260,"statfs":{"total":95995035648,"available":95961354240,"internally_reserved":0,"allocated":1818624,"data_stored":1215821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495820,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1806336,"data_stored":1215421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757457,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1208320,"data_stored":621210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019095,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1224704,"data_stored":625714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856467983,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32748,"kb_used_data":5272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93712404,"statfs":{"total":95995035648,"available":95961501696,"internally_reserved":0,"allocated":5398528,"data_stored":4815218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":28109988},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631509,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32808,"kb_used_data":5276,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712344,"statfs":{"total":95995035648,"available":95961440256,"internally_reserved":0,"allocated":5402624,"data_stored":4815744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":28172255},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795036,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1176,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1204224,"data_stored":620843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":28172914},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958563,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37548,"kb_used_data":5856,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93707604,"statfs":{"total":95995035648,"available":95956586496,"internally_reserved":0,"allocated":5996544,"data_stored":5406100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":32432615},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:48.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:42:48.722 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:48.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:48 smithi154 ceph-mon[102314]: pgmap v280: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:48.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:48 smithi154 ceph-mon[102314]: from='client.44480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:48.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:48 smithi154 ceph-mon[107038]: pgmap v280: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:48.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:48 smithi154 ceph-mon[107038]: from='client.44480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:48.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:48 smithi172 ceph-mon[100696]: pgmap v280: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:48.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:48 smithi172 ceph-mon[100696]: from='client.44480 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:50.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[102314]: pgmap v281: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:50.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:50.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:50.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[107038]: pgmap v281: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:50.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:50.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:50.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:50 smithi172 ceph-mon[100696]: pgmap v281: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:42:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:42:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:42:50.652 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:42:50.653 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:42:51.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:51 smithi154 ceph-mon[102314]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:51.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:51 smithi154 ceph-mon[107038]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:51.270 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-13T14:42:48.776853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":262208,"kb_used_data":23496,"kb_used_omap":34,"kb_used_meta":238557,"kb_avail":749699008,"statfs":{"total":767960285184,"available":767691784192,"internally_reserved":0,"allocated":24059904,"data_stored":19336071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34821,"internal_metadata":244283387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002518"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488438+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:41:41.488438+0000","last_peered":"2024-09-13T14:41:41.488438+0000","last_clean":"2024-09-13T14:41:41.488438+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:41:41.488438+0000","last_undegraded":"2024-09-13T14:41:41.488438+0000","last_fullsized":"2024-09-13T14:41:41.488438+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481357+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:41:41.481357+0000","last_peered":"2024-09-13T14:41:41.481357+0000","last_clean":"2024-09-13T14:41:41.481357+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:41:41.481357+0000","last_undegraded":"2024-09-13T14:41:41.481357+0000","last_fullsized":"2024-09-13T14:41:41.481357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.494559+0000","last_change":"2024-09-13T14:41:41.493742+0000","last_active":"2024-09-13T14:41:41.494559+0000","last_peered":"2024-09-13T14:41:41.494559+0000","last_clean":"2024-09-13T14:41:41.494559+0000","last_became_active":"2024-09-13T14:41:41.493388+0000","last_became_peered":"2024-09-13T14:41:41.493388+0000","last_unstale":"2024-09-13T14:41:41.494559+0000","last_undegraded":"2024-09-13T14:41:41.494559+0000","last_fullsized":"2024-09-13T14:41:41.494559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:46.304592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496213+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:41:41.496213+0000","last_peered":"2024-09-13T14:41:41.496213+0000","last_clean":"2024-09-13T14:41:41.496213+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:41:41.496213+0000","last_undegraded":"2024-09-13T14:41:41.496213+0000","last_fullsized":"2024-09-13T14:41:41.496213+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490816+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:41:41.490816+0000","last_peered":"2024-09-13T14:41:41.490816+0000","last_clean":"2024-09-13T14:41:41.490816+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:41:41.490816+0000","last_undegraded":"2024-09-13T14:41:41.490816+0000","last_fullsized":"2024-09-13T14:41:41.490816+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":56,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482849+0000","last_change":"2024-09-13T14:28:14.292467+0000","last_active":"2024-09-13T14:41:41.482849+0000","last_peered":"2024-09-13T14:41:41.482849+0000","last_clean":"2024-09-13T14:41:41.482849+0000","last_became_active":"2024-09-13T14:28:14.292208+0000","last_became_peered":"2024-09-13T14:28:14.292208+0000","last_unstale":"2024-09-13T14:41:41.482849+0000","last_undegraded":"2024-09-13T14:41:41.482849+0000","last_fullsized":"2024-09-13T14:41:41.482849+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:51.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486054+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:41:41.486054+0000","last_peered":"2024-09-13T14:41:41.486054+0000","last_clean":"2024-09-13T14:41:41.486054+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:41:41.486054+0000","last_undegraded":"2024-09-13T14:41:41.486054+0000","last_fullsized":"2024-09-13T14:41:41.486054+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479873+0000","last_change":"2024-09-13T14:40:04.161607+0000","last_active":"2024-09-13T14:41:41.479873+0000","last_peered":"2024-09-13T14:41:41.479873+0000","last_clean":"2024-09-13T14:41:41.479873+0000","last_became_active":"2024-09-13T14:40:04.161226+0000","last_became_peered":"2024-09-13T14:40:04.161226+0000","last_unstale":"2024-09-13T14:41:41.479873+0000","last_undegraded":"2024-09-13T14:41:41.479873+0000","last_fullsized":"2024-09-13T14:41:41.479873+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:21.273027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484875+0000","last_change":"2024-09-13T14:40:04.164112+0000","last_active":"2024-09-13T14:41:41.484875+0000","last_peered":"2024-09-13T14:41:41.484875+0000","last_clean":"2024-09-13T14:41:41.484875+0000","last_became_active":"2024-09-13T14:40:04.163753+0000","last_became_peered":"2024-09-13T14:40:04.163753+0000","last_unstale":"2024-09-13T14:41:41.484875+0000","last_undegraded":"2024-09-13T14:41:41.484875+0000","last_fullsized":"2024-09-13T14:41:41.484875+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.685538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":55,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477581+0000","last_change":"2024-09-13T14:28:14.409515+0000","last_active":"2024-09-13T14:41:40.477581+0000","last_peered":"2024-09-13T14:41:40.477581+0000","last_clean":"2024-09-13T14:41:40.477581+0000","last_became_active":"2024-09-13T14:28:14.409202+0000","last_became_peered":"2024-09-13T14:28:14.409202+0000","last_unstale":"2024-09-13T14:41:40.477581+0000","last_undegraded":"2024-09-13T14:41:40.477581+0000","last_fullsized":"2024-09-13T14:41:40.477581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:33:24.530144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484456+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:41:41.484456+0000","last_peered":"2024-09-13T14:41:41.484456+0000","last_clean":"2024-09-13T14:41:41.484456+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:41:41.484456+0000","last_undegraded":"2024-09-13T14:41:41.484456+0000","last_fullsized":"2024-09-13T14:41:41.484456+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493612+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:41:41.493612+0000","last_peered":"2024-09-13T14:41:41.493612+0000","last_clean":"2024-09-13T14:41:41.493612+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:41:41.493612+0000","last_undegraded":"2024-09-13T14:41:41.493612+0000","last_fullsized":"2024-09-13T14:41:41.493612+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":57,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484512+0000","last_change":"2024-09-13T14:28:14.289966+0000","last_active":"2024-09-13T14:41:41.484512+0000","last_peered":"2024-09-13T14:41:41.484512+0000","last_clean":"2024-09-13T14:41:41.484512+0000","last_became_active":"2024-09-13T14:28:14.289682+0000","last_became_peered":"2024-09-13T14:28:14.289682+0000","last_unstale":"2024-09-13T14:41:41.484512+0000","last_undegraded":"2024-09-13T14:41:41.484512+0000","last_fullsized":"2024-09-13T14:41:41.484512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:37.096298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":69,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483095+0000","last_change":"2024-09-13T14:40:36.880233+0000","last_active":"2024-09-13T14:41:40.483095+0000","last_peered":"2024-09-13T14:41:40.483095+0000","last_clean":"2024-09-13T14:41:40.483095+0000","last_became_active":"2024-09-13T14:40:36.879886+0000","last_became_peered":"2024-09-13T14:40:36.879886+0000","last_unstale":"2024-09-13T14:41:40.483095+0000","last_undegraded":"2024-09-13T14:41:40.483095+0000","last_fullsized":"2024-09-13T14:41:40.483095+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:49.693282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491428+0000","last_change":"2024-09-13T14:41:41.490296+0000","last_active":"2024-09-13T14:41:41.491428+0000","last_peered":"2024-09-13T14:41:41.491428+0000","last_clean":"2024-09-13T14:41:41.491428+0000","last_became_active":"2024-09-13T14:41:41.489937+0000","last_became_peered":"2024-09-13T14:41:41.489937+0000","last_unstale":"2024-09-13T14:41:41.491428+0000","last_undegraded":"2024-09-13T14:41:41.491428+0000","last_fullsized":"2024-09-13T14:41:41.491428+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:30.748843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482550+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:41:41.482550+0000","last_peered":"2024-09-13T14:41:41.482550+0000","last_clean":"2024-09-13T14:41:41.482550+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:41:41.482550+0000","last_undegraded":"2024-09-13T14:41:41.482550+0000","last_fullsized":"2024-09-13T14:41:41.482550+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491300+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:41:41.491300+0000","last_peered":"2024-09-13T14:41:41.491300+0000","last_clean":"2024-09-13T14:41:41.491300+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:41:41.491300+0000","last_undegraded":"2024-09-13T14:41:41.491300+0000","last_fullsized":"2024-09-13T14:41:41.491300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.491488+0000","last_change":"2024-09-13T14:41:41.491050+0000","last_active":"2024-09-13T14:41:41.491488+0000","last_peered":"2024-09-13T14:41:41.491488+0000","last_clean":"2024-09-13T14:41:41.491488+0000","last_became_active":"2024-09-13T14:41:41.490645+0000","last_became_peered":"2024-09-13T14:41:41.490645+0000","last_unstale":"2024-09-13T14:41:41.491488+0000","last_undegraded":"2024-09-13T14:41:41.491488+0000","last_fullsized":"2024-09-13T14:41:41.491488+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:01.779411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483326+0000","last_change":"2024-09-13T14:40:36.879533+0000","last_active":"2024-09-13T14:41:40.483326+0000","last_peered":"2024-09-13T14:41:40.483326+0000","last_clean":"2024-09-13T14:41:40.483326+0000","last_became_active":"2024-09-13T14:40:36.879212+0000","last_became_peered":"2024-09-13T14:40:36.879212+0000","last_unstale":"2024-09-13T14:41:40.483326+0000","last_undegraded":"2024-09-13T14:41:40.483326+0000","last_fullsized":"2024-09-13T14:41:40.483326+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:03.087074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481810+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:41:41.481810+0000","last_peered":"2024-09-13T14:41:41.481810+0000","last_clean":"2024-09-13T14:41:41.481810+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:41:41.481810+0000","last_undegraded":"2024-09-13T14:41:41.481810+0000","last_fullsized":"2024-09-13T14:41:41.481810+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481078+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:41:40.481078+0000","last_peered":"2024-09-13T14:41:40.481078+0000","last_clean":"2024-09-13T14:41:40.481078+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:41:40.481078+0000","last_undegraded":"2024-09-13T14:41:40.481078+0000","last_fullsized":"2024-09-13T14:41:40.481078+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481692+0000","last_change":"2024-09-13T14:40:36.875852+0000","last_active":"2024-09-13T14:41:41.481692+0000","last_peered":"2024-09-13T14:41:41.481692+0000","last_clean":"2024-09-13T14:41:41.481692+0000","last_became_active":"2024-09-13T14:40:36.875486+0000","last_became_peered":"2024-09-13T14:40:36.875486+0000","last_unstale":"2024-09-13T14:41:41.481692+0000","last_undegraded":"2024-09-13T14:41:41.481692+0000","last_fullsized":"2024-09-13T14:41:41.481692+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:18.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488653+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:41:41.488653+0000","last_peered":"2024-09-13T14:41:41.488653+0000","last_clean":"2024-09-13T14:41:41.488653+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:41:41.488653+0000","last_undegraded":"2024-09-13T14:41:41.488653+0000","last_fullsized":"2024-09-13T14:41:41.488653+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488397+0000","last_change":"2024-09-13T14:41:41.488308+0000","last_active":"2024-09-13T14:41:41.488397+0000","last_peered":"2024-09-13T14:41:41.488397+0000","last_clean":"2024-09-13T14:41:41.488397+0000","last_became_active":"2024-09-13T14:41:41.487374+0000","last_became_peered":"2024-09-13T14:41:41.487374+0000","last_unstale":"2024-09-13T14:41:41.488397+0000","last_undegraded":"2024-09-13T14:41:41.488397+0000","last_fullsized":"2024-09-13T14:41:41.488397+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:51.242762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":402,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.483583+0000","last_change":"2024-09-13T14:40:04.164078+0000","last_active":"2024-09-13T14:41:41.483583+0000","last_peered":"2024-09-13T14:41:41.483583+0000","last_clean":"2024-09-13T14:41:41.483583+0000","last_became_active":"2024-09-13T14:40:04.163685+0000","last_became_peered":"2024-09-13T14:40:04.163685+0000","last_unstale":"2024-09-13T14:41:41.483583+0000","last_undegraded":"2024-09-13T14:41:41.483583+0000","last_fullsized":"2024-09-13T14:41:41.483583+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:40.476401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":601,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478473+0000","last_change":"2024-09-13T14:41:09.647983+0000","last_active":"2024-09-13T14:41:40.478473+0000","last_peered":"2024-09-13T14:41:40.478473+0000","last_clean":"2024-09-13T14:41:40.478473+0000","last_became_active":"2024-09-13T14:41:09.647486+0000","last_became_peered":"2024-09-13T14:41:09.647486+0000","last_unstale":"2024-09-13T14:41:40.478473+0000","last_undegraded":"2024-09-13T14:41:40.478473+0000","last_fullsized":"2024-09-13T14:41:40.478473+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:42.155770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.489718+0000","last_change":"2024-09-13T14:41:41.489630+0000","last_active":"2024-09-13T14:41:41.489718+0000","last_peered":"2024-09-13T14:41:41.489718+0000","last_clean":"2024-09-13T14:41:41.489718+0000","last_became_active":"2024-09-13T14:41:41.488964+0000","last_became_peered":"2024-09-13T14:41:41.488964+0000","last_unstale":"2024-09-13T14:41:41.489718+0000","last_undegraded":"2024-09-13T14:41:41.489718+0000","last_fullsized":"2024-09-13T14:41:41.489718+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:18.124795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.483900+0000","last_change":"2024-09-13T14:28:14.409690+0000","last_active":"2024-09-13T14:41:40.483900+0000","last_peered":"2024-09-13T14:41:40.483900+0000","last_clean":"2024-09-13T14:41:40.483900+0000","last_became_active":"2024-09-13T14:28:14.409379+0000","last_became_peered":"2024-09-13T14:28:14.409379+0000","last_unstale":"2024-09-13T14:41:40.483900+0000","last_undegraded":"2024-09-13T14:41:40.483900+0000","last_fullsized":"2024-09-13T14:41:40.483900+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:15.731901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":260,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.484376+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:41:40.484376+0000","last_peered":"2024-09-13T14:41:40.484376+0000","last_clean":"2024-09-13T14:41:40.484376+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:41:40.484376+0000","last_undegraded":"2024-09-13T14:41:40.484376+0000","last_fullsized":"2024-09-13T14:41:40.484376+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496253+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:41:41.496253+0000","last_peered":"2024-09-13T14:41:41.496253+0000","last_clean":"2024-09-13T14:41:41.496253+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:41:41.496253+0000","last_undegraded":"2024-09-13T14:41:41.496253+0000","last_fullsized":"2024-09-13T14:41:41.496253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":75,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482067+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:41:41.482067+0000","last_peered":"2024-09-13T14:41:41.482067+0000","last_clean":"2024-09-13T14:41:41.482067+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:41:41.482067+0000","last_undegraded":"2024-09-13T14:41:41.482067+0000","last_fullsized":"2024-09-13T14:41:41.482067+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":627,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492696+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:41:41.492696+0000","last_peered":"2024-09-13T14:41:41.492696+0000","last_clean":"2024-09-13T14:41:41.492696+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:41:41.492696+0000","last_undegraded":"2024-09-13T14:41:41.492696+0000","last_fullsized":"2024-09-13T14:41:41.492696+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476974+0000","last_change":"2024-09-13T14:41:09.650296+0000","last_active":"2024-09-13T14:41:41.476974+0000","last_peered":"2024-09-13T14:41:41.476974+0000","last_clean":"2024-09-13T14:41:41.476974+0000","last_became_active":"2024-09-13T14:41:09.650016+0000","last_became_peered":"2024-09-13T14:41:09.650016+0000","last_unstale":"2024-09-13T14:41:41.476974+0000","last_undegraded":"2024-09-13T14:41:41.476974+0000","last_fullsized":"2024-09-13T14:41:41.476974+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:29.994396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479484+0000","last_change":"2024-09-13T14:41:09.655336+0000","last_active":"2024-09-13T14:41:41.479484+0000","last_peered":"2024-09-13T14:41:41.479484+0000","last_clean":"2024-09-13T14:41:41.479484+0000","last_became_active":"2024-09-13T14:41:09.654964+0000","last_became_peered":"2024-09-13T14:41:09.654964+0000","last_unstale":"2024-09-13T14:41:41.479484+0000","last_undegraded":"2024-09-13T14:41:41.479484+0000","last_fullsized":"2024-09-13T14:41:41.479484+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:40.102489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":724,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477655+0000","last_change":"2024-09-13T14:40:36.879942+0000","last_active":"2024-09-13T14:41:40.477655+0000","last_peered":"2024-09-13T14:41:40.477655+0000","last_clean":"2024-09-13T14:41:40.477655+0000","last_became_active":"2024-09-13T14:40:36.879489+0000","last_became_peered":"2024-09-13T14:40:36.879489+0000","last_unstale":"2024-09-13T14:41:40.477655+0000","last_undegraded":"2024-09-13T14:41:40.477655+0000","last_fullsized":"2024-09-13T14:41:40.477655+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:51.584189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":986,"num_read_kb":2939,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.481639+0000","last_change":"2024-09-13T14:40:04.165067+0000","last_active":"2024-09-13T14:41:40.481639+0000","last_peered":"2024-09-13T14:41:40.481639+0000","last_clean":"2024-09-13T14:41:40.481639+0000","last_became_active":"2024-09-13T14:40:04.164682+0000","last_became_peered":"2024-09-13T14:40:04.164682+0000","last_unstale":"2024-09-13T14:41:40.481639+0000","last_undegraded":"2024-09-13T14:41:40.481639+0000","last_fullsized":"2024-09-13T14:41:40.481639+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:56.250533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":66,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484161+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:41:41.484161+0000","last_peered":"2024-09-13T14:41:41.484161+0000","last_clean":"2024-09-13T14:41:41.484161+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:41:41.484161+0000","last_undegraded":"2024-09-13T14:41:41.484161+0000","last_fullsized":"2024-09-13T14:41:41.484161+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.494473+0000","last_change":"2024-09-13T14:41:41.494473+0000","last_active":"2024-09-13T14:41:41.494473+0000","last_peered":"2024-09-13T14:41:41.494473+0000","last_clean":"2024-09-13T14:41:41.494473+0000","last_became_active":"2024-09-13T14:41:41.493836+0000","last_became_peered":"2024-09-13T14:41:41.493836+0000","last_unstale":"2024-09-13T14:41:41.494473+0000","last_undegraded":"2024-09-13T14:41:41.494473+0000","last_fullsized":"2024-09-13T14:41:41.494473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.352976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":594,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.477391+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:41:41.477391+0000","last_peered":"2024-09-13T14:41:41.477391+0000","last_clean":"2024-09-13T14:41:41.477391+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:41:41.477391+0000","last_undegraded":"2024-09-13T14:41:41.477391+0000","last_fullsized":"2024-09-13T14:41:41.477391+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476798+0000","last_change":"2024-09-13T14:40:36.878867+0000","last_active":"2024-09-13T14:41:41.476798+0000","last_peered":"2024-09-13T14:41:41.476798+0000","last_clean":"2024-09-13T14:41:41.476798+0000","last_became_active":"2024-09-13T14:40:36.873722+0000","last_became_peered":"2024-09-13T14:40:36.873722+0000","last_unstale":"2024-09-13T14:41:41.476798+0000","last_undegraded":"2024-09-13T14:41:41.476798+0000","last_fullsized":"2024-09-13T14:41:41.476798+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:36.046735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.477716+0000","last_change":"2024-09-13T14:41:09.651306+0000","last_active":"2024-09-13T14:41:41.477716+0000","last_peered":"2024-09-13T14:41:41.477716+0000","last_clean":"2024-09-13T14:41:41.477716+0000","last_became_active":"2024-09-13T14:41:09.650597+0000","last_became_peered":"2024-09-13T14:41:09.650597+0000","last_unstale":"2024-09-13T14:41:41.477716+0000","last_undegraded":"2024-09-13T14:41:41.477716+0000","last_fullsized":"2024-09-13T14:41:41.477716+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:02.341051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":421,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.485681+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:41:41.485681+0000","last_peered":"2024-09-13T14:41:41.485681+0000","last_clean":"2024-09-13T14:41:41.485681+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:41:41.485681+0000","last_undegraded":"2024-09-13T14:41:41.485681+0000","last_fullsized":"2024-09-13T14:41:41.485681+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482473+0000","last_change":"2024-09-13T14:40:04.163073+0000","last_active":"2024-09-13T14:41:41.482473+0000","last_peered":"2024-09-13T14:41:41.482473+0000","last_clean":"2024-09-13T14:41:41.482473+0000","last_became_active":"2024-09-13T14:40:04.162461+0000","last_became_peered":"2024-09-13T14:40:04.162461+0000","last_unstale":"2024-09-13T14:41:41.482473+0000","last_undegraded":"2024-09-13T14:41:41.482473+0000","last_fullsized":"2024-09-13T14:41:41.482473+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:28.279878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":72,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493597+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:41:41.493597+0000","last_peered":"2024-09-13T14:41:41.493597+0000","last_clean":"2024-09-13T14:41:41.493597+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:41:41.493597+0000","last_undegraded":"2024-09-13T14:41:41.493597+0000","last_fullsized":"2024-09-13T14:41:41.493597+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":478,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476856+0000","last_change":"2024-09-13T14:40:36.875554+0000","last_active":"2024-09-13T14:41:41.476856+0000","last_peered":"2024-09-13T14:41:41.476856+0000","last_clean":"2024-09-13T14:41:41.476856+0000","last_became_active":"2024-09-13T14:40:36.874924+0000","last_became_peered":"2024-09-13T14:40:36.874924+0000","last_unstale":"2024-09-13T14:41:41.476856+0000","last_undegraded":"2024-09-13T14:41:41.476856+0000","last_fullsized":"2024-09-13T14:41:41.476856+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:11.594853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486798+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:41:41.486798+0000","last_peered":"2024-09-13T14:41:41.486798+0000","last_clean":"2024-09-13T14:41:41.486798+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:41:41.486798+0000","last_undegraded":"2024-09-13T14:41:41.486798+0000","last_fullsized":"2024-09-13T14:41:41.486798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481946+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:41:41.481946+0000","last_peered":"2024-09-13T14:41:41.481946+0000","last_clean":"2024-09-13T14:41:41.481946+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:41:41.481946+0000","last_undegraded":"2024-09-13T14:41:41.481946+0000","last_fullsized":"2024-09-13T14:41:41.481946+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":675,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480619+0000","last_change":"2024-09-13T14:41:09.650053+0000","last_active":"2024-09-13T14:41:41.480619+0000","last_peered":"2024-09-13T14:41:41.480619+0000","last_clean":"2024-09-13T14:41:41.480619+0000","last_became_active":"2024-09-13T14:41:09.649530+0000","last_became_peered":"2024-09-13T14:41:09.649530+0000","last_unstale":"2024-09-13T14:41:41.480619+0000","last_undegraded":"2024-09-13T14:41:41.480619+0000","last_fullsized":"2024-09-13T14:41:41.480619+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:39.045795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":55,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479822+0000","last_change":"2024-09-13T14:28:14.320314+0000","last_active":"2024-09-13T14:41:41.479822+0000","last_peered":"2024-09-13T14:41:41.479822+0000","last_clean":"2024-09-13T14:41:41.479822+0000","last_became_active":"2024-09-13T14:28:14.319934+0000","last_became_peered":"2024-09-13T14:28:14.319934+0000","last_unstale":"2024-09-13T14:41:41.479822+0000","last_undegraded":"2024-09-13T14:41:41.479822+0000","last_fullsized":"2024-09-13T14:41:41.479822+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:21.514353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480732+0000","last_change":"2024-09-13T14:40:36.870296+0000","last_active":"2024-09-13T14:41:41.480732+0000","last_peered":"2024-09-13T14:41:41.480732+0000","last_clean":"2024-09-13T14:41:41.480732+0000","last_became_active":"2024-09-13T14:40:36.869969+0000","last_became_peered":"2024-09-13T14:40:36.869969+0000","last_unstale":"2024-09-13T14:41:41.480732+0000","last_undegraded":"2024-09-13T14:41:41.480732+0000","last_fullsized":"2024-09-13T14:41:41.480732+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:57.021118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":78,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.481406+0000","last_change":"2024-09-13T14:28:14.293329+0000","last_active":"2024-09-13T14:41:41.481406+0000","last_peered":"2024-09-13T14:41:41.481406+0000","last_clean":"2024-09-13T14:41:41.481406+0000","last_became_active":"2024-09-13T14:28:14.293075+0000","last_became_peered":"2024-09-13T14:28:14.293075+0000","last_unstale":"2024-09-13T14:41:41.481406+0000","last_undegraded":"2024-09-13T14:41:41.481406+0000","last_fullsized":"2024-09-13T14:41:41.481406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:20.310761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.476555+0000","last_change":"2024-09-13T14:41:09.650180+0000","last_active":"2024-09-13T14:41:41.476555+0000","last_peered":"2024-09-13T14:41:41.476555+0000","last_clean":"2024-09-13T14:41:41.476555+0000","last_became_active":"2024-09-13T14:41:09.649908+0000","last_became_peered":"2024-09-13T14:41:09.649908+0000","last_unstale":"2024-09-13T14:41:41.476555+0000","last_undegraded":"2024-09-13T14:41:41.476555+0000","last_fullsized":"2024-09-13T14:41:41.476555+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:08.029267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490320+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:41:41.490320+0000","last_peered":"2024-09-13T14:41:41.490320+0000","last_clean":"2024-09-13T14:41:41.490320+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:41:41.490320+0000","last_undegraded":"2024-09-13T14:41:41.490320+0000","last_fullsized":"2024-09-13T14:41:41.490320+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":71,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484873+0000","last_change":"2024-09-13T14:40:36.879188+0000","last_active":"2024-09-13T14:41:41.484873+0000","last_peered":"2024-09-13T14:41:41.484873+0000","last_clean":"2024-09-13T14:41:41.484873+0000","last_became_active":"2024-09-13T14:40:36.878788+0000","last_became_peered":"2024-09-13T14:40:36.878788+0000","last_unstale":"2024-09-13T14:41:41.484873+0000","last_undegraded":"2024-09-13T14:41:41.484873+0000","last_fullsized":"2024-09-13T14:41:41.484873+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:27:11.826299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.484913+0000","last_change":"2024-09-13T14:41:41.484817+0000","last_active":"2024-09-13T14:41:41.484913+0000","last_peered":"2024-09-13T14:41:41.484913+0000","last_clean":"2024-09-13T14:41:41.484913+0000","last_became_active":"2024-09-13T14:41:41.484505+0000","last_became_peered":"2024-09-13T14:41:41.484505+0000","last_unstale":"2024-09-13T14:41:41.484913+0000","last_undegraded":"2024-09-13T14:41:41.484913+0000","last_fullsized":"2024-09-13T14:41:41.484913+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:49.618334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.496242+0000","last_change":"2024-09-13T14:41:41.491895+0000","last_active":"2024-09-13T14:41:41.496242+0000","last_peered":"2024-09-13T14:41:41.496242+0000","last_clean":"2024-09-13T14:41:41.496242+0000","last_became_active":"2024-09-13T14:41:41.491517+0000","last_became_peered":"2024-09-13T14:41:41.491517+0000","last_unstale":"2024-09-13T14:41:41.496242+0000","last_undegraded":"2024-09-13T14:41:41.496242+0000","last_fullsized":"2024-09-13T14:41:41.496242+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:08.848118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":74,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490024+0000","last_change":"2024-09-13T14:41:41.489947+0000","last_active":"2024-09-13T14:41:41.490024+0000","last_peered":"2024-09-13T14:41:41.490024+0000","last_clean":"2024-09-13T14:41:41.490024+0000","last_became_active":"2024-09-13T14:41:41.487936+0000","last_became_peered":"2024-09-13T14:41:41.487936+0000","last_unstale":"2024-09-13T14:41:41.490024+0000","last_undegraded":"2024-09-13T14:41:41.490024+0000","last_fullsized":"2024-09-13T14:41:41.490024+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.480513+0000","last_change":"2024-09-13T14:40:36.875850+0000","last_active":"2024-09-13T14:41:41.480513+0000","last_peered":"2024-09-13T14:41:41.480513+0000","last_clean":"2024-09-13T14:41:41.480513+0000","last_became_active":"2024-09-13T14:40:36.875550+0000","last_became_peered":"2024-09-13T14:40:36.875550+0000","last_unstale":"2024-09-13T14:41:41.480513+0000","last_undegraded":"2024-09-13T14:41:41.480513+0000","last_fullsized":"2024-09-13T14:41:41.480513+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:57.875071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.492293+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:41:41.492293+0000","last_peered":"2024-09-13T14:41:41.492293+0000","last_clean":"2024-09-13T14:41:41.492293+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:41:41.492293+0000","last_undegraded":"2024-09-13T14:41:41.492293+0000","last_fullsized":"2024-09-13T14:41:41.492293+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":64,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490839+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:41:41.490839+0000","last_peered":"2024-09-13T14:41:41.490839+0000","last_clean":"2024-09-13T14:41:41.490839+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:41:41.490839+0000","last_undegraded":"2024-09-13T14:41:41.490839+0000","last_fullsized":"2024-09-13T14:41:41.490839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":54,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486110+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:41:41.486110+0000","last_peered":"2024-09-13T14:41:41.486110+0000","last_clean":"2024-09-13T14:41:41.486110+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:41:41.486110+0000","last_undegraded":"2024-09-13T14:41:41.486110+0000","last_fullsized":"2024-09-13T14:41:41.486110+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.493149+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:41:41.493149+0000","last_peered":"2024-09-13T14:41:41.493149+0000","last_clean":"2024-09-13T14:41:41.493149+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:41:41.493149+0000","last_undegraded":"2024-09-13T14:41:41.493149+0000","last_fullsized":"2024-09-13T14:41:41.493149+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.479047+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:41:40.479047+0000","last_peered":"2024-09-13T14:41:40.479047+0000","last_clean":"2024-09-13T14:41:40.479047+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:41:40.479047+0000","last_undegraded":"2024-09-13T14:41:40.479047+0000","last_fullsized":"2024-09-13T14:41:40.479047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490987+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:41:41.490987+0000","last_peered":"2024-09-13T14:41:41.490987+0000","last_clean":"2024-09-13T14:41:41.490987+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:41:41.490987+0000","last_undegraded":"2024-09-13T14:41:41.490987+0000","last_fullsized":"2024-09-13T14:41:41.490987+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482609+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:41:41.482609+0000","last_peered":"2024-09-13T14:41:41.482609+0000","last_clean":"2024-09-13T14:41:41.482609+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:41:41.482609+0000","last_undegraded":"2024-09-13T14:41:41.482609+0000","last_fullsized":"2024-09-13T14:41:41.482609+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490102+0000","last_change":"2024-09-13T14:41:41.490102+0000","last_active":"2024-09-13T14:41:41.490102+0000","last_peered":"2024-09-13T14:41:41.490102+0000","last_clean":"2024-09-13T14:41:41.490102+0000","last_became_active":"2024-09-13T14:41:41.489822+0000","last_became_peered":"2024-09-13T14:41:41.489822+0000","last_unstale":"2024-09-13T14:41:41.490102+0000","last_undegraded":"2024-09-13T14:41:41.490102+0000","last_fullsized":"2024-09-13T14:41:41.490102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:28.447149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.477814+0000","last_change":"2024-09-13T14:40:36.879235+0000","last_active":"2024-09-13T14:41:40.477814+0000","last_peered":"2024-09-13T14:41:40.477814+0000","last_clean":"2024-09-13T14:41:40.477814+0000","last_became_active":"2024-09-13T14:40:36.878874+0000","last_became_peered":"2024-09-13T14:40:36.878874+0000","last_unstale":"2024-09-13T14:41:40.477814+0000","last_undegraded":"2024-09-13T14:41:40.477814+0000","last_fullsized":"2024-09-13T14:41:40.477814+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:07.605097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.490555+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:41:41.490555+0000","last_peered":"2024-09-13T14:41:41.490555+0000","last_clean":"2024-09-13T14:41:41.490555+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:41:41.490555+0000","last_undegraded":"2024-09-13T14:41:41.490555+0000","last_fullsized":"2024-09-13T14:41:41.490555+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":57,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-13T14:41:40.478200+0000","last_change":"2024-09-13T14:28:14.297761+0000","last_active":"2024-09-13T14:41:40.478200+0000","last_peered":"2024-09-13T14:41:40.478200+0000","last_clean":"2024-09-13T14:41:40.478200+0000","last_became_active":"2024-09-13T14:28:14.297478+0000","last_became_peered":"2024-09-13T14:28:14.297478+0000","last_unstale":"2024-09-13T14:41:40.478200+0000","last_undegraded":"2024-09-13T14:41:40.478200+0000","last_fullsized":"2024-09-13T14:41:40.478200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:48.450673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.488366+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:41:41.488366+0000","last_peered":"2024-09-13T14:41:41.488366+0000","last_clean":"2024-09-13T14:41:41.488366+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:41:41.488366+0000","last_undegraded":"2024-09-13T14:41:41.488366+0000","last_fullsized":"2024-09-13T14:41:41.488366+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":81,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.486185+0000","last_change":"2024-09-13T14:41:41.486093+0000","last_active":"2024-09-13T14:41:41.486185+0000","last_peered":"2024-09-13T14:41:41.486185+0000","last_clean":"2024-09-13T14:41:41.486185+0000","last_became_active":"2024-09-13T14:41:41.485759+0000","last_became_peered":"2024-09-13T14:41:41.485759+0000","last_unstale":"2024-09-13T14:41:41.486185+0000","last_undegraded":"2024-09-13T14:41:41.486185+0000","last_fullsized":"2024-09-13T14:41:41.486185+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:05.424465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.479295+0000","last_change":"2024-09-13T14:40:36.876656+0000","last_active":"2024-09-13T14:41:41.479295+0000","last_peered":"2024-09-13T14:41:41.479295+0000","last_clean":"2024-09-13T14:41:41.479295+0000","last_became_active":"2024-09-13T14:40:36.876361+0000","last_became_peered":"2024-09-13T14:40:36.876361+0000","last_unstale":"2024-09-13T14:41:41.479295+0000","last_undegraded":"2024-09-13T14:41:41.479295+0000","last_fullsized":"2024-09-13T14:41:41.479295+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:51.843071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T14:41:41.482257+0000","last_change":"2024-09-13T14:41:09.653285+0000","last_active":"2024-09-13T14:41:41.482257+0000","last_peered":"2024-09-13T14:41:41.482257+0000","last_clean":"2024-09-13T14:41:41.482257+0000","last_became_active":"2024-09-13T14:41:09.652737+0000","last_became_peered":"2024-09-13T14:41:09.652737+0000","last_unstale":"2024-09-13T14:41:41.482257+0000","last_undegraded":"2024-09-13T14:41:41.482257+0000","last_fullsized":"2024-09-13T14:41:41.482257+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:13.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234185,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32892,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712260,"statfs":{"total":95995035648,"available":95961354240,"internally_reserved":0,"allocated":1818624,"data_stored":1215821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495821,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1806336,"data_stored":1215421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757458,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32304,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712848,"statfs":{"total":95995035648,"available":95961956352,"internally_reserved":0,"allocated":1208320,"data_stored":621210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019096,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1224704,"data_stored":625714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856467983,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32748,"kb_used_data":5272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93712404,"statfs":{"total":95995035648,"available":95961501696,"internally_reserved":0,"allocated":5398528,"data_stored":4815218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":28109988},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631510,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32808,"kb_used_data":5276,"kb_used_omap":8,"kb_used_meta":27511,"kb_avail":93712344,"statfs":{"total":95995035648,"available":95961440256,"internally_reserved":0,"allocated":5402624,"data_stored":4815744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":28172255},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795036,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1176,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1204224,"data_stored":620843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":28172914},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958563,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37548,"kb_used_data":5856,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93707604,"statfs":{"total":95995035648,"available":95956586496,"internally_reserved":0,"allocated":5996544,"data_stored":5406100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":32432615},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:42:51.273 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T14:42:51.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump 2024-09-13T14:42:51.472 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:51.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:51 smithi172 ceph-mon[100696]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:42:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:52 smithi154 ceph-mon[102314]: pgmap v282: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:52.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:52 smithi154 ceph-mon[107038]: pgmap v282: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:42:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:42:51] "GET /metrics HTTP/1.1" 200 33946 "" "Prometheus/2.51.0" 2024-09-13T14:42:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:52 smithi172 ceph-mon[100696]: pgmap v282: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:53.190 INFO:teuthology.orchestra.run.smithi154.stdout:epoch 109 2024-09-13T14:42:53.190 INFO:teuthology.orchestra.run.smithi154.stdout:fsid 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T14:42:53.190 INFO:teuthology.orchestra.run.smithi154.stdout:created 2024-09-13T14:21:30.909284+0000 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:modified 2024-09-13T14:41:41.473972+0000 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:crush_version 22 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:full_ratio 0.95 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:backfillfull_ratio 0.9 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:nearfull_ratio 0.85 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:require_min_compat_client luminous 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:min_compat_client luminous 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:require_osd_release reef 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:stretch_mode_enabled false 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-13T14:42:53.191 INFO:teuthology.orchestra.run.smithi154.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 85 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:max_osd 8 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0 up in weight 1 up_from 93 up_thru 108 down_at 90 last_clean_interval [8,89) [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643] [v2:172.21.15.154:6804/3729051643,v1:172.21.15.154:6805/3729051643] exists,up fa9860db-d569-4402-8a9e-c41b083f2042 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 up in weight 1 up_from 98 up_thru 108 down_at 95 last_clean_interval [13,94) [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834] [v2:172.21.15.154:6812/148745834,v1:172.21.15.154:6813/148745834] exists,up 539dd807-6e38-4237-8d23-28b57f0f1b4f 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 up in weight 1 up_from 103 up_thru 108 down_at 100 last_clean_interval [18,99) [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431] [v2:172.21.15.154:6820/2711669431,v1:172.21.15.154:6821/2711669431] exists,up ba93fa41-fa08-4d94-b499-149ded9151f0 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 up in weight 1 up_from 108 up_thru 108 down_at 105 last_clean_interval [25,104) [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861] [v2:172.21.15.154:6828/381867861,v1:172.21.15.154:6829/381867861] exists,up 4422727c-268e-4356-83fe-118d25e6cb4d 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4 up in weight 1 up_from 30 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689] [v2:172.21.15.172:6802/822621689,v1:172.21.15.172:6803/822621689] exists,up 9d3372f5-f7f2-4798-a221-5adc2acfc3c6 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5 up in weight 1 up_from 38 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433] [v2:172.21.15.172:6810/1992976433,v1:172.21.15.172:6811/1992976433] exists,up 6458ef05-8c2f-4ef0-a0a1-080908418a07 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 up in weight 1 up_from 46 up_thru 105 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288] [v2:172.21.15.172:6818/1801776288,v1:172.21.15.172:6819/1801776288] exists,up 8e768676-0ffa-46e3-8a03-db3fdde2951c 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7 up in weight 1 up_from 54 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567] [v2:172.21.15.172:6826/173465567,v1:172.21.15.172:6827/173465567] exists,up ef36cdfb-77d4-4974-aa46-d64d79a2b0e0 2024-09-13T14:42:53.192 INFO:teuthology.orchestra.run.smithi154.stdout:pg_upmap_items 4.7 [1,2] 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:pg_upmap_items 4.9 [1,2] 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:pg_upmap_items 4.1b [1,5] 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/2090383084 expires 2024-09-14T14:34:12.641827+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/1584830159 expires 2024-09-14T14:34:12.641827+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/2189531829 expires 2024-09-14T14:34:12.641827+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6801/548682894 expires 2024-09-14T14:34:12.641827+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6800/548682894 expires 2024-09-14T14:34:12.641827+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/3255496958 expires 2024-09-14T14:31:04.278426+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6801/544101962 expires 2024-09-14T14:26:57.118038+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/18164567 expires 2024-09-14T14:31:04.278426+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6800/544101962 expires 2024-09-14T14:26:57.118038+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:0/4103428848 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.193 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:0/3191030600 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/1349885828 expires 2024-09-14T14:26:57.118038+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/4127949890 expires 2024-09-14T14:22:00.624052+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6801/302860313 expires 2024-09-14T14:31:04.278426+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:0/213038927 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/3680637464 expires 2024-09-14T14:34:12.641827+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/2810096914 expires 2024-09-14T14:22:23.331946+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/3579365666 expires 2024-09-14T14:22:00.624052+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/966915265 expires 2024-09-14T14:22:23.331946+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6800/302860313 expires 2024-09-14T14:31:04.278426+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6801/557082616 expires 2024-09-14T14:22:00.624052+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:0/2607163146 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/490371939 expires 2024-09-14T14:22:00.624052+0000 2024-09-13T14:42:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/1379294727 expires 2024-09-14T14:31:04.278426+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:0/2289953927 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/3482226595 expires 2024-09-14T14:26:57.118038+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/106598025 expires 2024-09-14T14:26:57.118038+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/3924873105 expires 2024-09-14T14:31:04.278426+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/3142124210 expires 2024-09-14T14:26:57.118038+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:0/3573827628 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6800/3662635163 expires 2024-09-14T14:22:23.331946+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6801/3662635163 expires 2024-09-14T14:22:23.331946+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:6834/3129749965 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:0/1231123258 expires 2024-09-14T14:22:23.331946+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.154:6800/557082616 expires 2024-09-14T14:22:00.624052+0000 2024-09-13T14:42:53.195 INFO:teuthology.orchestra.run.smithi154.stdout:blocklist 172.21.15.172:6835/3129749965 expires 2024-09-14T14:31:33.289382+0000 2024-09-13T14:42:53.874 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 93 up_thru 108 down_at 90 last_clean_interval [8,89) [v2:172.21.15.154:6802/3729051643,v1:172.21.15.154:6803/3729051643] [v2:172.21.15.154:6804/3729051643,v1:172.21.15.154:6805/3729051643] exists,up fa9860db-d569-4402-8a9e-c41b083f2042', 'osd.1 up in weight 1 up_from 98 up_thru 108 down_at 95 last_clean_interval [13,94) [v2:172.21.15.154:6810/148745834,v1:172.21.15.154:6811/148745834] [v2:172.21.15.154:6812/148745834,v1:172.21.15.154:6813/148745834] exists,up 539dd807-6e38-4237-8d23-28b57f0f1b4f', 'osd.2 up in weight 1 up_from 103 up_thru 108 down_at 100 last_clean_interval [18,99) [v2:172.21.15.154:6818/2711669431,v1:172.21.15.154:6819/2711669431] [v2:172.21.15.154:6820/2711669431,v1:172.21.15.154:6821/2711669431] exists,up ba93fa41-fa08-4d94-b499-149ded9151f0', 'osd.3 up in weight 1 up_from 108 up_thru 108 down_at 105 last_clean_interval [25,104) [v2:172.21.15.154:6826/381867861,v1:172.21.15.154:6827/381867861] [v2:172.21.15.154:6828/381867861,v1:172.21.15.154:6829/381867861] exists,up 4422727c-268e-4356-83fe-118d25e6cb4d', 'osd.4 up in weight 1 up_from 30 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/822621689,v1:172.21.15.172:6801/822621689] [v2:172.21.15.172:6802/822621689,v1:172.21.15.172:6803/822621689] exists,up 9d3372f5-f7f2-4798-a221-5adc2acfc3c6', 'osd.5 up in weight 1 up_from 38 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/1992976433,v1:172.21.15.172:6809/1992976433] [v2:172.21.15.172:6810/1992976433,v1:172.21.15.172:6811/1992976433] exists,up 6458ef05-8c2f-4ef0-a0a1-080908418a07', 'osd.6 up in weight 1 up_from 46 up_thru 105 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6816/1801776288,v1:172.21.15.172:6817/1801776288] [v2:172.21.15.172:6818/1801776288,v1:172.21.15.172:6819/1801776288] exists,up 8e768676-0ffa-46e3-8a03-db3fdde2951c', 'osd.7 up in weight 1 up_from 54 up_thru 108 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6824/173465567,v1:172.21.15.172:6825/173465567] [v2:172.21.15.172:6826/173465567,v1:172.21.15.172:6827/173465567] exists,up ef36cdfb-77d4-4974-aa46-d64d79a2b0e0'] 2024-09-13T14:42:53.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon mon.a config show 2024-09-13T14:42:54.090 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:54.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:54 smithi154 ceph-mon[102314]: pgmap v283: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:54.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1119739301' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:42:54.365 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:54 smithi154 ceph-mon[107038]: pgmap v283: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:54.365 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1119739301' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:42:54.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:54 smithi172 ceph-mon[100696]: pgmap v283: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1119739301' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-13T14:42:55.666 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:42:55.666 INFO:teuthology.orchestra.run.smithi154.stdout: "name": "mon.a", 2024-09-13T14:42:55.666 INFO:teuthology.orchestra.run.smithi154.stdout: "cluster": "ceph", 2024-09-13T14:42:55.666 INFO:teuthology.orchestra.run.smithi154.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T14:42:55.666 INFO:teuthology.orchestra.run.smithi154.stdout: "admin_socket_mode": "", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_client_required": "cephx, none", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_cluster_required": "cephx", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_debug": "true", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_service_required": "cephx", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "auth_supported": "", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_aio": "true", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T14:42:55.667 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_aio_reap_max": "16", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_async_discard_threads": "0", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_block_size": "4096", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_debug_aio": "false", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_enable_discard": "false", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_flock_retry": "3", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T14:42:55.668 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_inject_crash": "0", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_ioring": "false", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_ioring_hipri": "false", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bdev_type": "", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_allocator": "hybrid", 2024-09-13T14:42:55.669 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_buffered_io": "true", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T14:42:55.670 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_replay_recovery": "false", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_sync_write": "false", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_allocator": "hybrid", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T14:42:55.671 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_create": "true", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_db_create": "false", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_db_path": "", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_db_size": "0", 2024-09-13T14:42:55.672 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_path": "", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_size": "107374182400", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_wal_create": "false", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_wal_path": "", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bluefs": "true", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:42:55.673 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_autotune": "true", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_size": "0", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T14:42:55.674 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_type": "2q", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_clone_cow": "true", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_mode": "none", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T14:42:55.675 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_freelist": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T14:42:55.676 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T14:42:55.677 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fail_eio": "false", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T14:42:55.678 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T14:42:55.679 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_max_blob_size": "0", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T14:42:55.680 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T14:42:55.681 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T14:42:55.681 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T14:42:55.681 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T14:42:55.681 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T14:42:55.681 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T14:42:55.681 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T14:42:55.682 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T14:42:55.682 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T14:42:55.682 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T14:42:55.682 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_spdk_mem": "512", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T14:42:55.683 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_tracing": "false", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T14:42:55.684 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_write_v2": "false", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_write_v2_random": "false", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T14:42:55.685 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T14:42:55.686 INFO:teuthology.orchestra.run.smithi154.stdout: "cephx_cluster_require_version": "2", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "cephx_require_signatures": "false", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "cephx_require_version": "2", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "cephx_service_require_signatures": "false", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "cephx_service_require_version": "2", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "cephx_sign_messages": "true", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "chdir": "", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "client_acl_type": "", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "client_asio_thread_count": "2", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "client_cache_mid": "0.750000", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "client_cache_size": "16384", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "client_caps_release_delay": "5", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "client_check_pool_perm": "true", 2024-09-13T14:42:55.687 INFO:teuthology.orchestra.run.smithi154.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_debug_force_sync_read": "false", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_debug_getattr_caps": "false", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_debug_inject_features": "", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_die_on_failed_remount": "false", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_dirsize_rbytes": "true", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_force_lazyio": "false", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_fs": "", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_inject_release_failure": "false", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_max_inline_size": "4096", 2024-09-13T14:42:55.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_mds_namespace": "", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_metadata": "", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_mount_gid": "-1", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_mount_timeout": "300", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_mount_uid": "-1", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_mountpoint": "/", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_notify_timeout": "10", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_oc": "true", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_oc_max_objects": "1000", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_oc_size": "209715200", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T14:42:55.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_permissions": "true", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_quota": "true", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_quota_df": "true", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_readahead_max_bytes": "0", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_readahead_max_periods": "4", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_readahead_min": "131072", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_reconnect_stale": "false", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_shutdown_timeout": "30", 2024-09-13T14:42:55.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_snapdir": ".snap", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_tick_interval": "1", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_trace": "", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_use_faked_inos": "false", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_use_random_mds": "false", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "clog_to_graylog": "false", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "clog_to_graylog_port": "12201", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "clog_to_monitors": "default=true", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "clog_to_syslog": "false", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T14:42:55.691 INFO:teuthology.orchestra.run.smithi154.stdout: "clog_to_syslog_level": "info", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "cluster_addr": "-", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "cluster_network": "", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "cluster_network_interface": "", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "compressor_zlib_isal": "false", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "compressor_zlib_level": "5", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "compressor_zstd_level": "1", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T14:42:55.692 INFO:teuthology.orchestra.run.smithi154.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crush_location": "", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crush_location_hook": "", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "crush_location_hook_timeout": "10", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "daemonize": "false", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "daos_pool": "tank", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T14:42:55.693 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_alienstore": "0/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_asok": "1/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_asok_assert_abort": "false", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_auth": "1/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_bdev": "1/3", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_bluefs": "1/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_bluestore": "1/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_buffer": "0/1", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T14:42:55.694 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_cephsqlite": "0/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_client": "0/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_compressor": "1/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_context": "0/1", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_crush": "1/1", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_crypto": "1/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_cyanstore": "0/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_dpdk": "1/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_eventtrace": "1/5", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_filer": "0/1", 2024-09-13T14:42:55.695 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_filestore": "1/3", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_finisher": "1/1", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_fuse": "1/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_javaclient": "1/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_journal": "1/3", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_journaler": "0/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_kstore": "1/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_lockdep": "0/1", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mclock": "1/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mds": "1/5", 2024-09-13T14:42:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mds_balancer": "1/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mds_locker": "1/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mds_log": "1/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mds_migrator": "1/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_memstore": "1/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mgr": "2/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mgrc": "1/5", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_mon": "20/20", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_monc": "0/10", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_ms": "1/1", 2024-09-13T14:42:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_none": "0/5", 2024-09-13T14:42:55.698 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_objclass": "0/5", 2024-09-13T14:42:55.698 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_objectcacher": "0/5", 2024-09-13T14:42:55.698 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_objecter": "0/1", 2024-09-13T14:42:55.698 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_optracker": "0/5", 2024-09-13T14:42:55.698 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_osd": "1/5", 2024-09-13T14:42:55.698 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_paxos": "20/20", 2024-09-13T14:42:55.698 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_perfcounter": "1/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_prioritycache": "1/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rados": "0/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rbd": "0/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rbd_replay": "0/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_refs": "0/0", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_reserver": "1/1", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw": "1/5", 2024-09-13T14:42:55.699 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw_access": "1/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw_flight": "1/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw_notification": "1/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rgw_sync": "1/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_rocksdb": "4/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore": "0/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_backref": "0/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_cache": "0/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_device": "0/5", 2024-09-13T14:42:55.700 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_epm": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_journal": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_lba": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_odata": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_omap": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_onode": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_t": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_seastore_tm": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_striper": "0/1", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_test": "0/5", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_throttle": "1/1", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_timer": "0/1", 2024-09-13T14:42:55.701 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_tp": "0/5", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "debug_trace": "1/5", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "device_failure_prediction_mode": "none", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "err_to_graylog": "false", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "err_to_journald": "false", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "err_to_stderr": "true", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "err_to_syslog": "false", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event_tracing": "false", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_cert_file": "", 2024-09-13T14:42:55.702 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_http_port": "9926", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_key_file": "", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_prio_limit": "5", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_sort_metrics": "true", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "exporter_stats_period": "5", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "fake_statfs_for_testing": "0", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "fatal_signal_handlers": "true", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "filer_max_purge_ops": "10", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "filer_max_truncate_ops": "128", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_blackhole": "false", 2024-09-13T14:42:55.703 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_btrfs_snap": "true", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_caller_concurrency": "10", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_debug_omap_check": "false", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_debug_verify_split": "false", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_dump_file": "", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_fadvise": "true", 2024-09-13T14:42:55.704 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_fail_eio": "true", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_fd_cache_size": "128", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_fiemap": "false", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_inject_stall": "0", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_journal_parallel": "false", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_journal_trailing": "false", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_journal_writeahead": "false", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_kill_at": "0", 2024-09-13T14:42:55.705 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T14:42:55.706 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_merge_threshold": "-10", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_odsync_write": "false", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_omap_backend_path": "", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_op_threads": "2", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_punch_hole": "false", 2024-09-13T14:42:55.707 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_queue_max_ops": "50", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_seek_data_hole": "false", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_sloppy_crc": "false", 2024-09-13T14:42:55.708 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_splice": "false", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_split_multiple": "2", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_split_rand_factor": "20", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_update_to": "1000", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T14:42:55.709 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_xfs_extsize": "false", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "filestore_zfs_snap": "false", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "fio_dir": "/tmp/fio", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "fsid": "447b0cd2-71db-11ef-bceb-c7b262605968", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_allow_other": "true", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_big_writes": "true", 2024-09-13T14:42:55.710 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_debug": "false", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_default_permissions": "false", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_disable_pagecache": "false", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_max_write": "0", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_multithreaded": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_require_active_mds": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_set_user_groups": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_splice_move": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_splice_read": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_splice_write": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "gss_target_name": "ceph", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "heartbeat_file": "", 2024-09-13T14:42:55.711 INFO:teuthology.orchestra.run.smithi154.stdout: "heartbeat_inject_failure": "0", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "heartbeat_interval": "5", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "host": "smithi154", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T14:42:55.712 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "inject_early_sigterm": "false", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "jaeger_agent_port": "6799", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "jaeger_tracing_enable": "false", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_aio": "true", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_align_min_size": "65536", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_block_align": "true", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_block_size": "4096", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_dio": "true", 2024-09-13T14:42:55.713 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_discard": "false", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_force_aio": "false", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_ignore_corruption": "false", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_max_write_entries": "100", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_replay_from": "0", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_write_header_frequency": "0", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journal_zero_on_create": "false", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journaler_prefetch_periods": "10", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journaler_prezero_periods": "5", 2024-09-13T14:42:55.714 INFO:teuthology.orchestra.run.smithi154.stdout: "journaler_write_head_interval": "15", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "key": "", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "keyfile": "", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_backend": "rocksdb", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_max_bytes": "67108864", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_max_ops": "512", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_onode_map_size": "1024", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T14:42:55.715 INFO:teuthology.orchestra.run.smithi154.stdout: "kstore_sync_transaction": "false", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "librados_thread_count": "2", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "lockdep": "false", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "lockdep_force_backtrace": "false", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_coarse_timestamps": "true", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_flush_on_exit": "false", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_graylog_port": "12201", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_max_new": "1000", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_max_recent": "10000", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_stderr_prefix": "", 2024-09-13T14:42:55.716 INFO:teuthology.orchestra.run.smithi154.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "log_to_file": "true", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "log_to_graylog": "false", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "log_to_journald": "false", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "log_to_stderr": "false", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "log_to_syslog": "false", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_action_on_write_error": "1", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_alternate_name_max": "8192", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_asio_thread_count": "2", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_export_pin": "true", 2024-09-13T14:42:55.717 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_interval": "10", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_max": "-1", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_max_until": "-1", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_merge_size": "50", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_mode": "0", 2024-09-13T14:42:55.718 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_split_bits": "3", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_split_size": "10000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T14:42:55.719 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_mid": "0.700000", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_trim_interval": "1", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T14:42:55.720 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_debug_auth_pins": "false", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_debug_frag": "false", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_debug_scatterstat": "false", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_debug_subtrees": "false", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T14:42:55.721 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_default_dir_hash": "2", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_defer_session_stale": "true", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dir_max_entries": "0", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dir_prefetch": "true", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_early_reply": "true", 2024-09-13T14:42:55.722 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_enable_op_tracker": "true", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_enforce_unique_name": "true", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T14:42:55.723 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_inject_health_dummy": "false", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_join_fs": "", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_journal_format": "1", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_create_at": "0", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T14:42:55.724 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_export_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_import_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_journal_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_link_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_openc_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_rename_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T14:42:55.725 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_max_events": "-1", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_max_segments": "128", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_pause": "false", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_segment_size": "0", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_trim_threshold": "128", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_completed_requests": "100000", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_export_size": "20971520", 2024-09-13T14:42:55.726 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_file_recover": "32", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_purge_files": "64", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_purge_ops": "8192", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_metrics_update_interval": "2", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_min_caps_per_client": "100", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_numa_node": "-1", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T14:42:55.727 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_op_history_duration": "600", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_op_history_size": "20", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_op_log_threshold": "5", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_ping_grace": "15", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_ping_interval": "5", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_recall_max_caps": "30000", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T14:42:55.728 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_replay_interval": "1.000000", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_root_ino_gid": "0", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_root_ino_uid": "0", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T14:42:55.729 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_shutdown_check": "0", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_skip_ino": "0", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_snap_min_uid": "0", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_snap_rstat": "false", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T14:42:55.730 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_symlink_recovery": "true", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_thrash_exports": "0", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_thrash_fragments": "0", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_tick_interval": "5.000000", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_valgrind_exit": "false", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_verify_backtrace": "1", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_verify_scatter": "false", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_wipe_sessions": "false", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool_debug": "false", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "memstore_page_set": "false", 2024-09-13T14:42:55.731 INFO:teuthology.orchestra.run.smithi154.stdout: "memstore_page_size": "65536", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_client_bytes": "134217728", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_client_messages": "512", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_disabled_modules": "", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_mds_messages": "128", 2024-09-13T14:42:55.732 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_mon_messages": "128", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_op_history_duration": "600", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_op_history_size": "20", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_op_log_threshold": "5", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T14:42:55.733 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_osd_messages": "8192", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_pool": "true", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_standby_modules": "true", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_stats_period": "5", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_stats_threshold": "5", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_tick_period": "2", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_allow_pool_delete": "true", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_allow_pool_size_one": "false", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T14:42:55.734 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_bytes": "104857600", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_directed_command_retry": "2", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T14:42:55.735 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_client_target_rank": "-1", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_level": "debug", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_journald": "true", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T14:42:55.736 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_compact_on_start": "false", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_compact_on_trim": "true", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_cpu_threads": "4", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_data_avail_crit": "5", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_data_avail_warn": "5", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T14:42:55.737 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_dump_json": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_extra_checks": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_down_mkfs_grace": "60", 2024-09-13T14:42:55.738 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_down_uptime_grace": "60", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_election_default_strategy": "1", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_election_timeout": "5.000000", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_enable_op_tracker": "true", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_fake_pool_delete": "false", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_force_quorum_join": "false", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_health_detail_to_clog": "true", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_health_log_update_period": "5", 2024-09-13T14:42:55.739 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_health_max_detail": "50", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_health_to_clog": "true", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_host": "[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0]", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_host_override": "", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_initial_members": "", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_lease": "5.000000", 2024-09-13T14:42:55.740 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_log_full_interval": "50", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_log_max": "10000", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_log_max_summary": "50", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_log_epochs": "500", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_osd": "10000", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T14:42:55.741 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mds_skip_sanity": "true", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_memory_autotune": "true", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_memory_target": "2147483648", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mgr_digest_period": "5", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T14:42:55.742 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_min_osdmap_epochs": "500", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_op_complaint_time": "30", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_op_history_duration": "600", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_op_history_size": "20", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_op_log_threshold": "5", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T14:42:55.743 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_cache_size": "500", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T14:42:55.744 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T14:42:55.745 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T14:42:55.746 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T14:42:55.746 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T14:42:55.746 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T14:42:55.746 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_report_timeout": "900", 2024-09-13T14:42:55.746 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T14:42:55.747 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T14:42:55.747 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T14:42:55.748 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_reweight_max_osds": "4", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_scrub_interval": "86400", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_scrub_max_keys": "100", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_scrub_timeout": "300", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_session_timeout": "300", 2024-09-13T14:42:55.749 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_smart_report_timeout": "5", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_stretch_pool_size": "4", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_sync_debug": "false", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T14:42:55.750 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_tick_interval": "5", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T14:42:55.751 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_older_version": "true", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T14:42:55.752 INFO:teuthology.orchestra.run.smithi154.stdout: "monmap": "", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "motr_tracing_enabled": "false", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_op_threads": "3", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_cm": "false", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_device_name": "", 2024-09-13T14:42:55.753 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_sl": "3", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_rdma_type": "ib", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_async_reap_threshold": "5", 2024-09-13T14:42:55.754 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_before_connect": "false", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_ipv4": "true", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_ipv6": "false", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_msgr1": "true", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_msgr2": "true", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_port_max": "7568", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_port_min": "6800", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_retry_count": "3", 2024-09-13T14:42:55.755 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_bind_retry_delay": "5", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_blackhole_client": "false", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_blackhole_mds": "false", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_blackhole_mgr": "false", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_blackhole_mon": "false", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_blackhole_osd": "false", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_client_mode": "crc secure", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_cluster_type": "", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_compress_secure": "false", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T14:42:55.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_crc_data": "true", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_crc_header": "true", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_die_on_bug": "true", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_die_on_old_message": "true", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T14:42:55.757 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_hugepages": "", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_lro": "true", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_pmd": "", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_port_id": "0", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_dump_on_send": "false", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T14:42:55.758 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_inject_delay_type": "", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_inject_network_congestion": "0", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_inject_socket_failures": "0", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_max_accept_failures": "4", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_max_backoff": "15.000000", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T14:42:55.759 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_osd_compress_mode": "none", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_pq_min_cost": "65536", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_public_type": "", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_service_mode": "crc secure", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_tcp_nodelay": "true", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "ms_type": "async+posix", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "no_config_file": "false", 2024-09-13T14:42:55.760 INFO:teuthology.orchestra.run.smithi154.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_inflight_ops": "1024", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter_timeout": "10.000000", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objectstore_blackhole": "false", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T14:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "openssl_engine_opts": "", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_agent_max_ops": "4", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_agent_slop": "0.020000", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_asio_thread_count": "2", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T14:42:55.762 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_backfill_scan_max": "512", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_backfill_scan_min": "64", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_backoff_on_peering": "false", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_beacon_report_interval": "300", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bench_duration": "30", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_blkin_trace_all": "false", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T14:42:55.763 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_class_update_on_start": "true", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_client_message_cap": "256", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_client_op_priority": "63", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_client_watch_timeout": "30", 2024-09-13T14:42:55.764 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_command_max_records": "256", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_compact_on_start": "false", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_crush_update_on_start": "true", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T14:42:55.765 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_op_order": "false", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T14:42:55.766 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_shutdown": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_trim_objects": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T14:42:55.767 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_default_notify_timeout": "30", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_ec_partial_reads": "true", 2024-09-13T14:42:55.768 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_enable_op_tracker": "true", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_fast_info": "true", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_fast_shutdown": "true", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T14:42:55.769 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_function_tracing": "false", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_heartbeat_grace": "20", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_heartbeat_interval": "6", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_heartbeat_stale": "600", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T14:42:55.770 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_journal_size": "5120", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_kill_backfill_at": "0", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_map_cache_size": "50", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_map_dedup": "true", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_map_message_max": "40", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T14:42:55.771 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_attr_name_len": "100", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_attr_size": "0", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_backfills": "1", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_markdown_count": "5", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_markdown_period": "600", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_object_name_len": "2048", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_object_size": "134217728", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_pg_log_entries": "10000", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_pgls": "1024", 2024-09-13T14:42:55.772 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_push_cost": "8388608", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_push_objects": "10", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_scrubs": "3", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_max_write_size": "90", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T14:42:55.773 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_profile": "balanced", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T14:42:55.774 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_memory_base": "805306368", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_memory_target": "4294967296", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_memory_target_autotune": "false", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_min_pg_log_entries": "250", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_min_recovery_priority": "0", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T14:42:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mon_report_interval": "5", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_num_cache_shards": "32", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_numa_node": "-1", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_objecter_finishers": "1", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_objectstore": "bluestore", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_objectstore_fuse": "false", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T14:42:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_objectstore_tracing": "false", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_history_duration": "600", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_history_size": "20", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_log_threshold": "5", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_num_shards": "0", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T14:42:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_op_thread_timeout": "15", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_open_classes_on_start": "true", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_os_flags": "0", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_peering_op_priority": "255", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_delete_priority": "5", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T14:42:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_log_trim_min": "100", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_crimson": "false", 2024-09-13T14:42:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_flags": "0", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_min_size": "0", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T14:42:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_size": "3", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_default_type": "replicated", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_cost": "20971520", 2024-09-13T14:42:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_max_active": "0", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_op_priority": "3", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_priority": "5", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T14:42:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_repair_during_recovery": "false", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T14:42:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_cost": "52428800", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_end_hour": "0", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T14:42:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_priority": "5", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_skip_data_digest": "false", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_smart_report_timeout": "5", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_snap_trim_priority": "5", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T14:42:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T14:42:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T14:42:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T14:42:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T14:42:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_target_transaction_size": "30", 2024-09-13T14:42:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T14:42:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_tracing": "false", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_use_stale_snap": "false", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T14:42:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_kill_at": "0", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_max_join_drift": "10", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_min": "500", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_min_wait": "0.050000", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_service_trim_max": "500", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_service_trim_min": "250", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_stash_full_interval": "25", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_trim_max": "500", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "paxos_trim_min": "250", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "perf": "true", 2024-09-13T14:42:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "pid_file": "", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "public_addr": "-", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "public_addrv": "", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "public_bind_addr": "-", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "public_network": "172.21.0.0/20", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "public_network_interface": "", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "qat_compressor_enabled": "false", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T14:42:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "rados_mon_op_timeout": "0", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rados_osd_op_timeout": "0", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rados_tracing": "false", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_atime_update_interval": "60", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T14:42:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache": "true", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_policy": "writearound", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_size": "33554432", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_compression_hint": "none", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T14:42:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_clone_format": "auto", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_data_pool": "", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_features": "61", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_format": "2", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_map_options": "", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_order": "22", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_pool": "rbd", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_stripe_count": "0", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T14:42:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_io_scheduler": "simple", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_order": "24", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_pool": "", 2024-09-13T14:42:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_journal_splay_width": "4", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T14:42:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T14:42:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_op_threads": "1", 2024-09-13T14:42:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_plugins": "", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T14:42:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T14:42:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_tracing": "false", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_validate_names": "true", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "rbd_validate_pool": "true", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "restapi_base_url": "", 2024-09-13T14:42:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "restapi_log_level": "", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_admin_entry": "admin", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_backend_store": "rados", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_barbican_url": "", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_beast_enable_async": "true", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T14:42:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_cache_enabled": "true", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_config_store": "rados", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_content_length_compat": "false", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T14:42:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_cross_domain_policy": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T14:42:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T14:42:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T14:42:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T14:42:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data_log_window": "30", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T14:42:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_disable_s3select": "false", 2024-09-13T14:42:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T14:42:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dns_name": "", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dns_s3website_name": "", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_ops_log": "false", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_static_website": "false", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enable_usage_log": "false", 2024-09-13T14:42:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_expose_bucket": "false", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_extended_http_attrs": "", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_filter": "none", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_max_objs": "32", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T14:42:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_init_timeout": "300", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T14:42:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_password": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_project": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_token": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_admin_user": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T14:42:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_url": "", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T14:42:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lc_max_objs": "32", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lc_max_worker": "3", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ldap_uri": "", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T14:42:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_log_http_headers": "", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_attr_size": "0", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T14:42:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_listing_results": "1000", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_notify_retries": "10", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T14:42:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T14:42:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_num_control_oids": "8", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_numa_node": "-1", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T14:42:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_op_tracing": "false", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_opa_token": "", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_opa_url": "", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ops_log_rados": "false", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T14:42:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_print_continue": "true", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T14:42:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_rados_tracing": "false", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_realm": "", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_realm_id": "", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_region": "", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T14:42:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_request_uri": "", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_resolve_cname": "false", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_run_sync_thread": "true", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T14:42:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_script_uri": "", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_service_provider_name": "", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sts_client_id": "", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sts_client_secret": "", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sts_entry": "sts", 2024-09-13T14:42:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sts_key": "sts", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_auth_url": "", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_custom_header": "", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_need_stats": "true", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_tenant_name": "", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_url": "", 2024-09-13T14:42:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_lease_period": "120", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T14:42:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_thread_pool_size": "512", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_comment": "", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_createby": "", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_encoding": "", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_flag": "false", 2024-09-13T14:42:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_origin": "", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_torrent_tracker": "", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_usage_max_shards": "32", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_use_opa_authz": "false", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_counters_cache": "false", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T14:42:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_user_unique_email": "true", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_verify_ssl": "true", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_zone": "", 2024-09-13T14:42:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_zone_id": "", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_zonegroup": "", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_zonegroup_id": "", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_block_size": "4096", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T14:42:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_partition_filters": "false", 2024-09-13T14:42:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_perf": "false", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "run_dir": "/var/run/ceph", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_block_create": "true", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_device_size": "53687091200", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T14:42:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_full_integrity_check": "false", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_main_device_type": "SSD", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "seastore_segment_size": "67108864", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "set_keepcaps": "false", 2024-09-13T14:42:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "setgroup": "ceph", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "setuser": "ceph", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "setuser_match_path": "", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "thp": "false", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "threadpool_default_timeout": "60", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "throttler_perf_counter": "true", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "tmp_dir": "/tmp", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "uadk_compressor_enabled": "false", 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T14:42:55.833 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:42:56.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:56 smithi154 ceph-mon[102314]: pgmap v284: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:56.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:56 smithi154 ceph-mon[107038]: pgmap v284: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:56.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:56 smithi172 ceph-mon[100696]: pgmap v284: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:56.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T14:42:56.694 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:58.144 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:58 smithi154 ceph-mon[102314]: pgmap v285: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:58.144 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:58.144 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:58 smithi154 ceph-mon[107038]: pgmap v285: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:58.144 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:58.380 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a: {} 2024-09-13T14:42:58.380 INFO:teuthology.orchestra.run.smithi154.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:42:58.389 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b: {} 2024-09-13T14:42:58.390 INFO:teuthology.orchestra.run.smithi154.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:42:58.398 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c: {} 2024-09-13T14:42:58.398 INFO:teuthology.orchestra.run.smithi154.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T14:42:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:58 smithi172 ceph-mon[100696]: pgmap v285: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:42:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:42:58.980 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-13T14:42:58.981 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-13T14:42:58.982 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:42:58.984 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:42:58.984 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T14:42:58.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-13T14:42:58.985 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T14:42:58.985 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:42:58.985 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:42:58.985 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:42:58.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd out 5 2024-09-13T14:42:58.986 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T14:42:58.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:42:58.986 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T14:42:58.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:42:58.987 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T14:42:58.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:42:58.987 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T14:42:58.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:42:59.016 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:42:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:42:59.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:59.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:42:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:42:59.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:42:59 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4261569232' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:42:59.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:42:59 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4261569232' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:42:59.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:59.276 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:59.278 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:59.279 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:59.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:42:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:59.318 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:42:59.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:42:59.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:42:59 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4261569232' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T14:42:59.505 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:42:59.527 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:42:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:42:59.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:59.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:42:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:59.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:42:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:42:59.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:42:59.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:42:59.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:42:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:42:59.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:00.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:00.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:00.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:00.278 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:00.291 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:00 smithi154 ceph-mon[102314]: pgmap v286: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:00.291 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:00 smithi154 ceph-mon[107038]: pgmap v286: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:00.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.405 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:00 smithi172 ceph-mon[100696]: pgmap v286: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:00.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.533 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.653 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:00.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:00.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:00.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:00.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:00.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:00.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:01.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:01.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:01.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:01.159 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:01.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:01.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:01.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:01.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:01.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:01.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:01.533 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:01.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:01.659 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:01.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:01.785 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:01.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:01.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:01.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:02.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:02.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:02 smithi154 ceph-mon[102314]: pgmap v287: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:02.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:02 smithi154 ceph-mon[107038]: pgmap v287: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:43:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:43:01] "GET /metrics HTTP/1.1" 200 33945 "" "Prometheus/2.51.0" 2024-09-13T14:43:02.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:02.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:02.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:02.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:02 smithi172 ceph-mon[100696]: pgmap v287: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:02.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:02.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:02.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:02.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:02.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:02.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:02.909 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:02.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:03.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:03.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:03.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:03.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:03.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:03.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:03.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:03.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:03.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:03.539 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:03.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:03.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:03.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:03.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:03.794 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:03.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:03.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:03.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:03.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:04.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:04.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:04.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:04.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:04.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:04 smithi154 ceph-mon[102314]: pgmap v288: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:04.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:04 smithi154 ceph-mon[107038]: pgmap v288: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:04.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:04.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:04.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:04.421 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:04.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:04 smithi172 ceph-mon[100696]: pgmap v288: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:04.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:04.537 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:43:04.537 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:04.549 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:04.555 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:43:04.555 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:04.573 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:43:04.573 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T14:43:04.588 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:43:04.588 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T14:43:04.604 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:43:04.605 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:04.621 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:43:04.621 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:04.638 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:43:04.638 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:04.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:04.654 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:43:04.654 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:04.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:04.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.802 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:04.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:04.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:04.924 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:04.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:05.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:05.043 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:05.043 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:05.052 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:05.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:05.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:05.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:05 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2611190556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:05.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:05 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2611190556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:05.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:05.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:05.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:05.427 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:05.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:05 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2611190556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:05.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:05.556 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:05.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:05.683 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:05.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:05.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:05.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:05.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:05.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:06.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:06.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:06.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:06.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:06.318 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:06 smithi154 ceph-mon[102314]: pgmap v289: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:06.319 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:06 smithi154 ceph-mon[107038]: pgmap v289: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:06.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:06 smithi172 ceph-mon[100696]: pgmap v289: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:06.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:06.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:06.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.569 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:06.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:06.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.692 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:06.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:06.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:06.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:06.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:06.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:07.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:07.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:07.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:07.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:07.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:07.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:07.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:43:07.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:07.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:07.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:07.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:07.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.688 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:07.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:07.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.812 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:07.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:07.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:07.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:07.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:08.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:08.063 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_0' created 2024-09-13T14:43:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.065 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:08.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:08.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.188 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:08.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:08 smithi154 ceph-mon[102314]: pgmap v290: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2697248851' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:43:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:08 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:43:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:08 smithi154 ceph-mon[107038]: pgmap v290: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2697248851' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:43:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:08 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:43:08.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:08.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:08.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:08.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:08.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:08 smithi172 ceph-mon[100696]: pgmap v290: 73 pgs: 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2697248851' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:43:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:08 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-13T14:43:08.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:08.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:08.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:08.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:08.692 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:08 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T14:43:08.278+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e110 definitely_dead 0 2024-09-13T14:43:08.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T14:43:08.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.869 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:08.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:08.969 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:08.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:08.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:08.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:08.998 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:09.062 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:43:09.070 INFO:teuthology.orchestra.run.smithi154.stderr:marked out osd.5. 2024-09-13T14:43:09.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:09.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:09.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:09.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:09.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:09.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:09.411 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:43:09.411 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[102314]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/945690763' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2807536743' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[102314]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[107038]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/945690763' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:09.412 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2807536743' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:43:09.413 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:43:09.413 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:09 smithi154 ceph-mon[107038]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:43:09.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:09.487 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:09 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-13T14:43:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:09 smithi172 ceph-mon[100696]: osdmap e110: 8 total, 8 up, 8 in 2024-09-13T14:43:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:09 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/945690763' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:43:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:09 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2807536743' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:43:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:09 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-13T14:43:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:09 smithi172 ceph-mon[100696]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-13T14:43:09.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:09.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:09.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:09.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:09.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:09.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:09.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:09.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:09.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:10.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:10.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:10.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:10.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:10.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:10.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:10.446 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[102314]: pgmap v292: 89 pgs: 7 creating+peering, 9 unknown, 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:10.446 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[102314]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:43:10.446 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/945690763' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[102314]: osdmap e111: 8 total, 8 up, 7 in 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[102314]: osdmap e112: 8 total, 8 up, 7 in 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[107038]: pgmap v292: 89 pgs: 7 creating+peering, 9 unknown, 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[107038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/945690763' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[107038]: osdmap e111: 8 total, 8 up, 7 in 2024-09-13T14:43:10.447 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:10 smithi154 ceph-mon[107038]: osdmap e112: 8 total, 8 up, 7 in 2024-09-13T14:43:10.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:10.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:10 smithi172 ceph-mon[100696]: pgmap v292: 89 pgs: 7 creating+peering, 9 unknown, 73 active+clean; 4.6 MiB data, 256 MiB used, 715 GiB / 715 GiB avail 2024-09-13T14:43:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:10 smithi172 ceph-mon[100696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:43:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:10 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:10 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/945690763' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:43:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:10 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-13T14:43:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:10 smithi172 ceph-mon[100696]: osdmap e111: 8 total, 8 up, 7 in 2024-09-13T14:43:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:10 smithi172 ceph-mon[100696]: osdmap e112: 8 total, 8 up, 7 in 2024-09-13T14:43:10.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:10.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:10.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:10.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:10.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:10.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:10.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:10.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:10.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:10.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:11.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:11.094 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:11.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:11.217 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:11.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:11.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:11.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:11.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:11.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:11.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:11.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:11.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:11.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:11.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:11.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:11.938 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:11.960 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:11.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:11.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:11 smithi154 ceph-mon[102314]: osdmap e113: 8 total, 8 up, 7 in 2024-09-13T14:43:11.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:11 smithi154 ceph-mon[107038]: osdmap e113: 8 total, 8 up, 7 in 2024-09-13T14:43:11.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:43:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:43:11] "GET /metrics HTTP/1.1" 200 33945 "" "Prometheus/2.51.0" 2024-09-13T14:43:11.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:11 smithi172 ceph-mon[100696]: osdmap e113: 8 total, 8 up, 7 in 2024-09-13T14:43:12.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:12.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:12.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:12.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:12.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:12.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:12.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:12.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:12.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:12.576 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:12.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:12.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:12.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:12.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:12.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:12.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:12.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:43:12.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:12.938 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:12 smithi154 ceph-mon[102314]: pgmap v296: 89 pgs: 3 peering, 6 creating+peering, 7 unknown, 73 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 4 B/s, 1 objects/s recovering 2024-09-13T14:43:12.938 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:12 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:12.938 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:12 smithi154 ceph-mon[102314]: osdmap e114: 8 total, 8 up, 7 in 2024-09-13T14:43:12.938 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:12 smithi154 ceph-mon[107038]: pgmap v296: 89 pgs: 3 peering, 6 creating+peering, 7 unknown, 73 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 4 B/s, 1 objects/s recovering 2024-09-13T14:43:12.939 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:12 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:12.939 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:12 smithi154 ceph-mon[107038]: osdmap e114: 8 total, 8 up, 7 in 2024-09-13T14:43:12.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:12.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:12 smithi172 ceph-mon[100696]: pgmap v296: 89 pgs: 3 peering, 6 creating+peering, 7 unknown, 73 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 4 B/s, 1 objects/s recovering 2024-09-13T14:43:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:12 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:43:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:12 smithi172 ceph-mon[100696]: osdmap e114: 8 total, 8 up, 7 in 2024-09-13T14:43:13.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:13.087 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:13.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:13.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:13.258 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:43:13.259 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:43:13.259 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:43:13.259 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:43:13.259 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:43:13.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:13.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:13.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:13.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:13.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:13.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[102314]: osdmap e115: 8 total, 8 up, 7 in 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2334936463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[107038]: osdmap e115: 8 total, 8 up, 7 in 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:13.780 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2334936463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:13.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:13.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:13.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:13.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:13.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:13.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:13 smithi172 ceph-mon[100696]: osdmap e115: 8 total, 8 up, 7 in 2024-09-13T14:43:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2334936463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:14.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:14.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:14.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.208 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:14.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:14.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:14.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:43:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:14.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:14.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:14.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:14.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:14.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:14.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:14.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:14.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:14.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:14.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:14.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:14.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:14.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:14.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:14 smithi154 ceph-mon[102314]: pgmap v299: 89 pgs: 4 peering, 6 creating+peering, 79 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 161 B/s, 4 objects/s recovering 2024-09-13T14:43:14.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:14 smithi154 ceph-mon[107038]: pgmap v299: 89 pgs: 4 peering, 6 creating+peering, 79 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 161 B/s, 4 objects/s recovering 2024-09-13T14:43:14.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:14 smithi172 ceph-mon[100696]: pgmap v299: 89 pgs: 4 peering, 6 creating+peering, 79 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 161 B/s, 4 objects/s recovering 2024-09-13T14:43:15.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:15.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:15.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:15.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:15.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:15.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:15.374 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:43:15.374 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:43:15.374 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:43:15.374 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:43:15.374 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:43:15.375 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:43:15.375 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:43:15.375 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:43:15.376 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:15.376 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:15.376 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:15.376 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:15.376 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:15.377 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:15.377 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:15.377 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:15.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:15.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:15.470 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:43:15.471 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:43:15.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:15.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:15.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:15.695 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T14:43:15.710 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:15.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:15.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:15.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:15.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:15.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:15.969 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4105032562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:15.969 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:15 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:15.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4105032562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:15.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:15 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:15.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4105032562' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:15.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:15 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:43:16.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:16.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:16.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:16.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:16.201 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:16.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:16.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:16.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:16.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:16.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:16.619 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:16.620 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:16.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:16.705 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:16.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:16.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:16.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:16.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:16.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[102314]: pgmap v300: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 131 B/s, 4 objects/s recovering 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[102314]: osdmap e116: 8 total, 8 up, 7 in 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[107038]: pgmap v300: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 131 B/s, 4 objects/s recovering 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:43:16.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:16 smithi154 ceph-mon[107038]: osdmap e116: 8 total, 8 up, 7 in 2024-09-13T14:43:16.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:16 smithi172 ceph-mon[100696]: pgmap v300: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 131 B/s, 4 objects/s recovering 2024-09-13T14:43:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:16 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:43:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:16 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:43:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:16 smithi172 ceph-mon[100696]: osdmap e116: 8 total, 8 up, 7 in 2024-09-13T14:43:17.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:17.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:17.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:17.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:17.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:17.335 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:17.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:17.446 DEBUG:teuthology.orchestra.run.smithi154:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-13T14:43:17.446 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-13T14:43:17.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:43:17.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:17.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:17.461 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:43:17.461 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:43:17.513 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:hints = 1 2024-09-13T14:43:17.513 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:43:17.513 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object prefix: benchmark_data_smithi154_144140 2024-09-13T14:43:17.514 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:43:17.514 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:43:17.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:17.592 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:17.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:17.728 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:17.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:17.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:17.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:17.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:18.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:18.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:18.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:18.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:17 smithi154 ceph-mon[102314]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:43:18.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:17 smithi154 ceph-mon[107038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:43:18.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:17 smithi172 ceph-mon[100696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:43:18.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:18.279 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:18.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:18.410 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:18.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:18.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 1 16 3726 3710 231.828 231.875 0.00455187 0.00430372 2024-09-13T14:43:18.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:18.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:43:18.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:18.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:18.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:18.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:18.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:18.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:18.890 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:18.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:18.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:18 smithi154 ceph-mon[102314]: pgmap v302: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 108 B/s, 2 objects/s recovering 2024-09-13T14:43:18.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:18 smithi154 ceph-mon[107038]: pgmap v302: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 108 B/s, 2 objects/s recovering 2024-09-13T14:43:19.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:19.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:19.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:19.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:18 smithi172 ceph-mon[100696]: pgmap v302: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 225 MiB used, 626 GiB / 626 GiB avail; 108 B/s, 2 objects/s recovering 2024-09-13T14:43:19.241 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:43:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=cleanup t=2024-09-13T14:43:18.830970915Z level=info msg="Completed cleanup jobs" duration=21.501247ms 2024-09-13T14:43:19.241 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:43:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=plugins.update.checker t=2024-09-13T14:43:18.943179098Z level=info msg="Update check succeeded" duration=57.855415ms 2024-09-13T14:43:19.241 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:43:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana.update.checker t=2024-09-13T14:43:18.957223684Z level=info msg="Update check succeeded" duration=61.219823ms 2024-09-13T14:43:19.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:19.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:19.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:19.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:19.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:19.514 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 2 16 6940 6924 216.343 200.875 0.00478831 0.00461381 2024-09-13T14:43:19.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:19.590 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:19.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:19.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:19.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:19.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:19.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:19.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:20.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:20.200 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:43:20.200 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:94 5.3 5:c6caeca6:::benchmark_data_smithi154_144140_object93:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:17.542958+0000", 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.4111872480000001, 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012329343, 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.201 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 94 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.542958+0000", 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.542958+0000", 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.542959+0000", 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5899999999999995e-07 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.542980+0000", 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0862000000000001e-05 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.542990+0000", 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0057e-05 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.542998+0000", 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5920000000000003e-06 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.543183+0000", 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185303 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.543295+0000", 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011171900000000001 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.543448+0000", 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015369799999999999 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.544140+0000", 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069148199999999999 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.554953+0000", 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010813348 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555193+0000", 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024009500000000001 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555250+0000", 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6990999999999997e-05 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555288+0000", 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7336999999999998e-05 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:104 5.3 5:c5b250e0:::benchmark_data_smithi154_144140_object103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:17.545697+0000", 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.408448618, 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097693680000000005, 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 104 2024-09-13T14:43:20.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.545697+0000", 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.545697+0000", 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.545698+0000", 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1100000000000004e-07 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.545721+0000", 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3390000000000001e-05 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.545732+0000", 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.064e-05 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.545739+0000", 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1799999999999999e-06 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.545795+0000", 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5488000000000001e-05 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.546049+0000", 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000253904 2024-09-13T14:43:20.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.546181+0000", 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013250299999999999 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.546902+0000", 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000720617 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555092+0000", 2024-09-13T14:43:20.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081906619999999996 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555371+0000", 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000278466 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555422+0000", 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0797000000000002e-05 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555466+0000", 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.481e-05 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:111 5.3 5:c0b7fb28:::benchmark_data_smithi154_144140_object110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:17.547690+0000", 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.4064558200000001, 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091970650000000008, 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.214 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 111 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.547690+0000", 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.547690+0000", 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.547691+0000", 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.435e-06 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.547716+0000", 2024-09-13T14:43:20.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4346999999999999e-05 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.547727+0000", 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1766999999999999e-05 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.547912+0000", 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018437 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.547976+0000", 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4076000000000001e-05 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.548098+0000", 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121728 2024-09-13T14:43:20.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.548226+0000", 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000128045 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.548845+0000", 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00061950000000000004 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.555142+0000", 2024-09-13T14:43:20.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062972480000000001 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.556767+0000", 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016247919999999999 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.556831+0000", 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4107000000000001e-05 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.556887+0000", 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5649999999999997e-05 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:822 5.3 5:c67bfb30:::benchmark_data_smithi154_144140_object821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:17.746107+0000", 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.2080390479999998, 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092314029999999991, 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.221 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 822 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746107+0000", 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746107+0000", 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746107+0000", 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4799999999999997e-07 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746129+0000", 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1246999999999998e-05 2024-09-13T14:43:20.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746139+0000", 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0224000000000001e-05 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746146+0000", 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7460000000000003e-06 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746449+0000", 2024-09-13T14:43:20.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030350499999999997 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746557+0000", 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108083 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.746686+0000", 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000128789 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.748555+0000", 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001869075 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.748676+0000", 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121098 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.755144+0000", 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064676430000000004 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.755292+0000", 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014809099999999999 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.755338+0000", 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6054000000000003e-05 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:1730 5.5 5:ab9241e6:::benchmark_data_smithi154_144140_object1729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:17.984547+0000", 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.9695986539999999, 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090545810000000008, 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1730 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.228 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984547+0000", 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984547+0000", 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984549+0000", 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5650000000000001e-06 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984571+0000", 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2263999999999999e-05 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984585+0000", 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4278e-05 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984593+0000", 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1450000000000008e-06 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984660+0000", 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6369000000000001e-05 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984780+0000", 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120019 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.984924+0000", 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000144869 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.986371+0000", 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00144663 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.993111+0000", 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067402620000000003 2024-09-13T14:43:20.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.993488+0000", 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037665099999999998 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.993559+0000", 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1222000000000004e-05 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:17.993602+0000", 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2307e-05 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:3355 5.5 5:af6b000a:::benchmark_data_smithi154_144140_object3354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:18.405981+0000", 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.5481646529999999, 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010101384999999999, 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.234 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3355 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.405981+0000", 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.405981+0000", 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.405983+0000", 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9069999999999999e-06 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406012+0000", 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8850999999999999e-05 2024-09-13T14:43:20.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406027+0000", 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5160000000000001e-05 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406035+0000", 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1780000000000003e-06 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406221+0000", 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018572700000000001 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406334+0000", 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112865 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.407083+0000", 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000749469 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.408947+0000", 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00186349 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.408987+0000", 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0754999999999999e-05 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.415969+0000", 2024-09-13T14:43:20.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069813260000000004 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.416040+0000", 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1707999999999999e-05 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.416082+0000", 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1949000000000001e-05 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:3357 5.5 5:a69a0722:::benchmark_data_smithi154_144140_object3356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:18.406058+0000", 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.5480877479999999, 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010276197000000001, 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.241 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3357 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406058+0000", 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406058+0000", 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406058+0000", 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8899999999999999e-07 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406081+0000", 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2991999999999999e-05 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406090+0000", 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4239999999999993e-06 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.406096+0000", 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8919999999999997e-06 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.408211+0000", 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021154540000000001 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.408359+0000", 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014734200000000001 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.408486+0000", 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000127224 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.409089+0000", 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060317699999999999 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.410652+0000", 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001562744 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.416122+0000", 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054706520000000003 2024-09-13T14:43:20.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.416294+0000", 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000171223 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.416334+0000", 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0484000000000001e-05 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:3404 5.5 5:a1e67161:::benchmark_data_smithi154_144140_object3403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:18.421715+0000", 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.53243025, 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010353427, 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3404 2024-09-13T14:43:20.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.249 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.421715+0000", 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.421715+0000", 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.421717+0000", 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4389999999999999e-06 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.421742+0000", 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5326000000000001e-05 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.421791+0000", 2024-09-13T14:43:20.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8741e-05 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.421801+0000", 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0499999999999999e-05 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.422179+0000", 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037771099999999998 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.422314+0000", 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013500400000000001 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.423807+0000", 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001493093 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.424708+0000", 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090058399999999995 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.424919+0000", 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000211213 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.431980+0000", 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070608099999999998 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.432030+0000", 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0090000000000003e-05 2024-09-13T14:43:20.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.432069+0000", 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8915999999999998e-05 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:4768 5.3 5:c081aa4a:::benchmark_data_smithi154_144140_object4767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:18.845126+0000", 2024-09-13T14:43:20.258 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.1090192130000001, 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097929969999999995, 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4768 2024-09-13T14:43:20.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845126+0000", 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845126+0000", 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845128+0000", 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.637e-06 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845152+0000", 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4122999999999999e-05 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845165+0000", 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3006999999999999e-05 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845172+0000", 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1010000000000003e-06 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845610+0000", 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043736699999999997 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845716+0000", 2024-09-13T14:43:20.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106467 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.845841+0000", 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012450100000000001 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.848740+0000", 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028990349999999999 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.851716+0000", 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029766469999999998 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.854759+0000", 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030429570000000002 2024-09-13T14:43:20.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.854871+0000", 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011194200000000001 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.854919+0000", 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8213000000000001e-05 2024-09-13T14:43:20.265 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:4785 5.5 5:acfb1bd7:::benchmark_data_smithi154_144140_object4784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:18.849126+0000", 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.1050195039999999, 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009231085, 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4785 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849126+0000", 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849126+0000", 2024-09-13T14:43:20.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849127+0000", 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2799999999999995e-07 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849146+0000", 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9018e-05 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849162+0000", 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6201999999999998e-05 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849250+0000", 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8318000000000004e-05 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849433+0000", 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018242099999999999 2024-09-13T14:43:20.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849544+0000", 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111125 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.849669+0000", 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000125289 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.851434+0000", 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017646129999999999 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.852014+0000", 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058027000000000003 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.857312+0000", 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052978569999999996 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.857415+0000", 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010249 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:18.858357+0000", 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00094265400000000002 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.272 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:5795 5.e e116/111 5:72319f0f:::benchmark_data_smithi154_144140_object5794:head v 116'367, mlcod=116'365)", 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.165525+0000", 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.78862104799999999, 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011345210999999999, 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.165525+0000", 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.165525+0000", 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.165525+0000", 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0999999999999998e-07 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.165561+0000", 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5712e-05 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.165571+0000", 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7499999999999998e-06 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.165577+0000", 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0870000000000003e-06 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.168595+0000", 2024-09-13T14:43:20.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003018658 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.168615+0000", 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9247000000000001e-05 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.176816+0000", 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008201135 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.176870+0000", 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4011999999999997e-05 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:5806 5.3 5:cda44f9e:::benchmark_data_smithi154_144140_object5805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.166815+0000", 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.787330745, 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011558288999999999, 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.277 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5806 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.166815+0000", 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.166815+0000", 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.166817+0000", 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1390000000000002e-06 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.166841+0000", 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3572999999999999e-05 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.166869+0000", 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8309999999999998e-05 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.166875+0000", 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9739999999999999e-06 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.166929+0000", 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4261000000000001e-05 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167067+0000", 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137723 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167255+0000", 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000187987 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.168821+0000", 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015658079999999999 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.171937+0000", 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031166259999999999 2024-09-13T14:43:20.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.178210+0000", 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062726630000000004 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.178327+0000", 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011723299999999999 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.178373+0000", 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5992000000000003e-05 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:5803 5.2 e116/111 5:4975f4b3:::benchmark_data_smithi154_144140_object5802:head v 116'371, mlcod=116'369)", 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.167375+0000", 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.78677079100000002, 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097178079999999997, 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.284 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167375+0000", 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167375+0000", 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167375+0000", 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4799999999999999e-07 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167405+0000", 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9618e-05 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167412+0000", 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1230000000000002e-06 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167417+0000", 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1869999999999996e-06 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167468+0000", 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0461999999999999e-05 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167488+0000", 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9953999999999999e-05 2024-09-13T14:43:20.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.177039+0000", 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095519369999999999 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.177093+0000", 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3179e-05 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:5804 5.2 e116/111 5:483ce05e:::benchmark_data_smithi154_144140_object5803:head v 116'372, mlcod=116'369)", 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.167882+0000", 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.78626328599999995, 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092395510000000004, 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167882+0000", 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167882+0000", 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167883+0000", 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5599999999999995e-07 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167919+0000", 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6128999999999999e-05 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167928+0000", 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1519999999999998e-06 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.167934+0000", 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1750000000000002e-06 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.170334+0000", 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023998510000000002 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.171683+0000", 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013488009999999999 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.177097+0000", 2024-09-13T14:43:20.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054137999999999999 2024-09-13T14:43:20.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.177122+0000", 2024-09-13T14:43:20.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5086999999999999e-05 2024-09-13T14:43:20.293 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:6253 5.3 5:c60d8ab0:::benchmark_data_smithi154_144140_object6252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.310740+0000", 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.643405809, 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090582679999999995, 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.294 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6253 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.310740+0000", 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.310740+0000", 2024-09-13T14:43:20.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.310741+0000", 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6400000000000001e-07 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.310960+0000", 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021912 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.311083+0000", 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012299199999999999 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.311091+0000", 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5580000000000001e-06 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.311537+0000", 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00044614399999999999 2024-09-13T14:43:20.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.312917+0000", 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001379496 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.313130+0000", 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021356599999999999 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.317515+0000", 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043843629999999996 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.317548+0000", 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3129000000000001e-05 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.319650+0000", 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021020380000000001 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.319747+0000", 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6698000000000002e-05 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.319798+0000", 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1400000000000003e-05 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.300 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:6642 5.5 5:af2efe02:::benchmark_data_smithi154_144140_object6641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.421211+0000", 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.53293509800000005, 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010893185999999999, 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6642 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.421211+0000", 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.421211+0000", 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.421211+0000", 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9899999999999999e-07 2024-09-13T14:43:20.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.421233+0000", 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1916000000000002e-05 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.421247+0000", 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3379e-05 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.421254+0000", 2024-09-13T14:43:20.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0700000000000001e-06 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.423615+0000", 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002361513 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.424721+0000", 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011057700000000001 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.424853+0000", 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013226000000000001 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.427060+0000", 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022071130000000001 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.430297+0000", 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003236896 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.432011+0000", 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017134030000000001 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.432065+0000", 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4549999999999998e-05 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.432104+0000", 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8516999999999997e-05 2024-09-13T14:43:20.306 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:6693 5.5 5:a572bd1a:::benchmark_data_smithi154_144140_object6692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.442696+0000", 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.51145007600000003, 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011128195, 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.307 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6693 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.442696+0000", 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.442696+0000", 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.442697+0000", 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1489999999999999e-06 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.442724+0000", 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7208000000000001e-05 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.442733+0000", 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2539999999999998e-06 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.442739+0000", 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1680000000000001e-06 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.443012+0000", 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000273019 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.443130+0000", 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117508 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.443255+0000", 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012486600000000001 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.445158+0000", 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001903451 2024-09-13T14:43:20.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.445625+0000", 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046649700000000002 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.453631+0000", 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080058349999999993 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.453685+0000", 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4851000000000001e-05 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.453824+0000", 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013838899999999999 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:7851 5.3 5:cc2e62dd:::benchmark_data_smithi154_144140_object7850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.790902+0000", 2024-09-13T14:43:20.313 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.16324344199999999, 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00975525, 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7851 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.790902+0000", 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.790902+0000", 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.790903+0000", 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4499999999999997e-07 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.790924+0000", 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1288999999999999e-05 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.790935+0000", 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.042e-05 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.790941+0000", 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8199999999999999e-06 2024-09-13T14:43:20.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.792258+0000", 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013166849999999999 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.792372+0000", 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113527 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.792584+0000", 2024-09-13T14:43:20.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021281499999999999 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.795708+0000", 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031236599999999999 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.796052+0000", 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034379300000000001 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.800476+0000", 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044245020000000003 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.800616+0000", 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013949000000000001 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.800657+0000", 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1603999999999997e-05 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.319 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:8062 5.3 5:c4d68c72:::benchmark_data_smithi154_144140_object8061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.864851+0000", 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.089294178000000002, 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092297579999999994, 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.320 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8062 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.864851+0000", 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.864851+0000", 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.864852+0000", 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2200000000000003e-07 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.864881+0000", 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8303999999999999e-05 2024-09-13T14:43:20.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.864895+0000", 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4367e-05 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.864902+0000", 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8800000000000002e-06 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.866776+0000", 2024-09-13T14:43:20.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018742520000000001 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.866924+0000", 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014760500000000001 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.867058+0000", 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013391799999999999 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.869648+0000", 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025900049999999998 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.870442+0000", 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00079473900000000004 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.873960+0000", 2024-09-13T14:43:20.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035176790000000001 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.874028+0000", 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.874081+0000", 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2768e-05 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:8232 5.5 5:ab99ce58:::benchmark_data_smithi154_144140_object8231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:19.929432+0000", 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.024713708000000001, 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090572580000000003, 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8232 2024-09-13T14:43:20.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.929432+0000", 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.929432+0000", 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.929433+0000", 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.421e-06 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.929457+0000", 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3547999999999999e-05 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.929466+0000", 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2869999999999993e-06 2024-09-13T14:43:20.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.929473+0000", 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3430000000000001e-06 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.930037+0000", 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056489599999999995 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.930163+0000", 2024-09-13T14:43:20.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000125957 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.930291+0000", 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012726400000000001 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.933070+0000", 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027789289999999999 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.936462+0000", 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033922150000000001 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.938398+0000", 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019364549999999999 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.938452+0000", 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4184999999999997e-05 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:19.938489+0000", 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6758000000000003e-05 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:20.333 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:43:20.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:20.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.514 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 3 16 9924 9908 206.392 186.5 0.00257592 0.00483644 2024-09-13T14:43:20.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:20.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:20.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:20.761 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:43:20.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:20.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:20.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:20.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:21.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:21.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:21.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:21.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:21.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:20 smithi154 ceph-mon[102314]: pgmap v303: 89 pgs: 89 active+clean; 120 MiB data, 458 MiB used, 625 GiB / 626 GiB avail; 16 MiB/s wr, 259 op/s; 91 B/s, 2 objects/s recovering 2024-09-13T14:43:21.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1146816026' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:21.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:21.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:20 smithi154 ceph-mon[107038]: pgmap v303: 89 pgs: 89 active+clean; 120 MiB data, 458 MiB used, 625 GiB / 626 GiB avail; 16 MiB/s wr, 259 op/s; 91 B/s, 2 objects/s recovering 2024-09-13T14:43:21.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1146816026' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:21.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:21.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:20 smithi172 ceph-mon[100696]: pgmap v303: 89 pgs: 89 active+clean; 120 MiB data, 458 MiB used, 625 GiB / 626 GiB avail; 16 MiB/s wr, 259 op/s; 91 B/s, 2 objects/s recovering 2024-09-13T14:43:21.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1146816026' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:21.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:21.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:21.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:21.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:21.514 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 4 16 13013 12997 203.056 193.062 0.00591802 0.00491717 2024-09-13T14:43:21.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:21.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:21.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:21.715 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:21.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:21.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:21.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:21.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:21.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:22.011 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:21 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:22.011 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:21 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:22.011 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:21 smithi154 ceph-mon[102314]: osdmap e117: 8 total, 8 up, 7 in 2024-09-13T14:43:22.011 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:43:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:43:21] "GET /metrics HTTP/1.1" 200 36453 "" "Prometheus/2.51.0" 2024-09-13T14:43:22.011 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:21 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:22.011 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:21 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:22.011 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:21 smithi154 ceph-mon[107038]: osdmap e117: 8 total, 8 up, 7 in 2024-09-13T14:43:22.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:22.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:22.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:22.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:21 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:22.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:21 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:22.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:21 smithi172 ceph-mon[100696]: osdmap e117: 8 total, 8 up, 7 in 2024-09-13T14:43:22.245 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:22.272 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:22.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:22.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:22.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:22.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:22.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 5 16 15559 15543 194.267 159.125 0.00307217 0.00513934 2024-09-13T14:43:22.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:22.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:22.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:22.788 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:22.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:22.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:22.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:23.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:23.046 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:22 smithi154 ceph-mon[102314]: pgmap v305: 89 pgs: 89 active+clean; 211 MiB data, 665 MiB used, 625 GiB / 626 GiB avail; 26 MiB/s wr, 412 op/s; 8 B/s, 0 objects/s recovering 2024-09-13T14:43:23.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:22 smithi154 ceph-mon[107038]: pgmap v305: 89 pgs: 89 active+clean; 211 MiB data, 665 MiB used, 625 GiB / 626 GiB avail; 26 MiB/s wr, 412 op/s; 8 B/s, 0 objects/s recovering 2024-09-13T14:43:23.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:23.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:23.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:23.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:22 smithi172 ceph-mon[100696]: pgmap v305: 89 pgs: 89 active+clean; 211 MiB data, 665 MiB used, 625 GiB / 626 GiB avail; 26 MiB/s wr, 412 op/s; 8 B/s, 0 objects/s recovering 2024-09-13T14:43:23.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:23.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:23.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:23.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:23.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 6 15 17978 17963 187.095 151.25 0.00689219 0.00533744 2024-09-13T14:43:23.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:23.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:23.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:23.731 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:23.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:23.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:23.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:23.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:24.011 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:24.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:24.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:24.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:24.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:24.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:24.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:24.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:24.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:43:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:24.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 7 16 20433 20417 182.271 153.375 0.00194305 0.00547851 2024-09-13T14:43:24.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:24.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:24.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:24.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:24.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:24.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:24.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:24.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:25.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:25.068 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:24 smithi154 ceph-mon[102314]: pgmap v306: 89 pgs: 89 active+clean; 619 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:43:25.068 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:24 smithi154 ceph-mon[107038]: pgmap v306: 89 pgs: 89 active+clean; 619 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:43:25.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:25.164 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:25.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:24 smithi172 ceph-mon[100696]: pgmap v306: 89 pgs: 89 active+clean; 619 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:43:25.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:25.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:25.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:25.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:25.510 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:43:25.512 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:43:25.514 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 8 16 22771 22755 177.751 146.125 0.00540823 0.00561906 2024-09-13T14:43:25.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:25.562 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:25.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:25.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:25.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:25.826 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:25.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:25.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:25.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:26.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:26.094 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:25 smithi154 ceph-mon[102314]: pgmap v307: 89 pgs: 89 active+clean; 999 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:43:26.094 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:25 smithi154 ceph-mon[102314]: from='client.44555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:26.094 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:25 smithi154 ceph-mon[107038]: pgmap v307: 89 pgs: 89 active+clean; 999 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:43:26.094 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:25 smithi154 ceph-mon[107038]: from='client.44555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:26.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:26.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:26.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:25 smithi172 ceph-mon[100696]: pgmap v307: 89 pgs: 89 active+clean; 999 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:43:26.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:25 smithi172 ceph-mon[100696]: from='client.44555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:43:26.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:26.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:26.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:26.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:26.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 9 16 24816 24800 172.201 127.812 0.00601379 0.00580112 2024-09-13T14:43:26.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:26.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:26.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:26.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:43:26.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:26.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:26.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:27.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:27.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:27.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:27.165 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:27.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:27.297 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:27.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:27.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:27.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 10 16 26839 26823 167.624 126.438 0.00392663 0.0059606 2024-09-13T14:43:27.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:27.554 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:27.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:27.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:27.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:27.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:27.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:27.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:27.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:27.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:28.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:28.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:28.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:28.213 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:28.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:27 smithi154 ceph-mon[102314]: pgmap v308: 89 pgs: 89 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-13T14:43:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:27 smithi154 ceph-mon[107038]: pgmap v308: 89 pgs: 89 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-13T14:43:28.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:27 smithi172 ceph-mon[100696]: pgmap v308: 89 pgs: 89 active+clean; 1.2 GiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-13T14:43:28.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:28.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:28.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:28.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:28.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 11 16 28439 28423 161.475 100 0.00166084 0.00612268 2024-09-13T14:43:28.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:28.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:28.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:28.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:28.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:28.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:28.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:28.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:28.998 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:28.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:29.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:29.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:29.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:29.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:29.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:29.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:29.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:29.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:29.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:29.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:29.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:29.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 12 16 30563 30547 159.08 132.75 0.0141791 0.0062802 2024-09-13T14:43:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:29.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:29.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:29.669 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:29.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:29.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:29.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:29.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:29.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:30.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:30.060 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:30.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:30.187 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:30.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:29 smithi154 ceph-mon[102314]: pgmap v309: 89 pgs: 89 active+clean; 1.6 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:43:30.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:29 smithi154 ceph-mon[107038]: pgmap v309: 89 pgs: 89 active+clean; 1.6 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:43:30.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:29 smithi172 ceph-mon[100696]: pgmap v309: 89 pgs: 89 active+clean; 1.6 GiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-13T14:43:30.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:30.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:30.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:30.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:30.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 13 16 32396 32380 155.655 114.562 0.00828126 0.00641936 2024-09-13T14:43:30.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:30.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:30.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:30.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:30.802 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-13T14:43:24.785731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1047468005,"num_objects":15938,"num_object_clones":0,"num_object_copies":47814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15938,"num_whiteouts":0,"num_read":6333,"num_read_kb":13788,"num_write":17033,"num_write_kb":1033281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1771775,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17092,"ondisk_log_size":17092,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3092940,"kb_used_data":2860684,"kb_used_omap":32,"kb_used_meta":232159,"kb_avail":653123124,"statfs":{"total":671965249536,"available":668798078976,"internally_reserved":0,"allocated":2929340416,"data_stored":2924979487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33228,"internal_metadata":237731380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,6,1,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1042677760,"num_objects":15910,"num_object_clones":0,"num_object_copies":47730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15910,"num_write_kb":1018240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.092553"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.765074+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:43:20.765074+0000","last_peered":"2024-09-13T14:43:20.765074+0000","last_clean":"2024-09-13T14:43:20.765074+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:43:20.765074+0000","last_undegraded":"2024-09-13T14:43:20.765074+0000","last_fullsized":"2024-09-13T14:43:20.765074+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":81,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.678924+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:43:16.678924+0000","last_peered":"2024-09-13T14:43:16.678924+0000","last_clean":"2024-09-13T14:43:16.678924+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:43:16.678924+0000","last_undegraded":"2024-09-13T14:43:16.678924+0000","last_fullsized":"2024-09-13T14:43:16.678924+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679124+0000","last_change":"2024-09-13T14:43:09.684988+0000","last_active":"2024-09-13T14:43:16.679124+0000","last_peered":"2024-09-13T14:43:16.679124+0000","last_clean":"2024-09-13T14:43:16.679124+0000","last_became_active":"2024-09-13T14:43:09.684691+0000","last_became_peered":"2024-09-13T14:43:09.684691+0000","last_unstale":"2024-09-13T14:43:16.679124+0000","last_undegraded":"2024-09-13T14:43:16.679124+0000","last_fullsized":"2024-09-13T14:43:16.679124+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:28.021166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":80,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.768762+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:43:20.768762+0000","last_peered":"2024-09-13T14:43:20.768762+0000","last_clean":"2024-09-13T14:43:20.768762+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:43:20.768762+0000","last_undegraded":"2024-09-13T14:43:20.768762+0000","last_fullsized":"2024-09-13T14:43:20.768762+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":80,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769193+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:43:20.769193+0000","last_peered":"2024-09-13T14:43:20.769193+0000","last_clean":"2024-09-13T14:43:20.769193+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:43:20.769193+0000","last_undegraded":"2024-09-13T14:43:20.769193+0000","last_fullsized":"2024-09-13T14:43:20.769193+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679057+0000","last_change":"2024-09-13T14:28:14.292467+0000","last_active":"2024-09-13T14:43:16.679057+0000","last_peered":"2024-09-13T14:43:16.679057+0000","last_clean":"2024-09-13T14:43:16.679057+0000","last_became_active":"2024-09-13T14:28:14.292208+0000","last_became_peered":"2024-09-13T14:28:14.292208+0000","last_unstale":"2024-09-13T14:43:16.679057+0000","last_undegraded":"2024-09-13T14:43:16.679057+0000","last_fullsized":"2024-09-13T14:43:16.679057+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:51.254048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.765135+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:43:20.765135+0000","last_peered":"2024-09-13T14:43:20.765135+0000","last_clean":"2024-09-13T14:43:20.765135+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:43:20.765135+0000","last_undegraded":"2024-09-13T14:43:20.765135+0000","last_fullsized":"2024-09-13T14:43:20.765135+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:15.694563+0000","last_change":"2024-09-13T14:43:09.689757+0000","last_active":"2024-09-13T14:43:15.694563+0000","last_peered":"2024-09-13T14:43:15.694563+0000","last_clean":"2024-09-13T14:43:15.694563+0000","last_became_active":"2024-09-13T14:43:09.682005+0000","last_became_peered":"2024-09-13T14:43:09.682005+0000","last_unstale":"2024-09-13T14:43:15.694563+0000","last_undegraded":"2024-09-13T14:43:15.694563+0000","last_fullsized":"2024-09-13T14:43:15.694563+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:23.236222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"117'1129","reported_seq":1145,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:23.705825+0000","last_change":"2024-09-13T14:43:09.076822+0000","last_active":"2024-09-13T14:43:23.705825+0000","last_peered":"2024-09-13T14:43:23.705825+0000","last_clean":"2024-09-13T14:43:23.705825+0000","last_became_active":"2024-09-13T14:43:09.076561+0000","last_became_peered":"2024-09-13T14:43:09.076561+0000","last_unstale":"2024-09-13T14:43:23.705825+0000","last_undegraded":"2024-09-13T14:43:23.705825+0000","last_fullsized":"2024-09-13T14:43:23.705825+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1129,"log_dups_size":0,"ondisk_log_size":1129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:17.219135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73990144,"num_objects":1129,"num_object_clones":0,"num_object_copies":3387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1129,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":80,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679415+0000","last_change":"2024-09-13T14:40:04.164112+0000","last_active":"2024-09-13T14:43:16.679415+0000","last_peered":"2024-09-13T14:43:16.679415+0000","last_clean":"2024-09-13T14:43:16.679415+0000","last_became_active":"2024-09-13T14:40:04.163753+0000","last_became_peered":"2024-09-13T14:40:04.163753+0000","last_unstale":"2024-09-13T14:43:16.679415+0000","last_undegraded":"2024-09-13T14:43:16.679415+0000","last_fullsized":"2024-09-13T14:43:16.679415+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.685538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679025+0000","last_change":"2024-09-13T14:43:09.677449+0000","last_active":"2024-09-13T14:43:16.679025+0000","last_peered":"2024-09-13T14:43:16.679025+0000","last_clean":"2024-09-13T14:43:16.679025+0000","last_became_active":"2024-09-13T14:43:09.677093+0000","last_became_peered":"2024-09-13T14:43:09.677093+0000","last_unstale":"2024-09-13T14:43:16.679025+0000","last_undegraded":"2024-09-13T14:43:16.679025+0000","last_fullsized":"2024-09-13T14:43:16.679025+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:31:32.531793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"117'1019","reported_seq":1044,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:22.662259+0000","last_change":"2024-09-13T14:43:09.080838+0000","last_active":"2024-09-13T14:43:22.662259+0000","last_peered":"2024-09-13T14:43:22.662259+0000","last_clean":"2024-09-13T14:43:22.662259+0000","last_became_active":"2024-09-13T14:43:09.080537+0000","last_became_peered":"2024-09-13T14:43:09.080537+0000","last_unstale":"2024-09-13T14:43:22.662259+0000","last_undegraded":"2024-09-13T14:43:22.662259+0000","last_fullsized":"2024-09-13T14:43:22.662259+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:21.615492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66781184,"num_objects":1019,"num_object_clones":0,"num_object_copies":3057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1019,"num_write_kb":65216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":81,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679388+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:43:16.679388+0000","last_peered":"2024-09-13T14:43:16.679388+0000","last_clean":"2024-09-13T14:43:16.679388+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:43:16.679388+0000","last_undegraded":"2024-09-13T14:43:16.679388+0000","last_fullsized":"2024-09-13T14:43:16.679388+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":101,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571219+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:43:16.571219+0000","last_peered":"2024-09-13T14:43:16.571219+0000","last_clean":"2024-09-13T14:43:16.571219+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:43:16.571219+0000","last_undegraded":"2024-09-13T14:43:16.571219+0000","last_fullsized":"2024-09-13T14:43:16.571219+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"117'1028","reported_seq":1052,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:22.645342+0000","last_change":"2024-09-13T14:43:09.680819+0000","last_active":"2024-09-13T14:43:22.645342+0000","last_peered":"2024-09-13T14:43:22.645342+0000","last_clean":"2024-09-13T14:43:22.645342+0000","last_became_active":"2024-09-13T14:43:09.680530+0000","last_became_peered":"2024-09-13T14:43:09.680530+0000","last_unstale":"2024-09-13T14:43:22.645342+0000","last_undegraded":"2024-09-13T14:43:22.645342+0000","last_fullsized":"2024-09-13T14:43:22.645342+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1028,"log_dups_size":0,"ondisk_log_size":1028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:01.394980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67371008,"num_objects":1028,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":65792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679329+0000","last_change":"2024-09-13T14:43:09.693180+0000","last_active":"2024-09-13T14:43:16.679329+0000","last_peered":"2024-09-13T14:43:16.679329+0000","last_clean":"2024-09-13T14:43:16.679329+0000","last_became_active":"2024-09-13T14:43:09.684725+0000","last_became_peered":"2024-09-13T14:43:09.684725+0000","last_unstale":"2024-09-13T14:43:16.679329+0000","last_undegraded":"2024-09-13T14:43:16.679329+0000","last_fullsized":"2024-09-13T14:43:16.679329+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:10.926138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:15.694679+0000","last_change":"2024-09-13T14:40:36.880233+0000","last_active":"2024-09-13T14:43:15.694679+0000","last_peered":"2024-09-13T14:43:15.694679+0000","last_clean":"2024-09-13T14:43:15.694679+0000","last_became_active":"2024-09-13T14:40:36.879886+0000","last_became_peered":"2024-09-13T14:40:36.879886+0000","last_unstale":"2024-09-13T14:43:15.694679+0000","last_undegraded":"2024-09-13T14:43:15.694679+0000","last_fullsized":"2024-09-13T14:43:15.694679+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:49.693282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"117'988","reported_seq":1012,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:22.667313+0000","last_change":"2024-09-13T14:43:09.077347+0000","last_active":"2024-09-13T14:43:22.667313+0000","last_peered":"2024-09-13T14:43:22.667313+0000","last_clean":"2024-09-13T14:43:22.667313+0000","last_became_active":"2024-09-13T14:43:09.077065+0000","last_became_peered":"2024-09-13T14:43:09.077065+0000","last_unstale":"2024-09-13T14:43:22.667313+0000","last_undegraded":"2024-09-13T14:43:22.667313+0000","last_fullsized":"2024-09-13T14:43:22.667313+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":988,"log_dups_size":0,"ondisk_log_size":988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:32.339084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":63232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.765209+0000","last_change":"2024-09-13T14:43:09.680636+0000","last_active":"2024-09-13T14:43:20.765209+0000","last_peered":"2024-09-13T14:43:20.765209+0000","last_clean":"2024-09-13T14:43:20.765209+0000","last_became_active":"2024-09-13T14:43:09.680358+0000","last_became_peered":"2024-09-13T14:43:09.680358+0000","last_unstale":"2024-09-13T14:43:20.765209+0000","last_undegraded":"2024-09-13T14:43:20.765209+0000","last_fullsized":"2024-09-13T14:43:20.765209+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:48.519707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":80,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769242+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:43:20.769242+0000","last_peered":"2024-09-13T14:43:20.769242+0000","last_clean":"2024-09-13T14:43:20.769242+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:43:20.769242+0000","last_undegraded":"2024-09-13T14:43:20.769242+0000","last_fullsized":"2024-09-13T14:43:20.769242+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"117'959","reported_seq":982,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:22.663343+0000","last_change":"2024-09-13T14:43:09.679251+0000","last_active":"2024-09-13T14:43:22.663343+0000","last_peered":"2024-09-13T14:43:22.663343+0000","last_clean":"2024-09-13T14:43:22.663343+0000","last_became_active":"2024-09-13T14:43:09.678926+0000","last_became_peered":"2024-09-13T14:43:09.678926+0000","last_unstale":"2024-09-13T14:43:22.663343+0000","last_undegraded":"2024-09-13T14:43:22.663343+0000","last_fullsized":"2024-09-13T14:43:22.663343+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":959,"log_dups_size":0,"ondisk_log_size":959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:10.810144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62849024,"num_objects":959,"num_object_clones":0,"num_object_copies":2877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":959,"num_write_kb":61376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.765118+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:43:20.765118+0000","last_peered":"2024-09-13T14:43:20.765118+0000","last_clean":"2024-09-13T14:43:20.765118+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:43:20.765118+0000","last_undegraded":"2024-09-13T14:43:20.765118+0000","last_fullsized":"2024-09-13T14:43:20.765118+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571431+0000","last_change":"2024-09-13T14:41:41.491050+0000","last_active":"2024-09-13T14:43:16.571431+0000","last_peered":"2024-09-13T14:43:16.571431+0000","last_clean":"2024-09-13T14:43:16.571431+0000","last_became_active":"2024-09-13T14:41:41.490645+0000","last_became_peered":"2024-09-13T14:41:41.490645+0000","last_unstale":"2024-09-13T14:43:16.571431+0000","last_undegraded":"2024-09-13T14:43:16.571431+0000","last_fullsized":"2024-09-13T14:43:16.571431+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:01.779411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"116'494","reported_seq":517,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:19.878125+0000","last_change":"2024-09-13T14:43:09.079510+0000","last_active":"2024-09-13T14:43:19.878125+0000","last_peered":"2024-09-13T14:43:19.878125+0000","last_clean":"2024-09-13T14:43:19.878125+0000","last_became_active":"2024-09-13T14:43:09.078574+0000","last_became_peered":"2024-09-13T14:43:09.078574+0000","last_unstale":"2024-09-13T14:43:19.878125+0000","last_undegraded":"2024-09-13T14:43:19.878125+0000","last_fullsized":"2024-09-13T14:43:19.878125+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:10.643756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32374784,"num_objects":494,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":494,"num_write_kb":31616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:15.694898+0000","last_change":"2024-09-13T14:40:36.879533+0000","last_active":"2024-09-13T14:43:15.694898+0000","last_peered":"2024-09-13T14:43:15.694898+0000","last_clean":"2024-09-13T14:43:15.694898+0000","last_became_active":"2024-09-13T14:40:36.879212+0000","last_became_peered":"2024-09-13T14:40:36.879212+0000","last_unstale":"2024-09-13T14:43:15.694898+0000","last_undegraded":"2024-09-13T14:43:15.694898+0000","last_fullsized":"2024-09-13T14:43:15.694898+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:45:03.087074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571611+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:43:16.571611+0000","last_peered":"2024-09-13T14:43:16.571611+0000","last_clean":"2024-09-13T14:43:16.571611+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:43:16.571611+0000","last_undegraded":"2024-09-13T14:43:16.571611+0000","last_fullsized":"2024-09-13T14:43:16.571611+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"117'1170","reported_seq":1185,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:23.715277+0000","last_change":"2024-09-13T14:43:09.683294+0000","last_active":"2024-09-13T14:43:23.715277+0000","last_peered":"2024-09-13T14:43:23.715277+0000","last_clean":"2024-09-13T14:43:23.715277+0000","last_became_active":"2024-09-13T14:43:09.682982+0000","last_became_peered":"2024-09-13T14:43:09.682982+0000","last_unstale":"2024-09-13T14:43:23.715277+0000","last_undegraded":"2024-09-13T14:43:23.715277+0000","last_fullsized":"2024-09-13T14:43:23.715277+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:05.063594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1170,"num_write_kb":74880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571305+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:43:16.571305+0000","last_peered":"2024-09-13T14:43:16.571305+0000","last_clean":"2024-09-13T14:43:16.571305+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:43:16.571305+0000","last_undegraded":"2024-09-13T14:43:16.571305+0000","last_fullsized":"2024-09-13T14:43:16.571305+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571842+0000","last_change":"2024-09-13T14:40:36.875852+0000","last_active":"2024-09-13T14:43:16.571842+0000","last_peered":"2024-09-13T14:43:16.571842+0000","last_clean":"2024-09-13T14:43:16.571842+0000","last_became_active":"2024-09-13T14:40:36.875486+0000","last_became_peered":"2024-09-13T14:40:36.875486+0000","last_unstale":"2024-09-13T14:43:16.571842+0000","last_undegraded":"2024-09-13T14:43:16.571842+0000","last_fullsized":"2024-09-13T14:43:16.571842+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:18.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"117'1202","reported_seq":1219,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:23.705731+0000","last_change":"2024-09-13T14:43:09.683235+0000","last_active":"2024-09-13T14:43:23.705731+0000","last_peered":"2024-09-13T14:43:23.705731+0000","last_clean":"2024-09-13T14:43:23.705731+0000","last_became_active":"2024-09-13T14:43:09.682916+0000","last_became_peered":"2024-09-13T14:43:09.682916+0000","last_unstale":"2024-09-13T14:43:23.705731+0000","last_undegraded":"2024-09-13T14:43:23.705731+0000","last_fullsized":"2024-09-13T14:43:23.705731+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1202,"log_dups_size":0,"ondisk_log_size":1202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:19.580771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78774272,"num_objects":1202,"num_object_clones":0,"num_object_copies":3606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1202,"num_write_kb":76928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.764395+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:43:20.764395+0000","last_peered":"2024-09-13T14:43:20.764395+0000","last_clean":"2024-09-13T14:43:20.764395+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:43:20.764395+0000","last_undegraded":"2024-09-13T14:43:20.764395+0000","last_fullsized":"2024-09-13T14:43:20.764395+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679208+0000","last_change":"2024-09-13T14:41:41.488308+0000","last_active":"2024-09-13T14:43:16.679208+0000","last_peered":"2024-09-13T14:43:16.679208+0000","last_clean":"2024-09-13T14:43:16.679208+0000","last_became_active":"2024-09-13T14:41:41.487374+0000","last_became_peered":"2024-09-13T14:41:41.487374+0000","last_unstale":"2024-09-13T14:43:16.679208+0000","last_undegraded":"2024-09-13T14:43:16.679208+0000","last_fullsized":"2024-09-13T14:43:16.679208+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:51.242762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"117'1159","reported_seq":1176,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:23.721452+0000","last_change":"2024-09-13T14:43:09.679474+0000","last_active":"2024-09-13T14:43:23.721452+0000","last_peered":"2024-09-13T14:43:23.721452+0000","last_clean":"2024-09-13T14:43:23.721452+0000","last_became_active":"2024-09-13T14:43:09.679179+0000","last_became_peered":"2024-09-13T14:43:09.679179+0000","last_unstale":"2024-09-13T14:43:23.721452+0000","last_undegraded":"2024-09-13T14:43:23.721452+0000","last_fullsized":"2024-09-13T14:43:23.721452+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1159,"log_dups_size":0,"ondisk_log_size":1159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:01.483055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75956224,"num_objects":1159,"num_object_clones":0,"num_object_copies":3477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1159,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":410,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.678737+0000","last_change":"2024-09-13T14:40:04.164078+0000","last_active":"2024-09-13T14:43:16.678737+0000","last_peered":"2024-09-13T14:43:16.678737+0000","last_clean":"2024-09-13T14:43:16.678737+0000","last_became_active":"2024-09-13T14:40:04.163685+0000","last_became_peered":"2024-09-13T14:40:04.163685+0000","last_unstale":"2024-09-13T14:43:16.678737+0000","last_undegraded":"2024-09-13T14:43:16.678737+0000","last_fullsized":"2024-09-13T14:43:16.678737+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:40.476401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":594,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.771206+0000","last_change":"2024-09-13T14:43:12.684107+0000","last_active":"2024-09-13T14:43:20.771206+0000","last_peered":"2024-09-13T14:43:20.771206+0000","last_clean":"2024-09-13T14:43:20.771206+0000","last_became_active":"2024-09-13T14:43:12.683522+0000","last_became_peered":"2024-09-13T14:43:12.683522+0000","last_unstale":"2024-09-13T14:43:20.771206+0000","last_undegraded":"2024-09-13T14:43:20.771206+0000","last_fullsized":"2024-09-13T14:43:20.771206+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:52.396314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.765305+0000","last_change":"2024-09-13T14:43:09.680598+0000","last_active":"2024-09-13T14:43:20.765305+0000","last_peered":"2024-09-13T14:43:20.765305+0000","last_clean":"2024-09-13T14:43:20.765305+0000","last_became_active":"2024-09-13T14:43:09.680298+0000","last_became_peered":"2024-09-13T14:43:09.680298+0000","last_unstale":"2024-09-13T14:43:20.765305+0000","last_undegraded":"2024-09-13T14:43:20.765305+0000","last_fullsized":"2024-09-13T14:43:20.765305+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:54.667234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:15.694986+0000","last_change":"2024-09-13T14:28:14.409690+0000","last_active":"2024-09-13T14:43:15.694986+0000","last_peered":"2024-09-13T14:43:15.694986+0000","last_clean":"2024-09-13T14:43:15.694986+0000","last_became_active":"2024-09-13T14:28:14.409379+0000","last_became_peered":"2024-09-13T14:28:14.409379+0000","last_unstale":"2024-09-13T14:43:15.694986+0000","last_undegraded":"2024-09-13T14:43:15.694986+0000","last_fullsized":"2024-09-13T14:43:15.694986+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:15.731901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"116'531","reported_seq":552,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:19.871639+0000","last_change":"2024-09-13T14:43:09.681916+0000","last_active":"2024-09-13T14:43:19.871639+0000","last_peered":"2024-09-13T14:43:19.871639+0000","last_clean":"2024-09-13T14:43:19.871639+0000","last_became_active":"2024-09-13T14:43:09.681588+0000","last_became_peered":"2024-09-13T14:43:09.681588+0000","last_unstale":"2024-09-13T14:43:19.871639+0000","last_undegraded":"2024-09-13T14:43:19.871639+0000","last_fullsized":"2024-09-13T14:43:19.871639+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:27.266373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":531,"num_object_clones":0,"num_object_copies":1593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":33984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":269,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679044+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:43:16.679044+0000","last_peered":"2024-09-13T14:43:16.679044+0000","last_clean":"2024-09-13T14:43:16.679044+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:43:16.679044+0000","last_undegraded":"2024-09-13T14:43:16.679044+0000","last_fullsized":"2024-09-13T14:43:16.679044+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":80,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.768166+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:43:20.768166+0000","last_peered":"2024-09-13T14:43:20.768166+0000","last_clean":"2024-09-13T14:43:20.768166+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:43:20.768166+0000","last_undegraded":"2024-09-13T14:43:20.768166+0000","last_fullsized":"2024-09-13T14:43:20.768166+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769347+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:43:20.769347+0000","last_peered":"2024-09-13T14:43:20.769347+0000","last_clean":"2024-09-13T14:43:20.769347+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:43:20.769347+0000","last_undegraded":"2024-09-13T14:43:20.769347+0000","last_fullsized":"2024-09-13T14:43:20.769347+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"117'988","reported_seq":1012,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:22.668909+0000","last_change":"2024-09-13T14:43:09.679876+0000","last_active":"2024-09-13T14:43:22.668909+0000","last_peered":"2024-09-13T14:43:22.668909+0000","last_clean":"2024-09-13T14:43:22.668909+0000","last_became_active":"2024-09-13T14:43:09.679544+0000","last_became_peered":"2024-09-13T14:43:09.679544+0000","last_unstale":"2024-09-13T14:43:22.668909+0000","last_undegraded":"2024-09-13T14:43:22.668909+0000","last_fullsized":"2024-09-13T14:43:22.668909+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":988,"log_dups_size":0,"ondisk_log_size":988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:52.936882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":63232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":644,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.770431+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:43:20.770431+0000","last_peered":"2024-09-13T14:43:20.770431+0000","last_clean":"2024-09-13T14:43:20.770431+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:43:20.770431+0000","last_undegraded":"2024-09-13T14:43:20.770431+0000","last_fullsized":"2024-09-13T14:43:20.770431+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.678756+0000","last_change":"2024-09-13T14:41:09.650296+0000","last_active":"2024-09-13T14:43:16.678756+0000","last_peered":"2024-09-13T14:43:16.678756+0000","last_clean":"2024-09-13T14:43:16.678756+0000","last_became_active":"2024-09-13T14:41:09.650016+0000","last_became_peered":"2024-09-13T14:41:09.650016+0000","last_unstale":"2024-09-13T14:43:16.678756+0000","last_undegraded":"2024-09-13T14:43:16.678756+0000","last_fullsized":"2024-09-13T14:43:16.678756+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:29.994396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":100,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.772149+0000","last_change":"2024-09-13T14:43:09.679993+0000","last_active":"2024-09-13T14:43:20.772149+0000","last_peered":"2024-09-13T14:43:20.772149+0000","last_clean":"2024-09-13T14:43:20.772149+0000","last_became_active":"2024-09-13T14:43:09.679699+0000","last_became_peered":"2024-09-13T14:43:09.679699+0000","last_unstale":"2024-09-13T14:43:20.772149+0000","last_undegraded":"2024-09-13T14:43:20.772149+0000","last_fullsized":"2024-09-13T14:43:20.772149+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:20.969934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"117'846","reported_seq":871,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:21.716756+0000","last_change":"2024-09-13T14:43:09.078374+0000","last_active":"2024-09-13T14:43:21.716756+0000","last_peered":"2024-09-13T14:43:21.716756+0000","last_clean":"2024-09-13T14:43:21.716756+0000","last_became_active":"2024-09-13T14:43:09.078086+0000","last_became_peered":"2024-09-13T14:43:09.078086+0000","last_unstale":"2024-09-13T14:43:21.716756+0000","last_undegraded":"2024-09-13T14:43:21.716756+0000","last_fullsized":"2024-09-13T14:43:21.716756+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":846,"log_dups_size":0,"ondisk_log_size":846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:18.161679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55443456,"num_objects":846,"num_object_clones":0,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":846,"num_write_kb":54144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":658,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679306+0000","last_change":"2024-09-13T14:43:09.699916+0000","last_active":"2024-09-13T14:43:16.679306+0000","last_peered":"2024-09-13T14:43:16.679306+0000","last_clean":"2024-09-13T14:43:16.679306+0000","last_became_active":"2024-09-13T14:43:09.683720+0000","last_became_peered":"2024-09-13T14:43:09.683720+0000","last_unstale":"2024-09-13T14:43:16.679306+0000","last_undegraded":"2024-09-13T14:43:16.679306+0000","last_fullsized":"2024-09-13T14:43:16.679306+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:52.426769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":79,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.570818+0000","last_change":"2024-09-13T14:40:04.165067+0000","last_active":"2024-09-13T14:43:16.570818+0000","last_peered":"2024-09-13T14:43:16.570818+0000","last_clean":"2024-09-13T14:43:16.570818+0000","last_became_active":"2024-09-13T14:40:04.164682+0000","last_became_peered":"2024-09-13T14:40:04.164682+0000","last_unstale":"2024-09-13T14:43:16.570818+0000","last_undegraded":"2024-09-13T14:43:16.570818+0000","last_fullsized":"2024-09-13T14:43:16.570818+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:56.250533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":80,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679126+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:43:16.679126+0000","last_peered":"2024-09-13T14:43:16.679126+0000","last_clean":"2024-09-13T14:43:16.679126+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:43:16.679126+0000","last_undegraded":"2024-09-13T14:43:16.679126+0000","last_fullsized":"2024-09-13T14:43:16.679126+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"117'851","reported_seq":875,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:21.728882+0000","last_change":"2024-09-13T14:43:09.683803+0000","last_active":"2024-09-13T14:43:21.728882+0000","last_peered":"2024-09-13T14:43:21.728882+0000","last_clean":"2024-09-13T14:43:21.728882+0000","last_became_active":"2024-09-13T14:43:09.683391+0000","last_became_peered":"2024-09-13T14:43:09.683391+0000","last_unstale":"2024-09-13T14:43:21.728882+0000","last_undegraded":"2024-09-13T14:43:21.728882+0000","last_fullsized":"2024-09-13T14:43:21.728882+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:44:12.925534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55771136,"num_objects":851,"num_object_clones":0,"num_object_copies":2553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":851,"num_write_kb":54464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":100,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.764979+0000","last_change":"2024-09-13T14:43:09.681891+0000","last_active":"2024-09-13T14:43:20.764979+0000","last_peered":"2024-09-13T14:43:20.764979+0000","last_clean":"2024-09-13T14:43:20.764979+0000","last_became_active":"2024-09-13T14:43:09.681552+0000","last_became_peered":"2024-09-13T14:43:09.681552+0000","last_unstale":"2024-09-13T14:43:20.764979+0000","last_undegraded":"2024-09-13T14:43:20.764979+0000","last_fullsized":"2024-09-13T14:43:20.764979+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:46.613427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":608,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679311+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:43:16.679311+0000","last_peered":"2024-09-13T14:43:16.679311+0000","last_clean":"2024-09-13T14:43:16.679311+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:43:16.679311+0000","last_undegraded":"2024-09-13T14:43:16.679311+0000","last_fullsized":"2024-09-13T14:43:16.679311+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679510+0000","last_change":"2024-09-13T14:43:09.681228+0000","last_active":"2024-09-13T14:43:16.679510+0000","last_peered":"2024-09-13T14:43:16.679510+0000","last_clean":"2024-09-13T14:43:16.679510+0000","last_became_active":"2024-09-13T14:43:09.680908+0000","last_became_peered":"2024-09-13T14:43:09.680908+0000","last_unstale":"2024-09-13T14:43:16.679510+0000","last_undegraded":"2024-09-13T14:43:16.679510+0000","last_fullsized":"2024-09-13T14:43:16.679510+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:33.151302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"117'1125","reported_seq":1142,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:23.682411+0000","last_change":"2024-09-13T14:43:09.680725+0000","last_active":"2024-09-13T14:43:23.682411+0000","last_peered":"2024-09-13T14:43:23.682411+0000","last_clean":"2024-09-13T14:43:23.682411+0000","last_became_active":"2024-09-13T14:43:09.680418+0000","last_became_peered":"2024-09-13T14:43:09.680418+0000","last_unstale":"2024-09-13T14:43:23.682411+0000","last_undegraded":"2024-09-13T14:43:23.682411+0000","last_fullsized":"2024-09-13T14:43:23.682411+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:55.779980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73728000,"num_objects":1125,"num_object_clones":0,"num_object_copies":3375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1125,"num_write_kb":72000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679214+0000","last_change":"2024-09-13T14:43:09.680063+0000","last_active":"2024-09-13T14:43:16.679214+0000","last_peered":"2024-09-13T14:43:16.679214+0000","last_clean":"2024-09-13T14:43:16.679214+0000","last_became_active":"2024-09-13T14:43:09.679740+0000","last_became_peered":"2024-09-13T14:43:09.679740+0000","last_unstale":"2024-09-13T14:43:16.679214+0000","last_undegraded":"2024-09-13T14:43:16.679214+0000","last_fullsized":"2024-09-13T14:43:16.679214+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:55.701172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":429,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679335+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:43:16.679335+0000","last_peered":"2024-09-13T14:43:16.679335+0000","last_clean":"2024-09-13T14:43:16.679335+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:43:16.679335+0000","last_undegraded":"2024-09-13T14:43:16.679335+0000","last_fullsized":"2024-09-13T14:43:16.679335+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":76,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769157+0000","last_change":"2024-09-13T14:43:09.682650+0000","last_active":"2024-09-13T14:43:20.769157+0000","last_peered":"2024-09-13T14:43:20.769157+0000","last_clean":"2024-09-13T14:43:20.769157+0000","last_became_active":"2024-09-13T14:43:09.682335+0000","last_became_peered":"2024-09-13T14:43:09.682335+0000","last_unstale":"2024-09-13T14:43:20.769157+0000","last_undegraded":"2024-09-13T14:43:20.769157+0000","last_fullsized":"2024-09-13T14:43:20.769157+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:26.113631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"117'1147","reported_seq":1163,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:23.712608+0000","last_change":"2024-09-13T14:43:09.080169+0000","last_active":"2024-09-13T14:43:23.712608+0000","last_peered":"2024-09-13T14:43:23.712608+0000","last_clean":"2024-09-13T14:43:23.712608+0000","last_became_active":"2024-09-13T14:43:09.079885+0000","last_became_peered":"2024-09-13T14:43:09.079885+0000","last_unstale":"2024-09-13T14:43:23.712608+0000","last_undegraded":"2024-09-13T14:43:23.712608+0000","last_fullsized":"2024-09-13T14:43:23.712608+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1147,"log_dups_size":0,"ondisk_log_size":1147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:05.421672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75169792,"num_objects":1147,"num_object_clones":0,"num_object_copies":3441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1147,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.765338+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:43:20.765338+0000","last_peered":"2024-09-13T14:43:20.765338+0000","last_clean":"2024-09-13T14:43:20.765338+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:43:20.765338+0000","last_undegraded":"2024-09-13T14:43:20.765338+0000","last_fullsized":"2024-09-13T14:43:20.765338+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":492,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679389+0000","last_change":"2024-09-13T14:40:36.875554+0000","last_active":"2024-09-13T14:43:16.679389+0000","last_peered":"2024-09-13T14:43:16.679389+0000","last_clean":"2024-09-13T14:43:16.679389+0000","last_became_active":"2024-09-13T14:40:36.874924+0000","last_became_peered":"2024-09-13T14:40:36.874924+0000","last_unstale":"2024-09-13T14:43:16.679389+0000","last_undegraded":"2024-09-13T14:43:16.679389+0000","last_fullsized":"2024-09-13T14:43:16.679389+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:11.594853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":85,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.763113+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:43:20.763113+0000","last_peered":"2024-09-13T14:43:20.763113+0000","last_clean":"2024-09-13T14:43:20.763113+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:43:20.763113+0000","last_undegraded":"2024-09-13T14:43:20.763113+0000","last_fullsized":"2024-09-13T14:43:20.763113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"117'1274","reported_seq":1299,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:24.485195+0000","last_change":"2024-09-13T14:43:09.077743+0000","last_active":"2024-09-13T14:43:24.485195+0000","last_peered":"2024-09-13T14:43:24.485195+0000","last_clean":"2024-09-13T14:43:24.485195+0000","last_became_active":"2024-09-13T14:43:09.077468+0000","last_became_peered":"2024-09-13T14:43:09.077468+0000","last_unstale":"2024-09-13T14:43:24.485195+0000","last_undegraded":"2024-09-13T14:43:24.485195+0000","last_fullsized":"2024-09-13T14:43:24.485195+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:47.572693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1274,"num_write_kb":81536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":85,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769091+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:43:20.769091+0000","last_peered":"2024-09-13T14:43:20.769091+0000","last_clean":"2024-09-13T14:43:20.769091+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:43:20.769091+0000","last_undegraded":"2024-09-13T14:43:20.769091+0000","last_fullsized":"2024-09-13T14:43:20.769091+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":683,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:15.695287+0000","last_change":"2024-09-13T14:41:09.650053+0000","last_active":"2024-09-13T14:43:15.695287+0000","last_peered":"2024-09-13T14:43:15.695287+0000","last_clean":"2024-09-13T14:43:15.695287+0000","last_became_active":"2024-09-13T14:41:09.649530+0000","last_became_peered":"2024-09-13T14:41:09.649530+0000","last_unstale":"2024-09-13T14:43:15.695287+0000","last_undegraded":"2024-09-13T14:43:15.695287+0000","last_fullsized":"2024-09-13T14:43:15.695287+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:39.045795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571114+0000","last_change":"2024-09-13T14:43:09.681149+0000","last_active":"2024-09-13T14:43:16.571114+0000","last_peered":"2024-09-13T14:43:16.571114+0000","last_clean":"2024-09-13T14:43:16.571114+0000","last_became_active":"2024-09-13T14:43:09.680811+0000","last_became_peered":"2024-09-13T14:43:09.680811+0000","last_unstale":"2024-09-13T14:43:16.571114+0000","last_undegraded":"2024-09-13T14:43:16.571114+0000","last_fullsized":"2024-09-13T14:43:16.571114+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:21.687579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571390+0000","last_change":"2024-09-13T14:40:36.870296+0000","last_active":"2024-09-13T14:43:16.571390+0000","last_peered":"2024-09-13T14:43:16.571390+0000","last_clean":"2024-09-13T14:43:16.571390+0000","last_became_active":"2024-09-13T14:40:36.869969+0000","last_became_peered":"2024-09-13T14:40:36.869969+0000","last_unstale":"2024-09-13T14:43:16.571390+0000","last_undegraded":"2024-09-13T14:43:16.571390+0000","last_fullsized":"2024-09-13T14:43:16.571390+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:57.021118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571495+0000","last_change":"2024-09-13T14:28:14.293329+0000","last_active":"2024-09-13T14:43:16.571495+0000","last_peered":"2024-09-13T14:43:16.571495+0000","last_clean":"2024-09-13T14:43:16.571495+0000","last_became_active":"2024-09-13T14:28:14.293075+0000","last_became_peered":"2024-09-13T14:28:14.293075+0000","last_unstale":"2024-09-13T14:43:16.571495+0000","last_undegraded":"2024-09-13T14:43:16.571495+0000","last_fullsized":"2024-09-13T14:43:16.571495+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:20.310761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679409+0000","last_change":"2024-09-13T14:41:09.650180+0000","last_active":"2024-09-13T14:43:16.679409+0000","last_peered":"2024-09-13T14:43:16.679409+0000","last_clean":"2024-09-13T14:43:16.679409+0000","last_became_active":"2024-09-13T14:41:09.649908+0000","last_became_peered":"2024-09-13T14:41:09.649908+0000","last_unstale":"2024-09-13T14:43:16.679409+0000","last_undegraded":"2024-09-13T14:43:16.679409+0000","last_fullsized":"2024-09-13T14:43:16.679409+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:08.029267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679195+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:43:16.679195+0000","last_peered":"2024-09-13T14:43:16.679195+0000","last_clean":"2024-09-13T14:43:16.679195+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:43:16.679195+0000","last_undegraded":"2024-09-13T14:43:16.679195+0000","last_fullsized":"2024-09-13T14:43:16.679195+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.679287+0000","last_change":"2024-09-13T14:43:09.682269+0000","last_active":"2024-09-13T14:43:16.679287+0000","last_peered":"2024-09-13T14:43:16.679287+0000","last_clean":"2024-09-13T14:43:16.679287+0000","last_became_active":"2024-09-13T14:43:09.681735+0000","last_became_peered":"2024-09-13T14:43:09.681735+0000","last_unstale":"2024-09-13T14:43:16.679287+0000","last_undegraded":"2024-09-13T14:43:16.679287+0000","last_fullsized":"2024-09-13T14:43:16.679287+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:48.513102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":39,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.764627+0000","last_change":"2024-09-13T14:41:41.484817+0000","last_active":"2024-09-13T14:43:20.764627+0000","last_peered":"2024-09-13T14:43:20.764627+0000","last_clean":"2024-09-13T14:43:20.764627+0000","last_became_active":"2024-09-13T14:41:41.484505+0000","last_became_peered":"2024-09-13T14:41:41.484505+0000","last_unstale":"2024-09-13T14:43:20.764627+0000","last_undegraded":"2024-09-13T14:43:20.764627+0000","last_fullsized":"2024-09-13T14:43:20.764627+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:49.618334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":87,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769651+0000","last_change":"2024-09-13T14:43:09.682307+0000","last_active":"2024-09-13T14:43:20.769651+0000","last_peered":"2024-09-13T14:43:20.769651+0000","last_clean":"2024-09-13T14:43:20.769651+0000","last_became_active":"2024-09-13T14:43:09.682009+0000","last_became_peered":"2024-09-13T14:43:09.682009+0000","last_unstale":"2024-09-13T14:43:20.769651+0000","last_undegraded":"2024-09-13T14:43:20.769651+0000","last_fullsized":"2024-09-13T14:43:20.769651+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:04:45.346966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.762689+0000","last_change":"2024-09-13T14:41:41.489947+0000","last_active":"2024-09-13T14:43:20.762689+0000","last_peered":"2024-09-13T14:43:20.762689+0000","last_clean":"2024-09-13T14:43:20.762689+0000","last_became_active":"2024-09-13T14:41:41.487936+0000","last_became_peered":"2024-09-13T14:41:41.487936+0000","last_unstale":"2024-09-13T14:43:20.762689+0000","last_undegraded":"2024-09-13T14:43:20.762689+0000","last_fullsized":"2024-09-13T14:43:20.762689+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:50.293987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:15.695272+0000","last_change":"2024-09-13T14:43:09.681366+0000","last_active":"2024-09-13T14:43:15.695272+0000","last_peered":"2024-09-13T14:43:15.695272+0000","last_clean":"2024-09-13T14:43:15.695272+0000","last_became_active":"2024-09-13T14:43:09.681080+0000","last_became_peered":"2024-09-13T14:43:09.681080+0000","last_unstale":"2024-09-13T14:43:15.695272+0000","last_undegraded":"2024-09-13T14:43:15.695272+0000","last_fullsized":"2024-09-13T14:43:15.695272+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:16.367434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.764092+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:43:20.764092+0000","last_peered":"2024-09-13T14:43:20.764092+0000","last_clean":"2024-09-13T14:43:20.764092+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:43:20.764092+0000","last_undegraded":"2024-09-13T14:43:20.764092+0000","last_fullsized":"2024-09-13T14:43:20.764092+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":80,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769271+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:43:20.769271+0000","last_peered":"2024-09-13T14:43:20.769271+0000","last_clean":"2024-09-13T14:43:20.769271+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:43:20.769271+0000","last_undegraded":"2024-09-13T14:43:20.769271+0000","last_fullsized":"2024-09-13T14:43:20.769271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.678668+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:43:16.678668+0000","last_peered":"2024-09-13T14:43:16.678668+0000","last_clean":"2024-09-13T14:43:16.678668+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:43:16.678668+0000","last_undegraded":"2024-09-13T14:43:16.678668+0000","last_fullsized":"2024-09-13T14:43:16.678668+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":89,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.764758+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:43:20.764758+0000","last_peered":"2024-09-13T14:43:20.764758+0000","last_clean":"2024-09-13T14:43:20.764758+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:43:20.764758+0000","last_undegraded":"2024-09-13T14:43:20.764758+0000","last_fullsized":"2024-09-13T14:43:20.764758+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571272+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:43:16.571272+0000","last_peered":"2024-09-13T14:43:16.571272+0000","last_clean":"2024-09-13T14:43:16.571272+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:43:16.571272+0000","last_undegraded":"2024-09-13T14:43:16.571272+0000","last_fullsized":"2024-09-13T14:43:16.571272+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":109,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.770856+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:43:20.770856+0000","last_peered":"2024-09-13T14:43:20.770856+0000","last_clean":"2024-09-13T14:43:20.770856+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:43:20.770856+0000","last_undegraded":"2024-09-13T14:43:20.770856+0000","last_fullsized":"2024-09-13T14:43:20.770856+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":74,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769053+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:43:20.769053+0000","last_peered":"2024-09-13T14:43:20.769053+0000","last_clean":"2024-09-13T14:43:20.769053+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:43:20.769053+0000","last_undegraded":"2024-09-13T14:43:20.769053+0000","last_fullsized":"2024-09-13T14:43:20.769053+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:18.725374+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:43:18.725374+0000","last_peered":"2024-09-13T14:43:18.725374+0000","last_clean":"2024-09-13T14:43:18.725374+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:43:18.725374+0000","last_undegraded":"2024-09-13T14:43:18.725374+0000","last_fullsized":"2024-09-13T14:43:18.725374+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":32,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571458+0000","last_change":"2024-09-13T14:43:09.691788+0000","last_active":"2024-09-13T14:43:16.571458+0000","last_peered":"2024-09-13T14:43:16.571458+0000","last_clean":"2024-09-13T14:43:16.571458+0000","last_became_active":"2024-09-13T14:43:09.679287+0000","last_became_peered":"2024-09-13T14:43:09.679287+0000","last_unstale":"2024-09-13T14:43:16.571458+0000","last_undegraded":"2024-09-13T14:43:16.571458+0000","last_fullsized":"2024-09-13T14:43:16.571458+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:05.081262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571167+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:43:16.571167+0000","last_peered":"2024-09-13T14:43:16.571167+0000","last_clean":"2024-09-13T14:43:16.571167+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:43:16.571167+0000","last_undegraded":"2024-09-13T14:43:16.571167+0000","last_fullsized":"2024-09-13T14:43:16.571167+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":32,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.571664+0000","last_change":"2024-09-13T14:43:09.687780+0000","last_active":"2024-09-13T14:43:16.571664+0000","last_peered":"2024-09-13T14:43:16.571664+0000","last_clean":"2024-09-13T14:43:16.571664+0000","last_became_active":"2024-09-13T14:43:09.679048+0000","last_became_peered":"2024-09-13T14:43:09.679048+0000","last_unstale":"2024-09-13T14:43:16.571664+0000","last_undegraded":"2024-09-13T14:43:16.571664+0000","last_fullsized":"2024-09-13T14:43:16.571664+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:51.346871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":109,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.764943+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:43:20.764943+0000","last_peered":"2024-09-13T14:43:20.764943+0000","last_clean":"2024-09-13T14:43:20.764943+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:43:20.764943+0000","last_undegraded":"2024-09-13T14:43:20.764943+0000","last_fullsized":"2024-09-13T14:43:20.764943+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.765615+0000","last_change":"2024-09-13T14:41:41.486093+0000","last_active":"2024-09-13T14:43:20.765615+0000","last_peered":"2024-09-13T14:43:20.765615+0000","last_clean":"2024-09-13T14:43:20.765615+0000","last_became_active":"2024-09-13T14:41:41.485759+0000","last_became_peered":"2024-09-13T14:41:41.485759+0000","last_unstale":"2024-09-13T14:43:20.765615+0000","last_undegraded":"2024-09-13T14:43:20.765615+0000","last_fullsized":"2024-09-13T14:43:20.765615+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:05.424465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-09-13T14:43:16.570955+0000","last_change":"2024-09-13T14:40:36.876656+0000","last_active":"2024-09-13T14:43:16.570955+0000","last_peered":"2024-09-13T14:43:16.570955+0000","last_clean":"2024-09-13T14:43:16.570955+0000","last_became_active":"2024-09-13T14:40:36.876361+0000","last_became_peered":"2024-09-13T14:40:36.876361+0000","last_unstale":"2024-09-13T14:43:16.570955+0000","last_undegraded":"2024-09-13T14:43:16.570955+0000","last_fullsized":"2024-09-13T14:43:16.570955+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:51.843071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-13T14:43:20.769548+0000","last_change":"2024-09-13T14:43:09.690395+0000","last_active":"2024-09-13T14:43:20.769548+0000","last_peered":"2024-09-13T14:43:20.769548+0000","last_clean":"2024-09-13T14:43:20.769548+0000","last_became_active":"2024-09-13T14:43:09.685936+0000","last_became_peered":"2024-09-13T14:43:09.685936+0000","last_unstale":"2024-09-13T14:43:20.769548+0000","last_undegraded":"2024-09-13T14:43:20.769548+0000","last_fullsized":"2024-09-13T14:43:20.769548+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:20.511544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1042677760,"num_objects":15910,"num_object_clones":0,"num_object_copies":47730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15910,"num_write_kb":1018240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3157327872,"data_stored":3157327872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15910,"ondisk_log_size":15910,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":652,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12582988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234192,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":442520,"kb_used_data":403216,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":93302632,"statfs":{"total":95995035648,"available":95541895168,"internally_reserved":0,"allocated":412893184,"data_stored":412259427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40237515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495828,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302288,"kb_used_data":271176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93442864,"statfs":{"total":95995035648,"available":95685492736,"internally_reserved":0,"allocated":277684224,"data_stored":277058841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019103,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":767700,"kb_used_data":728396,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":92977452,"statfs":{"total":95995035648,"available":95208910848,"internally_reserved":0,"allocated":745877504,"data_stored":745247560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856467991,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349316,"kb_used_data":321852,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93395836,"statfs":{"total":95995035648,"available":95637336064,"internally_reserved":0,"allocated":329576448,"data_stored":328958158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":28109988},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631517,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":576520,"kb_used_data":544828,"kb_used_omap":8,"kb_used_meta":31671,"kb_avail":93168632,"statfs":{"total":95995035648,"available":95404679168,"internally_reserved":0,"allocated":557903872,"data_stored":557286038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":32432095},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795043,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139444,"kb_used_data":107756,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93605708,"statfs":{"total":95995035648,"available":95852244992,"internally_reserved":0,"allocated":110342144,"data_stored":109731898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":32432754},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958570,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515152,"kb_used_data":483460,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93230000,"statfs":{"total":95995035648,"available":95467520000,"internally_reserved":0,"allocated":495063040,"data_stored":494437565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":32432615},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":562954240,"data_stored":562954240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165543936,"data_stored":165543936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":588709888,"data_stored":588709888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":332791808,"data_stored":332791808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":758579200,"data_stored":758579200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":298713088,"data_stored":298713088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":450035712,"data_stored":450035712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:43:30.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-09-13T14:43:30.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.850 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:30.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:30.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:30.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:30.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:31.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:31.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:31.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:31.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:31.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:31.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:31.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:31.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 14 16 34242 34226 152.777 115.375 0.0101428 0.00654005 2024-09-13T14:43:31.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:31.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:31.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:31.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:31.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:31.843 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:31.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:31.912 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:31.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:31.927 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:31.942 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:31.954 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:43:31.954 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:31.993 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:43:31.993 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:32.008 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:31 smithi154 ceph-mon[107038]: pgmap v310: 89 pgs: 89 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 156 MiB/s wr, 2.49k op/s 2024-09-13T14:43:32.008 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:31 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1186098323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:32.008 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:43:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:43:31] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:43:32.010 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:31 smithi154 ceph-mon[102314]: pgmap v310: 89 pgs: 89 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 156 MiB/s wr, 2.49k op/s 2024-09-13T14:43:32.010 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:31 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1186098323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:32.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:32.030 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:43:32.030 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:32.064 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:43:32.064 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.065 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:32.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:32.097 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:43:32.097 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:32.127 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:43:32.127 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:32.156 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:43:32.156 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:32.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:32.187 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:43:32.187 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:32.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:32.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:31 smithi172 ceph-mon[100696]: pgmap v310: 89 pgs: 89 active+clean; 1.7 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 156 MiB/s wr, 2.49k op/s 2024-09-13T14:43:32.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:31 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1186098323' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:32.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:32.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:32.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:32.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:32.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 15 16 36031 36015 150.046 111.812 0.0082694 0.00665811 2024-09-13T14:43:32.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:32.618 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:32.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:32.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:32.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:32.891 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:32.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:32.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:33.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:33.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:33.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:33.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:33.292 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:33.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:33.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:33.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 16 16 38044 38028 148.531 125.812 0.00531527 0.00672805 2024-09-13T14:43:33.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:33.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:33.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:33.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:33.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:33.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:33.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:33.960 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:33.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:34.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:34.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:34.190 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:33 smithi154 ceph-mon[102314]: pgmap v311: 89 pgs: 89 active+clean; 1.8 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-13T14:43:34.190 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:33 smithi154 ceph-mon[107038]: pgmap v311: 89 pgs: 89 active+clean; 1.8 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-13T14:43:34.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:34.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:34.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:33 smithi172 ceph-mon[100696]: pgmap v311: 89 pgs: 89 active+clean; 1.8 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-09-13T14:43:34.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:34.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:34.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:34.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:34.515 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 17 16 39952 39936 146.808 119.25 0.00301672 0.00680546 2024-09-13T14:43:34.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:34.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:34.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:34.752 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:34.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:34.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:34.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:34.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:35.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:35.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:35.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:35.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:35.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:35.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:35.439 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:35.516 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 18 16 41901 41885 145.418 121.812 0.00468077 0.00687109 2024-09-13T14:43:35.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:35.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:35.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:35.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:35.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:35.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:35.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:35.976 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:35.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:36.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:36.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:36.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:36.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:36.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:36.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:36.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:36.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:36.516 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 19 16 43694 43678 143.662 112.062 0.00450331 0.00695292 2024-09-13T14:43:36.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:36.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:36 smithi154 ceph-mon[102314]: pgmap v312: 89 pgs: 89 active+clean; 2.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-13T14:43:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:36 smithi154 ceph-mon[107038]: pgmap v312: 89 pgs: 89 active+clean; 2.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-13T14:43:36.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:36.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:36.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:36.923 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:36.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:36.941 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:36 smithi172 ceph-mon[100696]: pgmap v312: 89 pgs: 89 active+clean; 2.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-13T14:43:37.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:37.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:37.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:37.172 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:37.173 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:37.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:37.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:43:37.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:37.366 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:37.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:37.520 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:43:37.520874+0000 min lat: 0.00117109 max lat: 0.172776 avg lat: 0.00709461 2024-09-13T14:43:37.521 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:43:37.521 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 20 16 45090 45074 140.809 87.25 0.00243673 0.00709461 2024-09-13T14:43:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:37.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:37.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:37.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:37.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:37.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:37.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:37.920 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:37.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:37.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:43:38.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:38.047 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:38.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:38.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:38.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:38.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:38.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:38.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:38.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:38.524 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 21 16 46509 46493 138.314 88.6875 0.00587537 0.0072222 2024-09-13T14:43:38.528 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:38.539 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:38.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:38.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:38.691 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:43:38.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:38.779 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:38.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:38.907 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:38.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:38.958 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:38 smithi154 ceph-mon[102314]: pgmap v313: 89 pgs: 89 active+clean; 2.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:43:38.959 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3347310607' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:38.959 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:38 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:38.959 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:38 smithi154 ceph-mon[107038]: pgmap v313: 89 pgs: 89 active+clean; 2.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:43:38.959 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3347310607' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:38.959 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:38 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:38.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:38 smithi172 ceph-mon[100696]: pgmap v313: 89 pgs: 89 active+clean; 2.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-09-13T14:43:38.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3347310607' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:38.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:38 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:43:39.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:39.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:39.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:39.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:39.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:39.442 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:39.522 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 22 16 48174 48158 136.758 104.062 0.00552968 0.00730673 2024-09-13T14:43:39.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:39.576 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:39.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:39.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:39.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:39.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:39.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:39.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:39.968 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:40.042 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:39 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:40.042 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:39 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:40.042 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:39 smithi154 ceph-mon[102314]: osdmap e118: 8 total, 8 up, 7 in 2024-09-13T14:43:40.042 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:39 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:40.043 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:39 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:40.043 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:39 smithi154 ceph-mon[107038]: osdmap e118: 8 total, 8 up, 7 in 2024-09-13T14:43:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:39 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:43:40.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:39 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:43:40.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:39 smithi172 ceph-mon[100696]: osdmap e118: 8 total, 8 up, 7 in 2024-09-13T14:43:40.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:40.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:40.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:40.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:40.522 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 23 16 49859 49843 135.39 105.312 0.00305303 0.00738095 2024-09-13T14:43:40.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.610 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:40.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:40.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.735 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.865 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:40.881 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:40 smithi154 ceph-mon[102314]: pgmap v315: 89 pgs: 89 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-13T14:43:40.881 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:40 smithi154 ceph-mon[107038]: pgmap v315: 89 pgs: 89 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-13T14:43:40.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:40.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:40 smithi172 ceph-mon[100696]: pgmap v315: 89 pgs: 89 active+clean; 2.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-13T14:43:40.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:40.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:41.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:41.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:41.266 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:41.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:41.395 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:41.522 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 24 16 51421 51405 133.817 97.625 0.00433011 0.00746713 2024-09-13T14:43:41.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:41.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:41.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:41.731 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 5 pgp_num to 16 2024-09-13T14:43:41.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:41.802 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:41.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:41.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:41.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:41.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:41 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/525409959' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:43:42.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:42.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:42.187 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:41 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/525409959' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:43:42.187 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:43:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:43:41] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:43:42.187 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:41 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/525409959' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-09-13T14:43:42.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:42.225 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:42.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:42.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:43:42.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:42.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:42.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:42.523 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 25 16 53013 52997 132.444 99.5 0.00644508 0.00754481 2024-09-13T14:43:42.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:42.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:42.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:42.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:42.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:42.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:42.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:42 smithi154 ceph-mon[102314]: pgmap v316: 89 pgs: 89 active+clean; 2.8 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:43:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/525409959' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:43:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:42 smithi154 ceph-mon[102314]: osdmap e119: 8 total, 8 up, 7 in 2024-09-13T14:43:42.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:42 smithi154 ceph-mon[107038]: pgmap v316: 89 pgs: 89 active+clean; 2.8 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:43:42.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/525409959' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:43:42.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:42 smithi154 ceph-mon[107038]: osdmap e119: 8 total, 8 up, 7 in 2024-09-13T14:43:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:42 smithi172 ceph-mon[100696]: pgmap v316: 89 pgs: 89 active+clean; 2.8 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T14:43:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/525409959' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-13T14:43:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:42 smithi172 ceph-mon[100696]: osdmap e119: 8 total, 8 up, 7 in 2024-09-13T14:43:42.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:43.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:43.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:43.157 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:43.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:43.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:43.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:43.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:43.523 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 26 16 54412 54396 130.713 87.4375 0.00763308 0.00764509 2024-09-13T14:43:43.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:43.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:43.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:43.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:43.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:43.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:43.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:43.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:43.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:44.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:44.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:44.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:44.219 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:44.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:44.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:44.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.388 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:44.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:44.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:44.523 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 27 16 55901 55885 129.319 93.0625 0.0114578 0.00772775 2024-09-13T14:43:44.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:44.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:44.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:44.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:44.801 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:44.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:44.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:44.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:45.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:45.050 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:43:45.050 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:43:45.051 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:44 smithi154 ceph-mon[107038]: pgmap v318: 89 pgs: 89 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:43:45.051 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:44 smithi154 ceph-mon[102314]: pgmap v318: 89 pgs: 89 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:43:45.060 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:45.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:45.194 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:45.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:44 smithi172 ceph-mon[100696]: pgmap v318: 89 pgs: 89 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T14:43:45.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:45.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:45.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:45.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:45.523 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 28 16 57398 57382 128.042 93.5625 0.00964245 0.00780509 2024-09-13T14:43:45.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:45.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:45.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:45.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:45.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:43:45.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:45.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:45.911 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:45.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:46.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:46.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:46.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:46.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:46.230 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:46.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:46.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:46.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:46.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:46.523 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 29 15 58833 58818 126.72 89.75 0.00772451 0.00788601 2024-09-13T14:43:46.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:46.631 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:46.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:46.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:46.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:46.864 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:46 smithi154 ceph-mon[102314]: pgmap v319: 89 pgs: 89 active+clean; 3.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T14:43:46.865 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:46 smithi154 ceph-mon[107038]: pgmap v319: 89 pgs: 89 active+clean; 3.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T14:43:46.890 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:46.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:46.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:46 smithi172 ceph-mon[100696]: pgmap v319: 89 pgs: 89 active+clean; 3.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T14:43:47.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.269 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:47.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:47.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:47.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:47.524 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 30 16 60289 60273 125.525 90.9375 0.006142 0.00796043 2024-09-13T14:43:47.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:47.568 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:43:47.569 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:43:47.569 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 6, 7] 2024-09-13T14:43:47.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd out 6 2024-09-13T14:43:47.594 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:47.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:47.685 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:43:47.686 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:43:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:47.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:47.722 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:43:47.722 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:43:47.761 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:43:47.762 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:43:47.802 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:43:47.803 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:43:47.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:47.839 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:43:47.840 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:43:47.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:47.870 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:43:47.870 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:43:47.899 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:43:47.900 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:43:47.937 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:43:47.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:43:47.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:47.975 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:47 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/704931316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:47.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:47 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/704931316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:47.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:47.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:48.107 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:48.209 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:48.234 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:47 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/704931316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:43:48.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:48.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:48.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:48.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:48.525 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 31 16 61586 61570 124.088 81.0625 0.0127524 0.00805253 2024-09-13T14:43:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:48.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:48.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:48.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:48.802 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:48.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:48.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:48.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:48.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:48 smithi154 ceph-mon[102314]: pgmap v320: 89 pgs: 89 active+clean; 3.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T14:43:48.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:48 smithi154 ceph-mon[107038]: pgmap v320: 89 pgs: 89 active+clean; 3.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T14:43:49.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:49.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:49.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:49.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:49.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:49.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:48 smithi172 ceph-mon[100696]: pgmap v320: 89 pgs: 89 active+clean; 3.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T14:43:49.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:49.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:49.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:49.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:49.525 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 32 16 63044 63028 123.058 91.125 0.00788808 0.00812114 2024-09-13T14:43:49.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:49.597 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:49.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:49.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:49.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:49.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:49.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:49.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:49 smithi154 ceph-mon[102314]: pgmap v321: 89 pgs: 89 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:43:49.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:49 smithi154 ceph-mon[107038]: pgmap v321: 89 pgs: 89 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:43:49.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:50.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:50.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:50.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:50.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:49 smithi172 ceph-mon[100696]: pgmap v321: 89 pgs: 89 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-13T14:43:50.265 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:50.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:50.395 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:50.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:50.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:50.527 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 33 16 64366 64350 121.826 82.625 0.041616 0.00820255 2024-09-13T14:43:50.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:50.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:50.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:50.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:50.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:50.936 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:50.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:51.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:51.047 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:51.048 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:51.048 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:51.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:51.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:51.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:43:51.212 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:51.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:51.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:51.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:51.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:51.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:43:51.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:51.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:51.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:51.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:51.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:51.505 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:51.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:51.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 34 15 65658 65643 120.618 80.8125 0.0124991 0.00828567 2024-09-13T14:43:51.554 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:51.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:51.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.640 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:51.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:51.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:51.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:51.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:51.911 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:51.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:52.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:52.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:52.065 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:51 smithi154 ceph-mon[102314]: pgmap v322: 89 pgs: 89 active+clean; 3.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T14:43:52.065 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:51 smithi154 ceph-mon[107038]: pgmap v322: 89 pgs: 89 active+clean; 3.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T14:43:52.065 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:43:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:43:51] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:43:52.068 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:51 smithi172 ceph-mon[100696]: pgmap v322: 89 pgs: 89 active+clean; 3.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-13T14:43:52.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:52.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:52.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:52.310 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:52.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:26019 5.7 5:e2f031b8:::benchmark_data_smithi154_144140_object26018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.038664+0000", 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.401833051000001, 2024-09-13T14:43:52.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.078345820999999996, 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 26019 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038664+0000", 2024-09-13T14:43:52.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038664+0000", 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038665+0000", 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.33e-06 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038684+0000", 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9018e-05 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038696+0000", 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2362e-05 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038702+0000", 2024-09-13T14:43:52.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9580000000000004e-06 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038765+0000", 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3169000000000005e-05 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.038883+0000", 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011717099999999999 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.039012+0000", 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012923200000000001 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.039879+0000", 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00086668900000000002 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.040176+0000", 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029791399999999998 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.116914+0000", 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.076737371999999998 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.116976+0000", 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1681999999999997e-05 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.117009+0000", 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3924000000000002e-05 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:26741 5.7 5:e06e7aa6:::benchmark_data_smithi154_144140_object26740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.393165+0000", 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.047332054999998, 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.066778909999999997, 2024-09-13T14:43:52.452 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 26741 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393165+0000", 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393165+0000", 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393166+0000", 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.71e-07 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393183+0000", 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7901e-05 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393192+0000", 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2679999999999994e-06 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393197+0000", 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8940000000000001e-06 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393229+0000", 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2613000000000001e-05 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393299+0000", 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9672999999999995e-05 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.393372+0000", 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3269000000000006e-05 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.394144+0000", 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077203600000000003 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.394589+0000", 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000444723 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.459785+0000", 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.065195786000000006 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.459899+0000", 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011469 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.459944+0000", 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4186e-05 2024-09-13T14:43:52.458 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28390 5.7 5:e9dbce0c:::benchmark_data_smithi154_144140_object28389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.388646+0000", 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 24.051850701999999, 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13547147900000001, 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.459 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28390 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.388646+0000", 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.388646+0000", 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.388647+0000", 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0389999999999999e-06 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.388668+0000", 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1327000000000001e-05 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.388679+0000", 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0998999999999999e-05 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.388773+0000", 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3618999999999999e-05 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.388830+0000", 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6841e-05 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389011+0000", 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018108299999999999 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389129+0000", 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118227 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.521529+0000", 2024-09-13T14:43:52.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13239967 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.521562+0000", 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3250000000000002e-05 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.524010+0000", 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024482449999999999 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.524075+0000", 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4824000000000001e-05 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.524117+0000", 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2354999999999997e-05 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.465 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:28397 5.0 e117/110 5:0a7067a7:::benchmark_data_smithi154_144140_object28396:head v 117'1784, mlcod=117'1782)", 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.390341+0000", 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 24.050156033, 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.130060921, 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390341+0000", 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390341+0000", 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390341+0000", 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8000000000000005e-07 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390367+0000", 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5924000000000001e-05 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390378+0000", 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0937e-05 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390385+0000", 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0029999999999997e-06 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390467+0000", 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1383000000000003e-05 2024-09-13T14:43:52.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390484+0000", 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7782999999999999e-05 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520356+0000", 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12987178099999999 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520402+0000", 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5330000000000001e-05 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28398 5.7 5:e76daca3:::benchmark_data_smithi154_144140_object28397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.390921+0000", 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 24.049575873999999, 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13338502199999999, 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.471 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28398 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390921+0000", 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390921+0000", 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390922+0000", 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5820000000000001e-06 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390945+0000", 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.304e-05 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390957+0000", 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1724e-05 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.390965+0000", 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.089e-06 2024-09-13T14:43:52.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391041+0000", 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5238e-05 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391157+0000", 2024-09-13T14:43:52.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116822 2024-09-13T14:43:52.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520435+0000", 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12927746600000001 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.522882+0000", 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024471499999999999 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.522917+0000", 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5540999999999997e-05 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.524187+0000", 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012692719999999999 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.524267+0000", 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9796999999999995e-05 2024-09-13T14:43:52.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.524306+0000", 2024-09-13T14:43:52.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9301000000000002e-05 2024-09-13T14:43:52.479 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.479 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.479 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:28393 5.5 e117/110 5:a9fc8faa:::benchmark_data_smithi154_144140_object28392:head v 117'1765, mlcod=117'1763)", 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.391152+0000", 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 24.049344297000001, 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13223659700000001, 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391152+0000", 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391152+0000", 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391153+0000", 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1600000000000001e-07 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391179+0000", 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5596999999999999e-05 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391190+0000", 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0933e-05 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391196+0000", 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0630000000000002e-06 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520426+0000", 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12923024699999999 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520446+0000", 2024-09-13T14:43:52.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0021e-05 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.522831+0000", 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023848239999999998 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.523389+0000", 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055829600000000001 2024-09-13T14:43:52.485 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:28390 5.7 e117/111 ondisk, result = 0)", 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.391808+0000", 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 24.048688247000001, 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12972746900000001, 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391808+0000", 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391808+0000", 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391809+0000", 2024-09-13T14:43:52.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.61e-07 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391816+0000", 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7050000000000002e-06 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520321+0000", 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12850509199999999 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520350+0000", 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9482e-05 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.521516+0000", 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001165941 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.521523+0000", 2024-09-13T14:43:52.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8009999999999998e-06 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.521536+0000", 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2687000000000001e-05 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:28399 5.8 e117/110 5:1fdf9668:::benchmark_data_smithi154_144140_object28398:head v 117'1760, mlcod=117'1758)", 2024-09-13T14:43:52.490 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.393148+0000", 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 24.047349046000001, 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13215937699999999, 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393148+0000", 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393148+0000", 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393149+0000", 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0100000000000001e-06 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393173+0000", 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4286000000000001e-05 2024-09-13T14:43:52.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520323+0000", 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12714977899999999 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520345+0000", 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2079999999999999e-05 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520391+0000", 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5933999999999998e-05 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520410+0000", 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9037000000000001e-05 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.525261+0000", 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048512160000000002 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.525307+0000", 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6035000000000002e-05 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:31414 5.c 5:32316561:::benchmark_data_smithi154_144140_object31413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:29.988061+0000", 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 22.452435804, 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.060942238000000003, 2024-09-13T14:43:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 31414 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988061+0000", 2024-09-13T14:43:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988061+0000", 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988062+0000", 2024-09-13T14:43:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2389999999999999e-06 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988083+0000", 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1288e-05 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988096+0000", 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2113000000000001e-05 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988103+0000", 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7700000000000001e-06 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988184+0000", 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0642999999999993e-05 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988326+0000", 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014156499999999999 2024-09-13T14:43:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.988459+0000", 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013305200000000001 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.989573+0000", 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001114227 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:29.990153+0000", 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058022 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:30.048844+0000", 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058691015999999999 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:30.048948+0000", 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103606 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:30.049003+0000", 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5498999999999998e-05 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:33382 5.c 5:331f90e5:::benchmark_data_smithi154_144140_object33381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.019189+0000", 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.421307914, 2024-09-13T14:43:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.066703336000000002, 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 33382 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.019189+0000", 2024-09-13T14:43:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.019189+0000", 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.019190+0000", 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4399999999999999e-07 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.019211+0000", 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1539000000000002e-05 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.019222+0000", 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0497e-05 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.019229+0000", 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3189999999999997e-06 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.022114+0000", 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028846050000000002 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.022228+0000", 2024-09-13T14:43:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114032 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.022352+0000", 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012485299999999999 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023677+0000", 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013246639999999999 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.024251+0000", 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057428899999999996 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.028006+0000", 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037545510000000001 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085850+0000", 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.057844263 2024-09-13T14:43:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085892+0000", 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1879999999999999e-05 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:33389 5.7 5:e035bf50:::benchmark_data_smithi154_144140_object33388:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.023454+0000", 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.417043159999999, 2024-09-13T14:43:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.062774194000000005, 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 33389 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023454+0000", 2024-09-13T14:43:52.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023454+0000", 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023455+0000", 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4389999999999999e-06 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023476+0000", 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0777000000000001e-05 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023488+0000", 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.181e-05 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023495+0000", 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0319999999999998e-06 2024-09-13T14:43:52.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.024570+0000", 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001075679 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.024691+0000", 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012099000000000001 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.024819+0000", 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012738699999999999 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027580+0000", 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027613350000000002 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085912+0000", 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0583317 2024-09-13T14:43:52.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086005+0000", 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3362000000000005e-05 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086169+0000", 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016401400000000001 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086228+0000", 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8668999999999997e-05 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:33392 5.7 5:e319ff91:::benchmark_data_smithi154_144140_object33391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.023804+0000", 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.416693031000001, 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.062547601999999994, 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.516 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 33392 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023804+0000", 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023804+0000", 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023804+0000", 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7299999999999995e-07 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023823+0000", 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9083999999999999e-05 2024-09-13T14:43:52.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023834+0000", 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.066e-05 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.023841+0000", 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4509999999999997e-06 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026119+0000", 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022784189999999998 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026228+0000", 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010925999999999999 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026356+0000", 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012796500000000001 2024-09-13T14:43:52.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085890+0000", 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059533371000000002 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085920+0000", 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0858000000000002e-05 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086272+0000", 2024-09-13T14:43:52.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000351481 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086317+0000", 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5351e-05 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086351+0000", 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4029000000000002e-05 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.522 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:33396 5.0 e117/110 5:0e3ab43d:::benchmark_data_smithi154_144140_object33395:head v 117'2103, mlcod=117'2099)", 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.026668+0000", 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.413828922, 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059246264, 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.523 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.524 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:52.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:52.525 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026668+0000", 2024-09-13T14:43:52.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026668+0000", 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026669+0000", 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9599999999999998e-07 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026702+0000", 2024-09-13T14:43:52.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3031999999999998e-05 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026713+0000", 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0891e-05 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026719+0000", 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8079999999999999e-06 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026946+0000", 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000227037 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026965+0000", 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8206000000000002e-05 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085810+0000", 2024-09-13T14:43:52.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058845438 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085914+0000", 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010395600000000001 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:33399 5.7 5:e05c6bb1:::benchmark_data_smithi154_144140_object33398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.027289+0000", 2024-09-13T14:43:52.529 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.413207231000001, 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059202203000000002, 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 33399 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027289+0000", 2024-09-13T14:43:52.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027289+0000", 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027291+0000", 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.288e-06 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027311+0000", 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0064999999999998e-05 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027322+0000", 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1039000000000001e-05 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027329+0000", 2024-09-13T14:43:52.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6460000000000004e-06 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027610+0000", 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028118000000000001 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027716+0000", 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106109 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027831+0000", 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115383 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085964+0000", 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058132899000000002 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086397+0000", 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043335900000000002 2024-09-13T14:43:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086410+0000", 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2490000000000001e-05 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086454+0000", 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4477999999999997e-05 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086492+0000", 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7267000000000001e-05 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:33382 5.c e117/110 ondisk, result = 0)", 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.027879+0000", 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.412617488999999, 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058018687999999999, 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.536 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027879+0000", 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027879+0000", 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027880+0000", 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1900000000000002e-07 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027884+0000", 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7249999999999999e-06 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027901+0000", 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.732e-05 2024-09-13T14:43:52.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027907+0000", 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.117e-06 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.027995+0000", 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7570000000000004e-05 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.028001+0000", 2024-09-13T14:43:52.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4060000000000002e-06 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085898+0000", 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.057896831000000003 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:33401 5.5 e117/110 5:a33ed10c:::benchmark_data_smithi154_144140_object33400:head v 117'2062, mlcod=117'2060)", 2024-09-13T14:43:52.540 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.028740+0000", 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.411757069, 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059170970000000003, 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.028740+0000", 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.028740+0000", 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.028740+0000", 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9999999999999997e-07 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.028764+0000", 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3609999999999999e-05 2024-09-13T14:43:52.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085818+0000", 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.057054474000000001 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085829+0000", 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0399e-05 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085931+0000", 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102278 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085965+0000", 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3869999999999999e-05 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.087866+0000", 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001900559 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.087911+0000", 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5080000000000002e-05 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:33403 5.0 e117/110 5:094967b0:::benchmark_data_smithi154_144140_object33402:head v 117'2104, mlcod=117'2101)", 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.029699+0000", 2024-09-13T14:43:52.545 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.410797877, 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.057798975000000002, 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029699+0000", 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029699+0000", 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029700+0000", 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3e-07 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029732+0000", 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2183e-05 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085803+0000", 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.056070940999999999 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085815+0000", 2024-09-13T14:43:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2048000000000001e-05 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085940+0000", 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012521400000000001 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.085958+0000", 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8527999999999999e-05 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.087411+0000", 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014526649999999999 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.087498+0000", 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6666e-05 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:64805 5.7 5:e3e2982d:::benchmark_data_smithi154_144140_object64804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:50.818395+0000", 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.622102071, 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.061261628999999998, 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 64805 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818395+0000", 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818395+0000", 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818396+0000", 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3030000000000001e-06 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818417+0000", 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1168000000000001e-05 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818427+0000", 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7880000000000001e-06 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818433+0000", 2024-09-13T14:43:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4710000000000004e-06 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818485+0000", 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1981000000000003e-05 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818607+0000", 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121989 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818732+0000", 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012464600000000001 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879504+0000", 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.060771777999999999 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879517+0000", 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3044000000000001e-05 2024-09-13T14:43:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879558+0000", 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1486999999999998e-05 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879612+0000", 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3900000000000002e-05 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879656+0000", 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4073999999999998e-05 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:64806 5.7 5:e115aca4:::benchmark_data_smithi154_144140_object64805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:50.818624+0000", 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.6218730319999999, 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.061284365, 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 64806 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818624+0000", 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818624+0000", 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818625+0000", 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0500000000000002e-07 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818645+0000", 2024-09-13T14:43:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0523e-05 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818655+0000", 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8009999999999998e-06 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.818662+0000", 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9940000000000003e-06 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821706+0000", 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030445559999999999 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821823+0000", 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116848 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821953+0000", 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000129321 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879766+0000", 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.057813161000000002 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879784+0000", 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7847e-05 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879817+0000", 2024-09-13T14:43:52.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2889999999999999e-05 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879874+0000", 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7562999999999998e-05 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879908+0000", 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3955999999999998e-05 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.565 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:64810 5.7 5:e8336deb:::benchmark_data_smithi154_144140_object64809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:50.821084+0000", 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.6194126099999999, 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058974898999999997, 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 64810 2024-09-13T14:43:52.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821084+0000", 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821084+0000", 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821085+0000", 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5499999999999998e-07 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821103+0000", 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8383999999999999e-05 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821111+0000", 2024-09-13T14:43:52.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9920000000000007e-06 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.821117+0000", 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5520000000000004e-06 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.824191+0000", 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030745389999999998 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:43:52.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.824321+0000", 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013017700000000001 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.824452+0000", 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130856 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879788+0000", 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055336033 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879955+0000", 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000166756 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.879986+0000", 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1173999999999997e-05 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.880029+0000", 2024-09-13T14:43:52.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3208999999999999e-05 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.880059+0000", 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9572000000000001e-05 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:43:52.572 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:43:52.573 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 35 16 66884 66868 119.36 76.5625 0.0236521 0.0083711 2024-09-13T14:43:52.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:52.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:52.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:52.791 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:52.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:52.914 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:52.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:53.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:53.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:53.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:53.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:53.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:53.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:53.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:53.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:53.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:53.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 36 16 67917 67901 117.838 64.5625 0.0106525 0.00847991 2024-09-13T14:43:53.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:53.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:53.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:53.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:53.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:53.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:53.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:53.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:53.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:54.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:54.078 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:54.093 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:53 smithi154 ceph-mon[102314]: pgmap v323: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:43:54.094 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:53 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2556276992' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:54.094 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:53 smithi154 ceph-mon[107038]: pgmap v323: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:43:54.094 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:53 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2556276992' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:54.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:54.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:54.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:53 smithi172 ceph-mon[100696]: pgmap v323: 89 pgs: 89 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T14:43:54.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:53 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2556276992' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:43:54.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:54.359 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:54.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:54.492 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:54.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 37 16 69149 69133 116.734 77 0.0232858 0.00856092 2024-09-13T14:43:54.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:54.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:54.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:54.754 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:54.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:54.894 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:54.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:54.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:54 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:54.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:54 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:43:54.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2556276992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:54.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:54 smithi172 ceph-mon[100696]: osdmap e120: 8 total, 8 up, 7 in 2024-09-13T14:43:54.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:55.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:55.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:55.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:43:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2556276992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[102314]: osdmap e120: 8 total, 8 up, 7 in 2024-09-13T14:43:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:43:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:43:55.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2556276992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:43:55.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:54 smithi154 ceph-mon[107038]: osdmap e120: 8 total, 8 up, 7 in 2024-09-13T14:43:55.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:55.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:55.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:55.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 38 16 70434 70418 115.776 80.3125 0.00684067 0.00863241 2024-09-13T14:43:55.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:55.541 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:43:55.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:55.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:55.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:55.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:55.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:55.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:55.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:55.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:56.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:56.124 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:55 smithi154 ceph-mon[102314]: pgmap v325: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-13T14:43:56.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:55 smithi154 ceph-mon[107038]: pgmap v325: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-13T14:43:56.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:56.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:55 smithi172 ceph-mon[100696]: pgmap v325: 89 pgs: 89 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-13T14:43:56.261 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:56.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:56.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:56.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:56.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 39 16 71754 71738 114.923 82.5 0.00795322 0.00869636 2024-09-13T14:43:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:56.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:56.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:56.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:56.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:56.823 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:56.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:56.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:56.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:57.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:57.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:57.139 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:57.139 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:57.140 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:57.140 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:57.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:43:57.189 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:57.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:57.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:57.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:57.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:57.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:57.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:57.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:43:57.528440+0000 min lat: 0.00117109 max lat: 0.172776 avg lat: 0.00876606 2024-09-13T14:43:57.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:43:57.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 40 16 73009 72993 114.011 78.4375 0.00724022 0.00876606 2024-09-13T14:43:57.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:43:57.626 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:43:57.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:57.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:57.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:57.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:57.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:57.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:58.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:58.060 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:58.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:43:58.192 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:43:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:58 smithi154 ceph-mon[102314]: pgmap v326: 89 pgs: 89 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:43:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:43:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:58.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:58 smithi154 ceph-mon[107038]: pgmap v326: 89 pgs: 89 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:43:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:43:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:58.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:58.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:58.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:58.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:58 smithi172 ceph-mon[100696]: pgmap v326: 89 pgs: 89 active+clean; 4.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-13T14:43:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:43:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:43:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:43:58.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 41 16 74259 74243 113.136 78.125 0.00812458 0.00883414 2024-09-13T14:43:58.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:58.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:58.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:58.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:58.827 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:43:58.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:43:58.866 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:58.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:43:58.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:59.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:59.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:43:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:59.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:43:59.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:43:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:59.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:43:59.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:59.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 42 16 75490 75474 112.274 76.9375 0.0223823 0.00889991 2024-09-13T14:43:59.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:43:59.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:43:59.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:43:59.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:43:59.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:43:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:43:59.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:43:59.880 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:43:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:43:59.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:43:59.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:00.009 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:00.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:00.138 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:00.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:00 smithi154 ceph-mon[102314]: pgmap v327: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:00.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:00 smithi154 ceph-mon[107038]: pgmap v327: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:00.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:00.265 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:00.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:00 smithi172 ceph-mon[100696]: pgmap v327: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:44:00.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:00.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 43 16 76709 76693 111.435 76.1875 0.0152871 0.00896906 2024-09-13T14:44:00.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:00.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:00.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:00.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:00.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:00.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:00.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:00.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:01.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:01.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:01.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:01.244 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:01.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:01.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:01.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:01.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:01.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 44 16 77986 77970 110.716 79.8125 0.00800668 0.00902684 2024-09-13T14:44:01.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:01.632 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:01.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:01.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:01.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:01.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:01.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:01.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:02.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:02.138 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:02 smithi154 ceph-mon[102314]: pgmap v328: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:02.138 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:02 smithi154 ceph-mon[107038]: pgmap v328: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:02.139 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:44:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:44:01] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:44:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.176 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:02.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:02.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:02.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:02.407 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:02 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T14:44:02.291+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e120 definitely_dead 0 2024-09-13T14:44:02.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:02.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:02 smithi172 ceph-mon[100696]: pgmap v328: 89 pgs: 89 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:02.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 45 16 79100 79084 109.803 69.625 0.0180628 0.0091023 2024-09-13T14:44:02.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:02.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:02.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:02.690 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:44:02.690 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:44:02.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.690 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:79243 5.8 5:176a7d7e:::benchmark_data_smithi154_144140_object79242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-13T14:44:02.690 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.668971+0000", 2024-09-13T14:44:02.690 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.012479838, 2024-09-13T14:44:02.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012523282, 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 79243 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.668971+0000", 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.668971+0000", 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.668972+0000", 2024-09-13T14:44:02.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.195e-06 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.669001+0000", 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8541999999999999e-05 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.669011+0000", 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9760000000000007e-06 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.669017+0000", 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.302e-06 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.669074+0000", 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6632000000000003e-05 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:02.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.669207+0000", 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013346300000000001 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.669341+0000", 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000133676 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.672978+0000", 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036375349999999999 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676822+0000", 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038437010000000001 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:79247 5.6 5:6cbafe3b:::benchmark_data_smithi154_144140_object79246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.676612+0000", 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0048388320000000004, 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004976383, 2024-09-13T14:44:02.696 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 79247 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676612+0000", 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676612+0000", 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676613+0000", 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1519999999999999e-06 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676642+0000", 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8779999999999999e-05 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676652+0000", 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0074e-05 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676659+0000", 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1539999999999996e-06 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676770+0000", 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011084800000000001 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676925+0000", 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015443100000000001 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.677075+0000", 2024-09-13T14:44:02.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015060200000000001 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:79250 5.6 5:67a18610:::benchmark_data_smithi154_144140_object79249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.677344+0000", 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0041068859999999997, 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044347450000000004, 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:44:02.701 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 79250 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.677344+0000", 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.677344+0000", 2024-09-13T14:44:02.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.677345+0000", 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0100000000000003e-07 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.678598+0000", 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012534849999999999 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.678615+0000", 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6956000000000002e-05 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.678625+0000", 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6630000000000005e-06 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.679959+0000", 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001333877 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680090+0000", 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013081200000000001 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680222+0000", 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013193800000000001 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:79247 5.6 e120/111)", 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.680080+0000", 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.001370926, 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001818004, 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680080+0000", 2024-09-13T14:44:02.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680080+0000", 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680081+0000", 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.85e-07 2024-09-13T14:44:02.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680090+0000", 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0599999999999997e-06 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680104+0000", 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3791999999999999e-05 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.680111+0000", 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3769999999999998e-06 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:79238 5.6 e120/111)", 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.681047+0000", 2024-09-13T14:44:02.709 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.000404041, 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088928599999999998, 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681047+0000", 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681047+0000", 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681048+0000", 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6000000000000003e-07 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681055+0000", 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7060000000000004e-06 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681070+0000", 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4435e-05 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681076+0000", 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4930000000000003e-06 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:79247 5.6 e120/111)", 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.681100+0000", 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.000350854, 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088051799999999995, 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681100+0000", 2024-09-13T14:44:02.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681100+0000", 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681100+0000", 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.58e-07 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681103+0000", 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8490000000000002e-06 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681110+0000", 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5799999999999997e-06 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681114+0000", 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5120000000000002e-06 2024-09-13T14:44:02.715 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:79235 5.8 e120/110)", 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.681580+0000", 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4294967295.9998713, 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000445201, 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681580+0000", 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681580+0000", 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681581+0000", 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1999999999999999e-07 2024-09-13T14:44:02.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681585+0000", 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6360000000000002e-06 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681595+0000", 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0285999999999999e-05 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.681601+0000", 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9769999999999997e-06 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:79235 5.8 5:1f986a2d:::benchmark_data_smithi154_144140_object79234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.659239+0000", 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.022211499999999999, 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022832357000000001, 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.719 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 79235 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659239+0000", 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659239+0000", 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659242+0000", 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3360000000000002e-06 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659281+0000", 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9270000000000002e-05 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659297+0000", 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6325999999999999e-05 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659306+0000", 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6759999999999996e-06 2024-09-13T14:44:02.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659470+0000", 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016416900000000001 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659617+0000", 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000146713 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659753+0000", 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000136425 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.663691+0000", 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039376680000000001 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.663708+0000", 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6541e-05 2024-09-13T14:44:02.724 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:79238 5.6 5:6cb19148:::benchmark_data_smithi154_144140_object79237:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:02.659335+0000", 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.022115883999999999, 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022820764, 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:44:02.725 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 79238 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659335+0000", 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:02.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659335+0000", 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659336+0000", 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7400000000000003e-07 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659358+0000", 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2348e-05 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659366+0000", 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0490000000000004e-06 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.659372+0000", 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9730000000000002e-06 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.672647+0000", 2024-09-13T14:44:02.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013275215 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.672795+0000", 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014808599999999999 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.673252+0000", 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00045692999999999998 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676595+0000", 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033423799999999998 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:02.676606+0000", 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0992000000000001e-05 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:44:02.730 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 9 2024-09-13T14:44:02.731 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:44:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:02.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:02.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:02.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:02.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:02.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:02.963 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:44:02.963 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:03.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:03.011 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:44:03.011 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:03.051 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:44:03.052 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:03.082 INFO:teuthology.orchestra.run.smithi154.stderr:marked out osd.6. 2024-09-13T14:44:03.093 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:44:03.093 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:03.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:03.128 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:44:03.128 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:03.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:03.154 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:44:03.154 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:03.178 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:44:03.179 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:03.190 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:44:03.190 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:03.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3782106202' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:44:03.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:44:03.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[102314]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T14:44:03.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/269557556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:03.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3782106202' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:44:03.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:44:03.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[107038]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T14:44:03.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/269557556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:03.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:03.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:03.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:03.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:03.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:03.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3782106202' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:44:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:03 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:44:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:03 smithi172 ceph-mon[100696]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T14:44:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/269557556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:03.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:03.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 46 16 79736 79720 108.28 39.75 0.00317131 0.00914861 2024-09-13T14:44:03.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:03.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:03.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:03.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:03.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:03.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:03.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:03.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:03.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:04.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:04.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.138 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:04.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:04.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:04.263 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:04.304 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:04 smithi154 ceph-mon[102314]: pgmap v329: 89 pgs: 89 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:04.304 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:04 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T14:44:04.304 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:04 smithi154 ceph-mon[102314]: osdmap e121: 8 total, 8 up, 6 in 2024-09-13T14:44:04.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:04 smithi154 ceph-mon[107038]: pgmap v329: 89 pgs: 89 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:04.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:04 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T14:44:04.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:04 smithi154 ceph-mon[107038]: osdmap e121: 8 total, 8 up, 6 in 2024-09-13T14:44:04.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:44:04.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:04.388 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:04.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:04 smithi172 ceph-mon[100696]: pgmap v329: 89 pgs: 89 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:04 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T14:44:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:04 smithi172 ceph-mon[100696]: osdmap e121: 8 total, 8 up, 6 in 2024-09-13T14:44:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:04.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:04.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 47 16 79736 79720 105.977 0 - 0.00914861 2024-09-13T14:44:04.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:04.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:04.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:04.764 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:04.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:04.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:04.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:04.892 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:04.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:05.016 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:05.119 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:05.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:05.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:05.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:05 smithi154 ceph-mon[107038]: osdmap e122: 8 total, 8 up, 6 in 2024-09-13T14:44:05.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:05 smithi154 ceph-mon[107038]: osdmap e123: 8 total, 8 up, 6 in 2024-09-13T14:44:05.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:05 smithi154 ceph-mon[102314]: osdmap e122: 8 total, 8 up, 6 in 2024-09-13T14:44:05.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:05 smithi154 ceph-mon[102314]: osdmap e123: 8 total, 8 up, 6 in 2024-09-13T14:44:05.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:05.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:05.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:05.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:05 smithi172 ceph-mon[100696]: osdmap e122: 8 total, 8 up, 6 in 2024-09-13T14:44:05.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:05 smithi172 ceph-mon[100696]: osdmap e123: 8 total, 8 up, 6 in 2024-09-13T14:44:05.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:05.529 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 48 16 79945 79929 104.04 6.53125 0.0246197 0.00960469 2024-09-13T14:44:05.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:05.629 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:05.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:05.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:05.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:05.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:05.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:05.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:06.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:06.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:06.130 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:06.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:06 smithi154 ceph-mon[102314]: pgmap v332: 89 pgs: 1 unknown, 18 peering, 70 active+clean; 4.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:06.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:06 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:44:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:06 smithi154 ceph-mon[107038]: pgmap v332: 89 pgs: 1 unknown, 18 peering, 70 active+clean; 4.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:06 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:44:06.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:06.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:06.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:06.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:06 smithi172 ceph-mon[100696]: pgmap v332: 89 pgs: 1 unknown, 18 peering, 70 active+clean; 4.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-13T14:44:06.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:06 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-13T14:44:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:06.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:06.529 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 49 16 80764 80748 102.962 51.1875 0.0257648 0.00970618 2024-09-13T14:44:06.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:06.575 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:06.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:06.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:06.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:06.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:06.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:06.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:06.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:06.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:07.154 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:07 smithi172 ceph-mon[100696]: osdmap e124: 8 total, 8 up, 6 in 2024-09-13T14:44:07.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:07.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:07.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:07.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:07.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:07.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:07 smithi154 ceph-mon[102314]: osdmap e124: 8 total, 8 up, 6 in 2024-09-13T14:44:07.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:07 smithi154 ceph-mon[107038]: osdmap e124: 8 total, 8 up, 6 in 2024-09-13T14:44:07.529 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 50 16 81577 81561 101.919 50.8125 0.00596453 0.00980576 2024-09-13T14:44:07.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:07.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:07.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:07.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:07.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:07.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:07.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:07.982 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:07.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:08.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:08.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:08.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:08.250 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:08.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:08.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:08.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:08 smithi154 ceph-mon[102314]: pgmap v335: 89 pgs: 1 remapped+peering, 1 unknown, 18 peering, 69 active+clean; 4.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s; 4 B/s, 1 objects/s recovering 2024-09-13T14:44:08.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:08 smithi154 ceph-mon[102314]: osdmap e125: 8 total, 8 up, 6 in 2024-09-13T14:44:08.394 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:08 smithi154 ceph-mon[107038]: pgmap v335: 89 pgs: 1 remapped+peering, 1 unknown, 18 peering, 69 active+clean; 4.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s; 4 B/s, 1 objects/s recovering 2024-09-13T14:44:08.394 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:08 smithi154 ceph-mon[107038]: osdmap e125: 8 total, 8 up, 6 in 2024-09-13T14:44:08.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:08.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:08 smithi172 ceph-mon[100696]: pgmap v335: 89 pgs: 1 remapped+peering, 1 unknown, 18 peering, 69 active+clean; 4.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 67 MiB/s wr, 1.07k op/s; 4 B/s, 1 objects/s recovering 2024-09-13T14:44:08.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:08 smithi172 ceph-mon[100696]: osdmap e125: 8 total, 8 up, 6 in 2024-09-13T14:44:08.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:08.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:08.529 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 51 16 82405 82389 100.935 51.75 0.0278204 0.00990155 2024-09-13T14:44:08.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:08.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:08.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:08.802 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:08.809 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:08.809 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:08.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:08.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:08.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:09.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:09.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:09.160 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:44:09.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:09.221 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:09.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:09.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:09.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:09 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2721053812' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:09 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2721053812' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:09.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:09.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:09 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2721053812' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:44:09.499 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:09.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:09.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 52 16 83049 83033 99.7686 40.25 0.018001 0.0100168 2024-09-13T14:44:09.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:09.627 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:09.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:09.754 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:09.763 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:44:09.764 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:44:09.764 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T14:44:09.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:44:09.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:09.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:09.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:10.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:10.033 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:10.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:10.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:10.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:10.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:10.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[102314]: pgmap v337: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 84 active+clean; 5.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 42 MiB/s wr, 668 op/s; 10071/244800 objects degraded (4.114%); 9644/244800 objects misplaced (3.940%); 2.4 MiB/s, 44 objects/s recovering 2024-09-13T14:44:10.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:10.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 10071/244800 objects degraded (4.114%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:10.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:44:10.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2721053812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:10.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[102314]: osdmap e126: 8 total, 8 up, 6 in 2024-09-13T14:44:10.376 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[107038]: pgmap v337: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 84 active+clean; 5.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 42 MiB/s wr, 668 op/s; 10071/244800 objects degraded (4.114%); 9644/244800 objects misplaced (3.940%); 2.4 MiB/s, 44 objects/s recovering 2024-09-13T14:44:10.376 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:10.376 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 10071/244800 objects degraded (4.114%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:10.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:44:10.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2721053812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:10.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:10 smithi154 ceph-mon[107038]: osdmap e126: 8 total, 8 up, 6 in 2024-09-13T14:44:10.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:44:10.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:10.441 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:10.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:10 smithi172 ceph-mon[100696]: pgmap v337: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 remapped+peering, 84 active+clean; 5.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 42 MiB/s wr, 668 op/s; 10071/244800 objects degraded (4.114%); 9644/244800 objects misplaced (3.940%); 2.4 MiB/s, 44 objects/s recovering 2024-09-13T14:44:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:10 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:10 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 10071/244800 objects degraded (4.114%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:10 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-13T14:44:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:10 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2721053812' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:44:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:10 smithi172 ceph-mon[100696]: osdmap e126: 8 total, 8 up, 6 in 2024-09-13T14:44:10.533 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 53 16 83744 83728 98.7059 43.4375 0.0142273 0.0101236 2024-09-13T14:44:10.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:10.565 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:10.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:10.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:10.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:10.820 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:10.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:10.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:10.958 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:11.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:11.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:11.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:11.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:11.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:11.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:11.397 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:11.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:11.525 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:11.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 54 16 84501 84485 97.7543 47.3125 0.0426922 0.0102242 2024-09-13T14:44:11.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:11.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:11.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:11.809 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:11.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:11.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:11.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:12.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:12.074 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:12.152 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:44:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:44:11] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:44:12.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:12.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:12.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:12.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:12.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:12 smithi154 ceph-mon[102314]: pgmap v339: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 33 MiB/s wr, 529 op/s; 10071/246696 objects degraded (4.082%); 9644/246696 objects misplaced (3.909%); 2.0 MiB/s, 36 objects/s recovering 2024-09-13T14:44:12.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:12 smithi154 ceph-mon[107038]: pgmap v339: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 33 MiB/s wr, 529 op/s; 10071/246696 objects degraded (4.082%); 9644/246696 objects misplaced (3.909%); 2.0 MiB/s, 36 objects/s recovering 2024-09-13T14:44:12.473 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:12.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:12 smithi172 ceph-mon[100696]: pgmap v339: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 33 MiB/s wr, 529 op/s; 10071/246696 objects degraded (4.082%); 9644/246696 objects misplaced (3.909%); 2.0 MiB/s, 36 objects/s recovering 2024-09-13T14:44:12.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 55 16 85370 85354 96.9645 54.3125 0.0235776 0.0103084 2024-09-13T14:44:12.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:12.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:12.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:12.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:12.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:12.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:12.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:44:12.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:12.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:12.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:13.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:13.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:13.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:13.251 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:13.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:13.379 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:13.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:13.504 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:13.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 56 16 86029 86013 95.9686 41.1875 0.00485809 0.0104127 2024-09-13T14:44:13.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:13.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:13.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:13.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:13.793 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:13.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:13.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:13.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:14.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:14.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:14.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.218 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:14.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:13 smithi154 ceph-mon[102314]: pgmap v340: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 35 MiB/s wr, 559 op/s; 10071/248832 objects degraded (4.047%); 9525/248832 objects misplaced (3.828%); 2.8 MiB/s, 48 objects/s recovering 2024-09-13T14:44:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:13 smithi154 ceph-mon[107038]: pgmap v340: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 35 MiB/s wr, 559 op/s; 10071/248832 objects degraded (4.047%); 9525/248832 objects misplaced (3.828%); 2.8 MiB/s, 48 objects/s recovering 2024-09-13T14:44:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:13 smithi172 ceph-mon[100696]: pgmap v340: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 35 MiB/s wr, 559 op/s; 10071/248832 objects degraded (4.047%); 9525/248832 objects misplaced (3.828%); 2.8 MiB/s, 48 objects/s recovering 2024-09-13T14:44:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:14.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:14.351 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:14.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:14.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:14.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 57 16 86820 86804 95.1524 49.4375 0.0110403 0.0105034 2024-09-13T14:44:14.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:14.620 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:14.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:14.749 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:14.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:14.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:14.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:14.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:15.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:15.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:15.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:15.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:14 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/248832 objects degraded (4.047%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:14 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/248832 objects degraded (4.047%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:14 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/248832 objects degraded (4.047%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:15.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:15.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:15.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:15.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:15.532 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 58 16 87599 87583 94.3513 48.6875 0.0149509 0.0105929 2024-09-13T14:44:15.532 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:15.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:15.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:15.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:15.756 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:15.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:15.884 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:15.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:15.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:16.012 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:16.019 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:16.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:16.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:15 smithi172 ceph-mon[100696]: pgmap v341: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 49 MiB/s wr, 780 op/s; 10071/256374 objects degraded (3.928%); 9353/256374 objects misplaced (3.648%); 3.7 MiB/s, 62 objects/s recovering 2024-09-13T14:44:16.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:16.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:16.417 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:15 smithi154 ceph-mon[102314]: pgmap v341: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 49 MiB/s wr, 780 op/s; 10071/256374 objects degraded (3.928%); 9353/256374 objects misplaced (3.648%); 3.7 MiB/s, 62 objects/s recovering 2024-09-13T14:44:16.417 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:15 smithi154 ceph-mon[107038]: pgmap v341: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 49 MiB/s wr, 780 op/s; 10071/256374 objects degraded (3.928%); 9353/256374 objects misplaced (3.648%); 3.7 MiB/s, 62 objects/s recovering 2024-09-13T14:44:16.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:16.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:16.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 59 16 88356 88340 93.5542 47.3125 0.0138782 0.010682 2024-09-13T14:44:16.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:16.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:16.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:16.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:16.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:16.881 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:16.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:16.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:17.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:17.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:17.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:17.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:17.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:17.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:17.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:17.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:44:17.530834+0000 min lat: 0.00117109 max lat: 2.26349 avg lat: 0.0107651 2024-09-13T14:44:17.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:44:17.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 60 16 89172 89156 92.845 51 0.0131283 0.0107651 2024-09-13T14:44:17.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:17.561 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:17.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:17.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:17.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:17.838 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:17.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:17.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:17.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:18.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:18.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:18.191 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:17 smithi154 ceph-mon[102314]: pgmap v342: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 38 MiB/s wr, 615 op/s; 10071/257757 objects degraded (3.907%); 9353/257757 objects misplaced (3.629%); 2.8 MiB/s, 47 objects/s recovering 2024-09-13T14:44:18.191 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:17 smithi154 ceph-mon[107038]: pgmap v342: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 38 MiB/s wr, 615 op/s; 10071/257757 objects degraded (3.907%); 9353/257757 objects misplaced (3.629%); 2.8 MiB/s, 47 objects/s recovering 2024-09-13T14:44:18.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:18.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:17 smithi172 ceph-mon[100696]: pgmap v342: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.2 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 38 MiB/s wr, 615 op/s; 10071/257757 objects degraded (3.907%); 9353/257757 objects misplaced (3.629%); 2.8 MiB/s, 47 objects/s recovering 2024-09-13T14:44:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:18.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:18.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:18.395 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:18.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:18.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:18.522 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:18.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 61 16 89976 89960 92.1467 50.25 0.0088698 0.0108461 2024-09-13T14:44:18.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:18.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:18.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:18.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:18.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:18.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:18.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:19.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:19.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:19.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:19.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:19.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:19.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:19.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:19.519 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:19.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 62 16 90789 90773 91.4801 50.8125 0.0133841 0.0109254 2024-09-13T14:44:19.557 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:44:19.558 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:19.595 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:44:19.596 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:19.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:19.635 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:44:19.636 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:19.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:19.670 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:44:19.670 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:19.711 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:44:19.711 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:19.757 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:44:19.757 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:19.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:19.806 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:44:19.806 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:19.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:19.845 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:44:19.846 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:19.924 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:19.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:19.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:20.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:20.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:20.203 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:20.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:20.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:19 smithi172 ceph-mon[100696]: pgmap v343: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 47 MiB/s wr, 753 op/s; 10071/267399 objects degraded (3.766%); 9039/267399 objects misplaced (3.380%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:44:20.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3483259253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:20.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:19 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/267399 objects degraded (3.766%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.273 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:20.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:20.303 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:19 smithi154 ceph-mon[102314]: pgmap v343: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 47 MiB/s wr, 753 op/s; 10071/267399 objects degraded (3.766%); 9039/267399 objects misplaced (3.380%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:44:20.303 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3483259253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:20.303 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:19 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/267399 objects degraded (3.766%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:20.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:19 smithi154 ceph-mon[107038]: pgmap v343: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.4 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 47 MiB/s wr, 753 op/s; 10071/267399 objects degraded (3.766%); 9039/267399 objects misplaced (3.380%); 3.8 MiB/s, 61 objects/s recovering 2024-09-13T14:44:20.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3483259253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:20.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:19 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/267399 objects degraded (3.766%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:20.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:20.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:20.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:20.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:20.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:20.530 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 63 16 91513 91497 90.7463 45.25 0.0404453 0.0110131 2024-09-13T14:44:20.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:20.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:20.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:20.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:20.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:20.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:20.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:21.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:21.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:21.201 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:21.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:21.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:21.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:21.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:21.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 64 15 92206 92191 90.006 43.375 0.00668842 0.0111046 2024-09-13T14:44:21.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:21.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:21.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:21.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:21.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:21.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:21.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:21.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:21.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:21.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:21.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:44:21.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:44:21.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:22.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:22.023 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:44:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:44:21] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:44:22.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:22.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:22.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:22.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.301 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:44:22.301 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27212 5.a 5:58ee1d02:::benchmark_data_smithi154_144140_object27211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.054938108000002, 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17096689200000001, 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.302 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27212 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664198+0000", 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9800000000000003e-07 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664220+0000", 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1591999999999999e-05 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664227+0000", 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.712e-06 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664233+0000", 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8289999999999996e-06 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669218+0000", 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049845829999999999 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669320+0000", 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102304 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669434+0000", 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113581 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674379+0000", 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00494525 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674700+0000", 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032070699999999999 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.834993+0000", 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160292992 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835121+0000", 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012801800000000001 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835164+0000", 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3726000000000001e-05 2024-09-13T14:44:22.308 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27213 5.a 5:571e06b1:::benchmark_data_smithi154_144140_object27212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.054894075999997, 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17104000799999999, 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.309 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27213 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664242+0000", 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2e-07 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664265+0000", 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3476e-05 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664273+0000", 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2230000000000001e-06 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664278+0000", 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1599999999999997e-06 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.670614+0000", 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063359059999999997 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671024+0000", 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041025499999999999 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671138+0000", 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114418 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674735+0000", 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035970080000000001 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674771+0000", 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5506999999999997e-05 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835177+0000", 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16040633400000001 2024-09-13T14:44:22.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835217+0000", 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9641000000000002e-05 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835281+0000", 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4659999999999994e-05 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27221 5.a 5:51807044:::benchmark_data_smithi154_144140_object27220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.050740515000001, 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16984976800000001, 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.316 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27221 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668396+0000", 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9999999999999997e-07 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668416+0000", 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0516000000000001e-05 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668424+0000", 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6389999999999992e-06 2024-09-13T14:44:22.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668430+0000", 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9939999999999997e-06 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672751+0000", 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043213569999999996 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673077+0000", 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032614299999999998 2024-09-13T14:44:22.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673198+0000", 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120492 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833603+0000", 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160404986 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835418+0000", 2024-09-13T14:44:22.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018152349999999999 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838152+0000", 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002733989 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838206+0000", 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4165000000000001e-05 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838245+0000", 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8652000000000002e-05 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27226 5.a 5:58f709a9:::benchmark_data_smithi154_144140_object27225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.049979805, 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.169199034, 2024-09-13T14:44:22.322 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27226 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1900000000000001e-07 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669187+0000", 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0571999999999998e-05 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669198+0000", 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0920999999999999e-05 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669205+0000", 2024-09-13T14:44:22.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6479999999999997e-06 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674834+0000", 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056293519999999998 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674951+0000", 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011713 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:44:22.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675069+0000", 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117726 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835380+0000", 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160310762 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835455+0000", 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5813000000000001e-05 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838286+0000", 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028307559999999998 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838325+0000", 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8574e-05 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838355+0000", 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9961e-05 2024-09-13T14:44:22.328 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27229 5.2 e117/111 5:47a3bf38:::benchmark_data_smithi154_144140_object27228:head v 117'1668, mlcod=117'1665)", 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.047903826999999, 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16704787199999999, 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5599999999999995e-07 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671265+0000", 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3231000000000001e-05 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671274+0000", 2024-09-13T14:44:22.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9479999999999997e-06 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671280+0000", 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672510+0000", 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012303030000000001 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672591+0000", 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1227e-05 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838248+0000", 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16565628499999999 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838280+0000", 2024-09-13T14:44:22.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1881999999999997e-05 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27212 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.047803858000002, 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16395237200000001, 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:44:22.334 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8700000000000001e-07 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671335+0000", 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.858e-06 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671342+0000", 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4189999999999996e-06 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671347+0000", 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9089999999999999e-06 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674368+0000", 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003020371 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674374+0000", 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2829999999999998e-06 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835284+0000", 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160910145 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.340 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27231 5.d 5:bd1c1b1b:::benchmark_data_smithi154_144140_object27230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.047484859000001, 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16226318100000001, 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27231 2024-09-13T14:44:22.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671652+0000", 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0100000000000003e-07 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671671+0000", 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9686000000000001e-05 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671680+0000", 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6989999999999992e-06 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671685+0000", 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2950000000000001e-06 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671954+0000", 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026847900000000002 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672066+0000", 2024-09-13T14:44:22.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011244 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672249+0000", 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000182573 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674493+0000", 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022442439999999998 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674564+0000", 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1538000000000005e-05 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675871+0000", 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013069449999999999 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833838+0000", 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15796623000000001 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833914+0000", 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6150999999999996e-05 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:44:22.347 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.044060170999998, 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158535863, 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675076+0000", 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5400000000000002e-07 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675079+0000", 2024-09-13T14:44:22.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.365e-06 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675087+0000", 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6249999999999998e-06 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675093+0000", 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8300000000000001e-06 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833585+0000", 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158492884 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833596+0000", 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0645999999999999e-05 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833611+0000", 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5058999999999999e-05 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.351 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27237 5.b 5:d9e8c993:::benchmark_data_smithi154_144140_object27236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.044051125999999, 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16640950199999999, 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.352 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27237 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:44:22.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675086+0000", 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.274e-06 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675108+0000", 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2412000000000002e-05 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675120+0000", 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1729000000000001e-05 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675127+0000", 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7680000000000003e-06 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675594+0000", 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000466992 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675717+0000", 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012363700000000001 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833850+0000", 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158133311 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836021+0000", 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021703880000000001 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836035+0000", 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3657e-05 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841412+0000", 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053777620000000003 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841458+0000", 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5235000000000003e-05 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841494+0000", 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6337000000000001e-05 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.358 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27239 5.2 e117/111 5:4246bfbf:::benchmark_data_smithi154_144140_object27238:head v 117'1669, mlcod=117'1665)", 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.043585247999999, 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16283244499999999, 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675551+0000", 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4000000000000001e-07 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675624+0000", 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3522000000000005e-05 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675633+0000", 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0499999999999997e-06 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675639+0000", 2024-09-13T14:44:22.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8839999999999999e-06 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833644+0000", 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158004476 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833663+0000", 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9238000000000001e-05 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838359+0000", 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046954900000000001 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838383+0000", 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4145e-05 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.363 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27231 5.d e117/111 ondisk, result = 0)", 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.043444966999999, 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15823179200000001, 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8500000000000001e-07 2024-09-13T14:44:22.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675695+0000", 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9099999999999998e-06 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675705+0000", 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6670000000000008e-06 2024-09-13T14:44:22.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675710+0000", 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7810000000000002e-06 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675848+0000", 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137677 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675855+0000", 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1790000000000002e-06 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833922+0000", 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15806699299999999 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27241 5.6 5:6b00c174:::benchmark_data_smithi154_144140_object27240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.368 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.043397433999999, 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16225515099999999, 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27241 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675739+0000", 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6499999999999999e-07 2024-09-13T14:44:22.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675758+0000", 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8799000000000001e-05 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675766+0000", 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4649999999999994e-06 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675772+0000", 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6470000000000003e-06 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675823+0000", 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1010000000000001e-05 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675943+0000", 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120389 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833416+0000", 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15747296699999999 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835725+0000", 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023092400000000002 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835762+0000", 2024-09-13T14:44:22.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6693000000000003e-05 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837162+0000", 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001400152 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837949+0000", 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000787417 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837993+0000", 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3707e-05 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27240 5.e e117/111 5:77f483c1:::benchmark_data_smithi154_144140_object27239:head v 117'1698, mlcod=117'1696)", 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.043385905999997, 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16227988300000001, 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.375 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675755+0000", 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5079999999999996e-06 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675810+0000", 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.452e-05 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675818+0000", 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8659999999999995e-06 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675824+0000", 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7930000000000003e-06 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675915+0000", 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0556000000000005e-05 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675932+0000", 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6979000000000001e-05 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837997+0000", 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16206567699999999 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838029+0000", 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1983999999999997e-05 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27226 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.042246273000003, 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15849907399999999, 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.380 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676890+0000", 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7599999999999997e-07 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676894+0000", 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1590000000000003e-06 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833403+0000", 2024-09-13T14:44:22.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.156509065 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833418+0000", 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5142e-05 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835366+0000", 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019481570000000001 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835374+0000", 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5639999999999999e-06 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835388+0000", 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4411e-05 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.042175069999999, 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15846376300000001, 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676961+0000", 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4199999999999995e-07 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676965+0000", 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1130000000000003e-06 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833821+0000", 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15685534100000001 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833830+0000", 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1130000000000007e-06 2024-09-13T14:44:22.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835407+0000", 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015777499999999999 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835413+0000", 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835424+0000", 2024-09-13T14:44:22.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0829000000000001e-05 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28399 5.8 5:1fdf9668:::benchmark_data_smithi154_144140_object28398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.392192+0000", 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 53.326943020999998, 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13349727, 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.389 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28399 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392192+0000", 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392192+0000", 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392194+0000", 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1939999999999999e-06 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392216+0000", 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2221e-05 2024-09-13T14:44:22.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392225+0000", 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5349999999999993e-06 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392232+0000", 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8650000000000003e-06 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392312+0000", 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9623999999999995e-05 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392436+0000", 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000124354 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:22.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392561+0000", 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012431000000000001 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.394826+0000", 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022652509999999998 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395718+0000", 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00089194899999999995 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.525595+0000", 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12987754600000001 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.525649+0000", 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3437999999999999e-05 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.525690+0000", 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0982999999999997e-05 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28404 5.8 5:14015ee2:::benchmark_data_smithi154_144140_object28403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.393487+0000", 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 53.325648661000002, 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13356032500000001, 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.396 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28404 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393487+0000", 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393487+0000", 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393488+0000", 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9600000000000008e-07 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393518+0000", 2024-09-13T14:44:22.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0213e-05 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393529+0000", 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0975000000000001e-05 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393536+0000", 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0609999999999998e-06 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393701+0000", 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016536 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393810+0000", 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108785 2024-09-13T14:44:22.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.393981+0000", 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017062699999999999 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.394950+0000", 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096888499999999995 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395755+0000", 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080562099999999996 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.526946+0000", 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.131190275 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527010+0000", 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4063e-05 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527047+0000", 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7463999999999999e-05 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28412 5.8 5:10c84a58:::benchmark_data_smithi154_144140_object28411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.395838+0000", 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 53.323297113000002, 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.131451965, 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28412 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395838+0000", 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395838+0000", 2024-09-13T14:44:22.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395839+0000", 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9699999999999995e-07 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395871+0000", 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1414000000000001e-05 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395883+0000", 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.207e-05 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395890+0000", 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5190000000000003e-06 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.395981+0000", 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0737999999999998e-05 2024-09-13T14:44:22.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.396086+0000", 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105006 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.396202+0000", 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115748 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.397442+0000", 2024-09-13T14:44:22.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012403749999999999 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.398825+0000", 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001383305 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527207+0000", 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12838203300000001 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527254+0000", 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6312e-05 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527290+0000", 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6647999999999999e-05 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28418 5.8 5:1b6e0207:::benchmark_data_smithi154_144140_object28417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.399561+0000", 2024-09-13T14:44:22.409 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 53.319573982999998, 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.127891961, 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28418 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399561+0000", 2024-09-13T14:44:22.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399561+0000", 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399562+0000", 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5700000000000001e-07 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399584+0000", 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1656999999999999e-05 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399594+0000", 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0057e-05 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399601+0000", 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7819999999999996e-06 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399657+0000", 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6440000000000002e-05 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399802+0000", 2024-09-13T14:44:22.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014476800000000001 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.399940+0000", 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013839200000000001 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.401165+0000", 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012244910000000001 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.401414+0000", 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000248712 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527359+0000", 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.125945324 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527421+0000", 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2420000000000002e-05 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527453+0000", 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2060999999999997e-05 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.416 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28437 5.8 5:1c2cf91b:::benchmark_data_smithi154_144140_object28436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.413272+0000", 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 53.305863047000003, 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.114549341, 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28437 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:22.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413272+0000", 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413272+0000", 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413273+0000", 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.92e-07 2024-09-13T14:44:22.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413292+0000", 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9182e-05 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413303+0000", 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0740999999999999e-05 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:22.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413308+0000", 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9289999999999997e-06 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413350+0000", 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1854000000000003e-05 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413433+0000", 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3458e-05 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:44:22.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.413522+0000", 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8920999999999998e-05 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.414460+0000", 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00093787600000000003 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.414635+0000", 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017480499999999999 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527729+0000", 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.113094139 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527780+0000", 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0939000000000001e-05 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:22.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.527822+0000", 2024-09-13T14:44:22.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1705000000000001e-05 2024-09-13T14:44:22.423 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.423 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.423 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.423 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:22.423 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:22.423 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:44:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:22.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 65 16 92919 92903 89.306 44.5 0.0143807 0.0111915 2024-09-13T14:44:22.551 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:22.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:22.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:22.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:22.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:22.946 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:22.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:22.969 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:22 smithi154 ceph-mon[102314]: pgmap v344: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 44 MiB/s wr, 709 op/s; 10071/269562 objects degraded (3.736%); 9039/269562 objects misplaced (3.353%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:44:22.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:22 smithi154 ceph-mon[107038]: pgmap v344: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 44 MiB/s wr, 709 op/s; 10071/269562 objects degraded (3.736%); 9039/269562 objects misplaced (3.353%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:44:22.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:22 smithi172 ceph-mon[100696]: pgmap v344: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 44 MiB/s wr, 709 op/s; 10071/269562 objects degraded (3.736%); 9039/269562 objects misplaced (3.353%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:44:23.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:23.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:23.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:23.215 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:23.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:44:23.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:23.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:23.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.519 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:23.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:23.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 66 16 93635 93619 88.6309 44.75 0.0489841 0.0112765 2024-09-13T14:44:23.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:23.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:23.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:23.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:23.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:23.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:23.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:24.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:24.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:24.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:24.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:24.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:24.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:24.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:24.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:44:24.513 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:44:24.520 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:24.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 67 16 94240 94224 87.8725 37.8125 0.0329997 0.0113729 2024-09-13T14:44:24.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:24.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:24.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:44:24.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:24.772 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:24 smithi154 ceph-mon[102314]: pgmap v345: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 43 MiB/s wr, 694 op/s; 10071/271719 objects degraded (3.706%); 8914/271719 objects misplaced (3.281%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:44:24.772 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:24 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/271719 objects degraded (3.706%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:24.773 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:24 smithi154 ceph-mon[107038]: pgmap v345: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 43 MiB/s wr, 694 op/s; 10071/271719 objects degraded (3.706%); 8914/271719 objects misplaced (3.281%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:44:24.773 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:24 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/271719 objects degraded (3.706%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:24.796 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:24.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:24.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:24.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:24.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:24 smithi172 ceph-mon[100696]: pgmap v345: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 43 MiB/s wr, 694 op/s; 10071/271719 objects degraded (3.706%); 8914/271719 objects misplaced (3.281%); 3.8 MiB/s, 60 objects/s recovering 2024-09-13T14:44:24.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:24 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/271719 objects degraded (3.706%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:25.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:25.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:25.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:25.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:25.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:25.351 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:25.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:25.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:25.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 68 16 94933 94917 87.2172 43.3125 0.0145193 0.01146 2024-09-13T14:44:25.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:25.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:25.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:25.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:25.807 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-13T14:44:22.806079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5938747365,"num_objects":90573,"num_object_clones":0,"num_object_copies":271719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10071,"num_objects_misplaced":8914,"num_objects_unfound":0,"num_objects_dirty":90573,"num_whiteouts":0,"num_read":6333,"num_read_kb":13788,"num_write":91668,"num_write_kb":5809921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":976,"num_bytes_recovered":61612114,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91727,"ondisk_log_size":91727,"up":267,"acting":263,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":16572708,"kb_used_data":16114548,"kb_used_omap":30,"kb_used_meta":458081,"kb_avail":545898204,"statfs":{"total":575970213888,"available":558999760896,"internally_reserved":0,"allocated":16501297152,"data_stored":16497327678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":31635,"internal_metadata":469075053},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":546635776,"num_objects":8341,"num_object_clones":0,"num_object_copies":25023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-730,"num_objects_unfound":0,"num_objects_dirty":8341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8341,"num_write_kb":533824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":47841280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003810"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":109,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.174077+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:44:09.174077+0000","last_peered":"2024-09-13T14:44:09.174077+0000","last_clean":"2024-09-13T14:44:09.174077+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:44:09.174077+0000","last_undegraded":"2024-09-13T14:44:09.174077+0000","last_fullsized":"2024-09-13T14:44:09.174077+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135246+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:44:07.135246+0000","last_peered":"2024-09-13T14:44:07.135246+0000","last_clean":"2024-09-13T14:44:07.135246+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:44:07.135246+0000","last_undegraded":"2024-09-13T14:44:07.135246+0000","last_fullsized":"2024-09-13T14:44:07.135246+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":95,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135825+0000","last_change":"2024-09-13T14:44:04.109895+0000","last_active":"2024-09-13T14:44:07.135825+0000","last_peered":"2024-09-13T14:44:07.135825+0000","last_clean":"2024-09-13T14:44:07.135825+0000","last_became_active":"2024-09-13T14:44:04.109473+0000","last_became_peered":"2024-09-13T14:44:04.109473+0000","last_unstale":"2024-09-13T14:44:07.135825+0000","last_undegraded":"2024-09-13T14:44:07.135825+0000","last_fullsized":"2024-09-13T14:44:07.135825+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:43.179888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":98,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.189526+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:44:09.189526+0000","last_peered":"2024-09-13T14:44:09.189526+0000","last_clean":"2024-09-13T14:44:09.189526+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:44:09.189526+0000","last_undegraded":"2024-09-13T14:44:09.189526+0000","last_fullsized":"2024-09-13T14:44:09.189526+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":98,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.186507+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:44:09.186507+0000","last_peered":"2024-09-13T14:44:09.186507+0000","last_clean":"2024-09-13T14:44:09.186507+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:44:09.186507+0000","last_undegraded":"2024-09-13T14:44:09.186507+0000","last_fullsized":"2024-09-13T14:44:09.186507+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":82,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135520+0000","last_change":"2024-09-13T14:44:04.106908+0000","last_active":"2024-09-13T14:44:07.135520+0000","last_peered":"2024-09-13T14:44:07.135520+0000","last_clean":"2024-09-13T14:44:07.135520+0000","last_became_active":"2024-09-13T14:44:04.106563+0000","last_became_peered":"2024-09-13T14:44:04.106563+0000","last_unstale":"2024-09-13T14:44:07.135520+0000","last_undegraded":"2024-09-13T14:44:07.135520+0000","last_fullsized":"2024-09-13T14:44:07.135520+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:48.126470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":109,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.175508+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:44:09.175508+0000","last_peered":"2024-09-13T14:44:09.175508+0000","last_clean":"2024-09-13T14:44:09.175508+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:44:09.175508+0000","last_undegraded":"2024-09-13T14:44:09.175508+0000","last_fullsized":"2024-09-13T14:44:09.175508+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":44,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.185705+0000","last_change":"2024-09-13T14:44:04.131409+0000","last_active":"2024-09-13T14:44:09.185705+0000","last_peered":"2024-09-13T14:44:09.185705+0000","last_clean":"2024-09-13T14:44:09.185705+0000","last_became_active":"2024-09-13T14:44:04.122152+0000","last_became_peered":"2024-09-13T14:44:04.122152+0000","last_unstale":"2024-09-13T14:44:09.185705+0000","last_undegraded":"2024-09-13T14:44:09.185705+0000","last_fullsized":"2024-09-13T14:44:09.185705+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:40.298704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"126'5537","reported_seq":6577,"reported_epoch":126,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:44:18.727641+0000","last_change":"2024-09-13T14:44:05.337600+0000","last_active":"2024-09-13T14:44:18.727641+0000","last_peered":"2024-09-13T14:44:18.727641+0000","last_clean":"2024-09-13T14:44:03.047594+0000","last_became_active":"2024-09-13T14:44:05.301093+0000","last_became_peered":"2024-09-13T14:44:05.301093+0000","last_unstale":"2024-09-13T14:44:18.727641+0000","last_undegraded":"2024-09-13T14:44:18.727641+0000","last_fullsized":"2024-09-13T14:44:05.100049+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5537,"log_dups_size":0,"ondisk_log_size":5537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:40.048142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362872832,"num_objects":5537,"num_object_clones":0,"num_object_copies":16611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4399,"num_objects_unfound":0,"num_objects_dirty":5537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5537,"num_write_kb":354368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":503,"num_bytes_recovered":32964608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":4399},{"shards":"4,7","objects":1138}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:08.737572+0000","last_change":"2024-09-13T14:40:04.164112+0000","last_active":"2024-09-13T14:44:08.737572+0000","last_peered":"2024-09-13T14:44:08.737572+0000","last_clean":"2024-09-13T14:44:08.737572+0000","last_became_active":"2024-09-13T14:40:04.163753+0000","last_became_peered":"2024-09-13T14:40:04.163753+0000","last_unstale":"2024-09-13T14:44:08.737572+0000","last_undegraded":"2024-09-13T14:44:08.737572+0000","last_fullsized":"2024-09-13T14:44:08.737572+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:10.685538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":35,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135060+0000","last_change":"2024-09-13T14:44:04.108206+0000","last_active":"2024-09-13T14:44:07.135060+0000","last_peered":"2024-09-13T14:44:07.135060+0000","last_clean":"2024-09-13T14:44:07.135060+0000","last_became_active":"2024-09-13T14:44:04.107851+0000","last_became_peered":"2024-09-13T14:44:04.107851+0000","last_unstale":"2024-09-13T14:44:07.135060+0000","last_undegraded":"2024-09-13T14:44:07.135060+0000","last_fullsized":"2024-09-13T14:44:07.135060+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:52.745746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"126'5669","reported_seq":5713,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:17.990772+0000","last_change":"2024-09-13T14:43:09.080838+0000","last_active":"2024-09-13T14:44:17.990772+0000","last_peered":"2024-09-13T14:44:17.990772+0000","last_clean":"2024-09-13T14:44:17.990772+0000","last_became_active":"2024-09-13T14:43:09.080537+0000","last_became_peered":"2024-09-13T14:43:09.080537+0000","last_unstale":"2024-09-13T14:44:17.990772+0000","last_undegraded":"2024-09-13T14:44:17.990772+0000","last_fullsized":"2024-09-13T14:44:17.990772+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5669,"log_dups_size":0,"ondisk_log_size":5669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:48.270924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371523584,"num_objects":5669,"num_object_clones":0,"num_object_copies":17007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":362816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135735+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:44:07.135735+0000","last_peered":"2024-09-13T14:44:07.135735+0000","last_clean":"2024-09-13T14:44:07.135735+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:44:07.135735+0000","last_undegraded":"2024-09-13T14:44:07.135735+0000","last_fullsized":"2024-09-13T14:44:07.135735+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":111,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143699+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:44:07.143699+0000","last_peered":"2024-09-13T14:44:07.143699+0000","last_clean":"2024-09-13T14:44:07.143699+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:44:07.143699+0000","last_undegraded":"2024-09-13T14:44:07.143699+0000","last_fullsized":"2024-09-13T14:44:07.143699+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"126'5683","reported_seq":5726,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:17.986371+0000","last_change":"2024-09-13T14:43:09.680819+0000","last_active":"2024-09-13T14:44:17.986371+0000","last_peered":"2024-09-13T14:44:17.986371+0000","last_clean":"2024-09-13T14:44:17.986371+0000","last_became_active":"2024-09-13T14:43:09.680530+0000","last_became_peered":"2024-09-13T14:43:09.680530+0000","last_unstale":"2024-09-13T14:44:17.986371+0000","last_undegraded":"2024-09-13T14:44:17.986371+0000","last_fullsized":"2024-09-13T14:44:17.986371+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5683,"log_dups_size":0,"ondisk_log_size":5683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:41:19.694859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372441088,"num_objects":5683,"num_object_clones":0,"num_object_copies":17049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":363712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":101,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135951+0000","last_change":"2024-09-13T14:44:04.118317+0000","last_active":"2024-09-13T14:44:07.135951+0000","last_peered":"2024-09-13T14:44:07.135951+0000","last_clean":"2024-09-13T14:44:07.135951+0000","last_became_active":"2024-09-13T14:44:04.109164+0000","last_became_peered":"2024-09-13T14:44:04.109164+0000","last_unstale":"2024-09-13T14:44:07.135951+0000","last_undegraded":"2024-09-13T14:44:07.135951+0000","last_fullsized":"2024-09-13T14:44:07.135951+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:18.798567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":71,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.182820+0000","last_change":"2024-09-13T14:44:04.119895+0000","last_active":"2024-09-13T14:44:09.182820+0000","last_peered":"2024-09-13T14:44:09.182820+0000","last_clean":"2024-09-13T14:44:09.182820+0000","last_became_active":"2024-09-13T14:44:04.118753+0000","last_became_peered":"2024-09-13T14:44:04.118753+0000","last_unstale":"2024-09-13T14:44:09.182820+0000","last_undegraded":"2024-09-13T14:44:09.182820+0000","last_fullsized":"2024-09-13T14:44:09.182820+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:21.918147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"126'5586","reported_seq":5630,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:18.019234+0000","last_change":"2024-09-13T14:43:09.077347+0000","last_active":"2024-09-13T14:44:18.019234+0000","last_peered":"2024-09-13T14:44:18.019234+0000","last_clean":"2024-09-13T14:44:18.019234+0000","last_became_active":"2024-09-13T14:43:09.077065+0000","last_became_peered":"2024-09-13T14:43:09.077065+0000","last_unstale":"2024-09-13T14:44:18.019234+0000","last_undegraded":"2024-09-13T14:44:18.019234+0000","last_fullsized":"2024-09-13T14:44:18.019234+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:59.723003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366084096,"num_objects":5586,"num_object_clones":0,"num_object_copies":16758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5586,"num_write_kb":357504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":109,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.178463+0000","last_change":"2024-09-13T14:44:04.111963+0000","last_active":"2024-09-13T14:44:09.178463+0000","last_peered":"2024-09-13T14:44:09.178463+0000","last_clean":"2024-09-13T14:44:09.178463+0000","last_became_active":"2024-09-13T14:44:04.111627+0000","last_became_peered":"2024-09-13T14:44:04.111627+0000","last_unstale":"2024-09-13T14:44:09.178463+0000","last_undegraded":"2024-09-13T14:44:09.178463+0000","last_fullsized":"2024-09-13T14:44:09.178463+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:04.837093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":98,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.194463+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:44:09.194463+0000","last_peered":"2024-09-13T14:44:09.194463+0000","last_clean":"2024-09-13T14:44:09.194463+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:44:09.194463+0000","last_undegraded":"2024-09-13T14:44:09.194463+0000","last_fullsized":"2024-09-13T14:44:09.194463+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"126'5503","reported_seq":5545,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:18.033671+0000","last_change":"2024-09-13T14:43:09.679251+0000","last_active":"2024-09-13T14:44:18.033671+0000","last_peered":"2024-09-13T14:44:18.033671+0000","last_clean":"2024-09-13T14:44:18.033671+0000","last_became_active":"2024-09-13T14:43:09.678926+0000","last_became_peered":"2024-09-13T14:43:09.678926+0000","last_unstale":"2024-09-13T14:44:18.033671+0000","last_undegraded":"2024-09-13T14:44:18.033671+0000","last_fullsized":"2024-09-13T14:44:18.033671+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:19.700409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360644608,"num_objects":5503,"num_object_clones":0,"num_object_copies":16509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5503,"num_write_kb":352192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":107,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.174360+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:44:09.174360+0000","last_peered":"2024-09-13T14:44:09.174360+0000","last_clean":"2024-09-13T14:44:09.174360+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:44:09.174360+0000","last_undegraded":"2024-09-13T14:44:09.174360+0000","last_fullsized":"2024-09-13T14:44:09.174360+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":96,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143495+0000","last_change":"2024-09-13T14:44:04.117961+0000","last_active":"2024-09-13T14:44:07.143495+0000","last_peered":"2024-09-13T14:44:07.143495+0000","last_clean":"2024-09-13T14:44:07.143495+0000","last_became_active":"2024-09-13T14:44:04.117592+0000","last_became_peered":"2024-09-13T14:44:04.117592+0000","last_unstale":"2024-09-13T14:44:07.143495+0000","last_undegraded":"2024-09-13T14:44:07.143495+0000","last_fullsized":"2024-09-13T14:44:07.143495+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:42.652379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"126'5687","reported_seq":5731,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:19.834682+0000","last_change":"2024-09-13T14:43:09.079510+0000","last_active":"2024-09-13T14:44:19.834682+0000","last_peered":"2024-09-13T14:44:19.834682+0000","last_clean":"2024-09-13T14:44:19.834682+0000","last_became_active":"2024-09-13T14:43:09.078574+0000","last_became_peered":"2024-09-13T14:43:09.078574+0000","last_unstale":"2024-09-13T14:44:19.834682+0000","last_undegraded":"2024-09-13T14:44:19.834682+0000","last_fullsized":"2024-09-13T14:44:19.834682+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:32.105865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372703232,"num_objects":5687,"num_object_clones":0,"num_object_copies":17061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5687,"num_write_kb":363968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143320+0000","last_change":"2024-09-13T14:44:04.116730+0000","last_active":"2024-09-13T14:44:07.143320+0000","last_peered":"2024-09-13T14:44:07.143320+0000","last_clean":"2024-09-13T14:44:07.143320+0000","last_became_active":"2024-09-13T14:44:04.116447+0000","last_became_peered":"2024-09-13T14:44:04.116447+0000","last_unstale":"2024-09-13T14:44:07.143320+0000","last_undegraded":"2024-09-13T14:44:07.143320+0000","last_fullsized":"2024-09-13T14:44:07.143320+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:43.501240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":102,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143728+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:44:07.143728+0000","last_peered":"2024-09-13T14:44:07.143728+0000","last_clean":"2024-09-13T14:44:07.143728+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:44:07.143728+0000","last_undegraded":"2024-09-13T14:44:07.143728+0000","last_fullsized":"2024-09-13T14:44:07.143728+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"126'5689","reported_seq":5719,"reported_epoch":126,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:18.715266+0000","last_change":"2024-09-13T14:44:05.317448+0000","last_active":"2024-09-13T14:44:18.715266+0000","last_peered":"2024-09-13T14:44:18.715266+0000","last_clean":"2024-09-13T14:44:03.057991+0000","last_became_active":"2024-09-13T14:44:05.309445+0000","last_became_peered":"2024-09-13T14:44:05.309445+0000","last_unstale":"2024-09-13T14:44:18.715266+0000","last_undegraded":"2024-09-13T14:44:05.202807+0000","last_fullsized":"2024-09-13T14:44:05.099517+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:36.732464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372834304,"num_objects":5689,"num_object_clones":0,"num_object_copies":17067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5023,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5689,"num_write_kb":364096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2],"avail_no_missing":["4","2","6"],"object_location_counts":[{"shards":"2,4","objects":666},{"shards":"2,4,6","objects":5023}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":88,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143721+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:44:07.143721+0000","last_peered":"2024-09-13T14:44:07.143721+0000","last_clean":"2024-09-13T14:44:07.143721+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:44:07.143721+0000","last_undegraded":"2024-09-13T14:44:07.143721+0000","last_fullsized":"2024-09-13T14:44:07.143721+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":96,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.144008+0000","last_change":"2024-09-13T14:44:04.113973+0000","last_active":"2024-09-13T14:44:07.144008+0000","last_peered":"2024-09-13T14:44:07.144008+0000","last_clean":"2024-09-13T14:44:07.144008+0000","last_became_active":"2024-09-13T14:44:04.112945+0000","last_became_peered":"2024-09-13T14:44:04.112945+0000","last_unstale":"2024-09-13T14:44:07.144008+0000","last_undegraded":"2024-09-13T14:44:07.144008+0000","last_fullsized":"2024-09-13T14:44:07.144008+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:35:36.952813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"126'5605","reported_seq":5631,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:18.710885+0000","last_change":"2024-09-13T14:43:09.683235+0000","last_active":"2024-09-13T14:44:18.710885+0000","last_peered":"2024-09-13T14:44:18.710885+0000","last_clean":"2024-09-13T14:44:18.710885+0000","last_became_active":"2024-09-13T14:43:09.682916+0000","last_became_peered":"2024-09-13T14:43:09.682916+0000","last_unstale":"2024-09-13T14:44:18.710885+0000","last_undegraded":"2024-09-13T14:44:18.710885+0000","last_fullsized":"2024-09-13T14:44:18.710885+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:19.580771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367329280,"num_objects":5605,"num_object_clones":0,"num_object_copies":16815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":358720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":110,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.178313+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:44:09.178313+0000","last_peered":"2024-09-13T14:44:09.178313+0000","last_clean":"2024-09-13T14:44:09.178313+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:44:09.178313+0000","last_undegraded":"2024-09-13T14:44:09.178313+0000","last_fullsized":"2024-09-13T14:44:09.178313+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":89,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.191827+0000","last_change":"2024-09-13T14:41:41.488308+0000","last_active":"2024-09-13T14:44:09.191827+0000","last_peered":"2024-09-13T14:44:09.191827+0000","last_clean":"2024-09-13T14:44:09.191827+0000","last_became_active":"2024-09-13T14:41:41.487374+0000","last_became_peered":"2024-09-13T14:41:41.487374+0000","last_unstale":"2024-09-13T14:44:09.191827+0000","last_undegraded":"2024-09-13T14:44:09.191827+0000","last_fullsized":"2024-09-13T14:44:09.191827+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:51.242762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"126'5709","reported_seq":5735,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:18.663499+0000","last_change":"2024-09-13T14:43:09.679474+0000","last_active":"2024-09-13T14:44:18.663499+0000","last_peered":"2024-09-13T14:44:18.663499+0000","last_clean":"2024-09-13T14:44:18.663499+0000","last_became_active":"2024-09-13T14:43:09.679179+0000","last_became_peered":"2024-09-13T14:43:09.679179+0000","last_unstale":"2024-09-13T14:44:18.663499+0000","last_undegraded":"2024-09-13T14:44:18.663499+0000","last_fullsized":"2024-09-13T14:44:18.663499+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5709,"log_dups_size":0,"ondisk_log_size":5709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:01.483055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374145024,"num_objects":5709,"num_object_clones":0,"num_object_copies":17127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5709,"num_write_kb":365376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":442,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.200064+0000","last_change":"2024-09-13T14:44:07.199846+0000","last_active":"2024-09-13T14:44:07.200064+0000","last_peered":"2024-09-13T14:44:07.200064+0000","last_clean":"2024-09-13T14:44:07.200064+0000","last_became_active":"2024-09-13T14:44:07.199512+0000","last_became_peered":"2024-09-13T14:44:07.199512+0000","last_unstale":"2024-09-13T14:44:07.200064+0000","last_undegraded":"2024-09-13T14:44:07.200064+0000","last_fullsized":"2024-09-13T14:44:07.200064+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:18.724539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":612,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.185311+0000","last_change":"2024-09-13T14:43:12.684107+0000","last_active":"2024-09-13T14:44:09.185311+0000","last_peered":"2024-09-13T14:44:09.185311+0000","last_clean":"2024-09-13T14:44:09.185311+0000","last_became_active":"2024-09-13T14:43:12.683522+0000","last_became_peered":"2024-09-13T14:43:12.683522+0000","last_unstale":"2024-09-13T14:44:09.185311+0000","last_undegraded":"2024-09-13T14:44:09.185311+0000","last_fullsized":"2024-09-13T14:44:09.185311+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:52.396314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":72,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.174637+0000","last_change":"2024-09-13T14:44:04.110946+0000","last_active":"2024-09-13T14:44:09.174637+0000","last_peered":"2024-09-13T14:44:09.174637+0000","last_clean":"2024-09-13T14:44:09.174637+0000","last_became_active":"2024-09-13T14:44:04.110626+0000","last_became_peered":"2024-09-13T14:44:04.110626+0000","last_unstale":"2024-09-13T14:44:09.174637+0000","last_undegraded":"2024-09-13T14:44:09.174637+0000","last_fullsized":"2024-09-13T14:44:09.174637+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:34:57.354620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143591+0000","last_change":"2024-09-13T14:44:04.116341+0000","last_active":"2024-09-13T14:44:07.143591+0000","last_peered":"2024-09-13T14:44:07.143591+0000","last_clean":"2024-09-13T14:44:07.143591+0000","last_became_active":"2024-09-13T14:44:04.114940+0000","last_became_peered":"2024-09-13T14:44:04.114940+0000","last_unstale":"2024-09-13T14:44:07.143591+0000","last_undegraded":"2024-09-13T14:44:07.143591+0000","last_fullsized":"2024-09-13T14:44:07.143591+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:38.852464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"126'5756","reported_seq":5798,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:19.934901+0000","last_change":"2024-09-13T14:43:09.681916+0000","last_active":"2024-09-13T14:44:19.934901+0000","last_peered":"2024-09-13T14:44:19.934901+0000","last_clean":"2024-09-13T14:44:19.934901+0000","last_became_active":"2024-09-13T14:43:09.681588+0000","last_became_peered":"2024-09-13T14:43:09.681588+0000","last_unstale":"2024-09-13T14:44:19.934901+0000","last_undegraded":"2024-09-13T14:44:19.934901+0000","last_fullsized":"2024-09-13T14:44:19.934901+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:04.178796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377225216,"num_objects":5756,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5756,"num_write_kb":368384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":278,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135079+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:44:07.135079+0000","last_peered":"2024-09-13T14:44:07.135079+0000","last_clean":"2024-09-13T14:44:07.135079+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:44:07.135079+0000","last_undegraded":"2024-09-13T14:44:07.135079+0000","last_fullsized":"2024-09-13T14:44:07.135079+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":99,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.185873+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:44:09.185873+0000","last_peered":"2024-09-13T14:44:09.185873+0000","last_clean":"2024-09-13T14:44:09.185873+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:44:09.185873+0000","last_undegraded":"2024-09-13T14:44:09.185873+0000","last_fullsized":"2024-09-13T14:44:09.185873+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":110,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.186711+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:44:09.186711+0000","last_peered":"2024-09-13T14:44:09.186711+0000","last_clean":"2024-09-13T14:44:09.186711+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:44:09.186711+0000","last_undegraded":"2024-09-13T14:44:09.186711+0000","last_fullsized":"2024-09-13T14:44:09.186711+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"126'5565","reported_seq":5608,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:18.041286+0000","last_change":"2024-09-13T14:43:09.679876+0000","last_active":"2024-09-13T14:44:18.041286+0000","last_peered":"2024-09-13T14:44:18.041286+0000","last_clean":"2024-09-13T14:44:18.041286+0000","last_became_active":"2024-09-13T14:43:09.679544+0000","last_became_peered":"2024-09-13T14:43:09.679544+0000","last_unstale":"2024-09-13T14:44:18.041286+0000","last_undegraded":"2024-09-13T14:44:18.041286+0000","last_fullsized":"2024-09-13T14:44:18.041286+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:39:53.084013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364707840,"num_objects":5565,"num_object_clones":0,"num_object_copies":16695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5565,"num_write_kb":356160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":662,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.184174+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:44:09.184174+0000","last_peered":"2024-09-13T14:44:09.184174+0000","last_clean":"2024-09-13T14:44:09.184174+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:44:09.184174+0000","last_undegraded":"2024-09-13T14:44:09.184174+0000","last_fullsized":"2024-09-13T14:44:09.184174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":96,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.185043+0000","last_change":"2024-09-13T14:44:04.120828+0000","last_active":"2024-09-13T14:44:09.185043+0000","last_peered":"2024-09-13T14:44:09.185043+0000","last_clean":"2024-09-13T14:44:09.185043+0000","last_became_active":"2024-09-13T14:44:04.120323+0000","last_became_peered":"2024-09-13T14:44:04.120323+0000","last_unstale":"2024-09-13T14:44:09.185043+0000","last_undegraded":"2024-09-13T14:44:09.185043+0000","last_fullsized":"2024-09-13T14:44:09.185043+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:39.888620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":118,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.184938+0000","last_change":"2024-09-13T14:43:09.679993+0000","last_active":"2024-09-13T14:44:09.184938+0000","last_peered":"2024-09-13T14:44:09.184938+0000","last_clean":"2024-09-13T14:44:09.184938+0000","last_became_active":"2024-09-13T14:43:09.679699+0000","last_became_peered":"2024-09-13T14:43:09.679699+0000","last_unstale":"2024-09-13T14:44:09.184938+0000","last_undegraded":"2024-09-13T14:44:09.184938+0000","last_fullsized":"2024-09-13T14:44:09.184938+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:20.969934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"126'5733","reported_seq":5777,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:21.923045+0000","last_change":"2024-09-13T14:43:09.078374+0000","last_active":"2024-09-13T14:44:21.923045+0000","last_peered":"2024-09-13T14:44:21.923045+0000","last_clean":"2024-09-13T14:44:21.923045+0000","last_became_active":"2024-09-13T14:43:09.078086+0000","last_became_peered":"2024-09-13T14:43:09.078086+0000","last_unstale":"2024-09-13T14:44:21.923045+0000","last_undegraded":"2024-09-13T14:44:21.923045+0000","last_fullsized":"2024-09-13T14:44:21.923045+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5733,"log_dups_size":0,"ondisk_log_size":5733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:59.636710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375717888,"num_objects":5733,"num_object_clones":0,"num_object_copies":17199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5733,"num_write_kb":366912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":706,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.183941+0000","last_change":"2024-09-13T14:44:04.136975+0000","last_active":"2024-09-13T14:44:09.183941+0000","last_peered":"2024-09-13T14:44:09.183941+0000","last_clean":"2024-09-13T14:44:09.183941+0000","last_became_active":"2024-09-13T14:44:04.119786+0000","last_became_peered":"2024-09-13T14:44:04.119786+0000","last_unstale":"2024-09-13T14:44:09.183941+0000","last_undegraded":"2024-09-13T14:44:09.183941+0000","last_fullsized":"2024-09-13T14:44:09.183941+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:39:34.248994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":97,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143691+0000","last_change":"2024-09-13T14:44:04.117509+0000","last_active":"2024-09-13T14:44:07.143691+0000","last_peered":"2024-09-13T14:44:07.143691+0000","last_clean":"2024-09-13T14:44:07.143691+0000","last_became_active":"2024-09-13T14:44:04.117206+0000","last_became_peered":"2024-09-13T14:44:04.117206+0000","last_unstale":"2024-09-13T14:44:07.143691+0000","last_undegraded":"2024-09-13T14:44:07.143691+0000","last_fullsized":"2024-09-13T14:44:07.143691+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:26.860537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":100,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.183572+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:44:09.183572+0000","last_peered":"2024-09-13T14:44:09.183572+0000","last_clean":"2024-09-13T14:44:09.183572+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:44:09.183572+0000","last_undegraded":"2024-09-13T14:44:09.183572+0000","last_fullsized":"2024-09-13T14:44:09.183572+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"126'5800","reported_seq":5842,"reported_epoch":126,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:44:21.894752+0000","last_change":"2024-09-13T14:44:05.304002+0000","last_active":"2024-09-13T14:44:21.894752+0000","last_peered":"2024-09-13T14:44:21.894752+0000","last_clean":"2024-09-13T14:44:03.071530+0000","last_became_active":"2024-09-13T14:44:05.294474+0000","last_became_peered":"2024-09-13T14:44:05.294474+0000","last_unstale":"2024-09-13T14:44:21.894752+0000","last_undegraded":"2024-09-13T14:44:05.137985+0000","last_fullsized":"2024-09-13T14:44:05.099986+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380108800,"num_objects":5800,"num_object_clones":0,"num_object_copies":17400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":371200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3","objects":752},{"shards":"0,3,6","objects":5048}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.174926+0000","last_change":"2024-09-13T14:44:04.109393+0000","last_active":"2024-09-13T14:44:09.174926+0000","last_peered":"2024-09-13T14:44:09.174926+0000","last_clean":"2024-09-13T14:44:09.174926+0000","last_became_active":"2024-09-13T14:44:04.109045+0000","last_became_peered":"2024-09-13T14:44:04.109045+0000","last_unstale":"2024-09-13T14:44:09.174926+0000","last_undegraded":"2024-09-13T14:44:09.174926+0000","last_fullsized":"2024-09-13T14:44:09.174926+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:34.970392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":628,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.183782+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:44:09.183782+0000","last_peered":"2024-09-13T14:44:09.183782+0000","last_clean":"2024-09-13T14:44:09.183782+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:44:09.183782+0000","last_undegraded":"2024-09-13T14:44:09.183782+0000","last_fullsized":"2024-09-13T14:44:09.183782+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":85,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.184172+0000","last_change":"2024-09-13T14:43:09.681228+0000","last_active":"2024-09-13T14:44:09.184172+0000","last_peered":"2024-09-13T14:44:09.184172+0000","last_clean":"2024-09-13T14:44:09.184172+0000","last_became_active":"2024-09-13T14:43:09.680908+0000","last_became_peered":"2024-09-13T14:43:09.680908+0000","last_unstale":"2024-09-13T14:44:09.184172+0000","last_undegraded":"2024-09-13T14:44:09.184172+0000","last_fullsized":"2024-09-13T14:44:09.184172+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:33.151302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"126'5803","reported_seq":6639,"reported_epoch":126,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:44:21.903633+0000","last_change":"2024-09-13T14:44:05.281772+0000","last_active":"2024-09-13T14:44:21.903633+0000","last_peered":"2024-09-13T14:44:21.903633+0000","last_clean":"2024-09-13T14:44:03.068628+0000","last_became_active":"2024-09-13T14:44:05.245569+0000","last_became_peered":"2024-09-13T14:44:05.245569+0000","last_unstale":"2024-09-13T14:44:21.903633+0000","last_undegraded":"2024-09-13T14:44:21.903633+0000","last_fullsized":"2024-09-13T14:44:05.100849+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380305408,"num_objects":5803,"num_object_clones":0,"num_object_copies":17409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4515,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":371392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":26214400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2","objects":1288},{"shards":"0,2,6","objects":4515}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":96,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.183683+0000","last_change":"2024-09-13T14:43:09.680063+0000","last_active":"2024-09-13T14:44:09.183683+0000","last_peered":"2024-09-13T14:44:09.183683+0000","last_clean":"2024-09-13T14:44:09.183683+0000","last_became_active":"2024-09-13T14:43:09.679740+0000","last_became_peered":"2024-09-13T14:43:09.679740+0000","last_unstale":"2024-09-13T14:44:09.183683+0000","last_undegraded":"2024-09-13T14:44:09.183683+0000","last_fullsized":"2024-09-13T14:44:09.183683+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:55.701172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":439,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135966+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:44:07.135966+0000","last_peered":"2024-09-13T14:44:07.135966+0000","last_clean":"2024-09-13T14:44:07.135966+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:44:07.135966+0000","last_undegraded":"2024-09-13T14:44:07.135966+0000","last_fullsized":"2024-09-13T14:44:07.135966+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":95,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.194421+0000","last_change":"2024-09-13T14:43:09.682650+0000","last_active":"2024-09-13T14:44:09.194421+0000","last_peered":"2024-09-13T14:44:09.194421+0000","last_clean":"2024-09-13T14:44:09.194421+0000","last_became_active":"2024-09-13T14:43:09.682335+0000","last_became_peered":"2024-09-13T14:43:09.682335+0000","last_unstale":"2024-09-13T14:44:09.194421+0000","last_undegraded":"2024-09-13T14:44:09.194421+0000","last_fullsized":"2024-09-13T14:44:09.194421+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:26.113631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"126'5581","reported_seq":5606,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:18.707655+0000","last_change":"2024-09-13T14:43:09.080169+0000","last_active":"2024-09-13T14:44:18.707655+0000","last_peered":"2024-09-13T14:44:18.707655+0000","last_clean":"2024-09-13T14:44:18.707655+0000","last_became_active":"2024-09-13T14:43:09.079885+0000","last_became_peered":"2024-09-13T14:43:09.079885+0000","last_unstale":"2024-09-13T14:44:18.707655+0000","last_undegraded":"2024-09-13T14:44:18.707655+0000","last_fullsized":"2024-09-13T14:44:18.707655+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:05.421672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365756416,"num_objects":5581,"num_object_clones":0,"num_object_copies":16743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":357184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":107,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.175698+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:44:09.175698+0000","last_peered":"2024-09-13T14:44:09.175698+0000","last_clean":"2024-09-13T14:44:09.175698+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:44:09.175698+0000","last_undegraded":"2024-09-13T14:44:09.175698+0000","last_fullsized":"2024-09-13T14:44:09.175698+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":536,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.183773+0000","last_change":"2024-09-13T14:44:07.202341+0000","last_active":"2024-09-13T14:44:09.183773+0000","last_peered":"2024-09-13T14:44:09.183773+0000","last_clean":"2024-09-13T14:44:09.183773+0000","last_became_active":"2024-09-13T14:44:07.201889+0000","last_became_peered":"2024-09-13T14:44:07.201889+0000","last_unstale":"2024-09-13T14:44:09.183773+0000","last_undegraded":"2024-09-13T14:44:09.183773+0000","last_fullsized":"2024-09-13T14:44:09.183773+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:49.837319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.178204+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:44:09.178204+0000","last_peered":"2024-09-13T14:44:09.178204+0000","last_clean":"2024-09-13T14:44:09.178204+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:44:09.178204+0000","last_undegraded":"2024-09-13T14:44:09.178204+0000","last_fullsized":"2024-09-13T14:44:09.178204+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"126'5639","reported_seq":5683,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:19.489021+0000","last_change":"2024-09-13T14:43:09.077743+0000","last_active":"2024-09-13T14:44:19.489021+0000","last_peered":"2024-09-13T14:44:19.489021+0000","last_clean":"2024-09-13T14:44:19.489021+0000","last_became_active":"2024-09-13T14:43:09.077468+0000","last_became_peered":"2024-09-13T14:43:09.077468+0000","last_unstale":"2024-09-13T14:44:19.489021+0000","last_undegraded":"2024-09-13T14:44:19.489021+0000","last_fullsized":"2024-09-13T14:44:19.489021+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:43:08.051225+0000","last_clean_scrub_stamp":"2024-09-13T14:43:08.051225+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:13:07.465815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369557504,"num_objects":5639,"num_object_clones":0,"num_object_copies":16917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":360896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.186452+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:44:09.186452+0000","last_peered":"2024-09-13T14:44:09.186452+0000","last_clean":"2024-09-13T14:44:09.186452+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:44:09.186452+0000","last_undegraded":"2024-09-13T14:44:09.186452+0000","last_fullsized":"2024-09-13T14:44:09.186452+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":687,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.202479+0000","last_change":"2024-09-13T14:44:07.202136+0000","last_active":"2024-09-13T14:44:07.202479+0000","last_peered":"2024-09-13T14:44:07.202479+0000","last_clean":"2024-09-13T14:44:07.202479+0000","last_became_active":"2024-09-13T14:44:07.201718+0000","last_became_peered":"2024-09-13T14:44:07.201718+0000","last_unstale":"2024-09-13T14:44:07.202479+0000","last_undegraded":"2024-09-13T14:44:07.202479+0000","last_fullsized":"2024-09-13T14:44:07.202479+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:02.665205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":90,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143508+0000","last_change":"2024-09-13T14:44:04.115846+0000","last_active":"2024-09-13T14:44:07.143508+0000","last_peered":"2024-09-13T14:44:07.143508+0000","last_clean":"2024-09-13T14:44:07.143508+0000","last_became_active":"2024-09-13T14:44:04.115530+0000","last_became_peered":"2024-09-13T14:44:04.115530+0000","last_unstale":"2024-09-13T14:44:07.143508+0000","last_undegraded":"2024-09-13T14:44:07.143508+0000","last_fullsized":"2024-09-13T14:44:07.143508+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:28.695197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":95,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143570+0000","last_change":"2024-09-13T14:44:04.113649+0000","last_active":"2024-09-13T14:44:07.143570+0000","last_peered":"2024-09-13T14:44:07.143570+0000","last_clean":"2024-09-13T14:44:07.143570+0000","last_became_active":"2024-09-13T14:44:04.113335+0000","last_became_peered":"2024-09-13T14:44:04.113335+0000","last_unstale":"2024-09-13T14:44:07.143570+0000","last_undegraded":"2024-09-13T14:44:07.143570+0000","last_fullsized":"2024-09-13T14:44:07.143570+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:25.644030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":114,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143688+0000","last_change":"2024-09-13T14:44:04.118495+0000","last_active":"2024-09-13T14:44:07.143688+0000","last_peered":"2024-09-13T14:44:07.143688+0000","last_clean":"2024-09-13T14:44:07.143688+0000","last_became_active":"2024-09-13T14:44:04.107228+0000","last_became_peered":"2024-09-13T14:44:04.107228+0000","last_unstale":"2024-09-13T14:44:07.143688+0000","last_undegraded":"2024-09-13T14:44:07.143688+0000","last_fullsized":"2024-09-13T14:44:07.143688+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:05.861515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4756,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":96,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.184040+0000","last_change":"2024-09-13T14:44:04.119544+0000","last_active":"2024-09-13T14:44:09.184040+0000","last_peered":"2024-09-13T14:44:09.184040+0000","last_clean":"2024-09-13T14:44:09.184040+0000","last_became_active":"2024-09-13T14:44:04.118754+0000","last_became_peered":"2024-09-13T14:44:04.118754+0000","last_unstale":"2024-09-13T14:44:09.184040+0000","last_undegraded":"2024-09-13T14:44:09.184040+0000","last_fullsized":"2024-09-13T14:44:09.184040+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:12.053865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":91,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135854+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:44:07.135854+0000","last_peered":"2024-09-13T14:44:07.135854+0000","last_clean":"2024-09-13T14:44:07.135854+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:44:07.135854+0000","last_undegraded":"2024-09-13T14:44:07.135854+0000","last_fullsized":"2024-09-13T14:44:07.135854+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":77,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.135662+0000","last_change":"2024-09-13T14:43:09.682269+0000","last_active":"2024-09-13T14:44:07.135662+0000","last_peered":"2024-09-13T14:44:07.135662+0000","last_clean":"2024-09-13T14:44:07.135662+0000","last_became_active":"2024-09-13T14:43:09.681735+0000","last_became_peered":"2024-09-13T14:43:09.681735+0000","last_unstale":"2024-09-13T14:44:07.135662+0000","last_undegraded":"2024-09-13T14:44:07.135662+0000","last_fullsized":"2024-09-13T14:44:07.135662+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:48.513102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":65,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.174905+0000","last_change":"2024-09-13T14:44:04.111704+0000","last_active":"2024-09-13T14:44:09.174905+0000","last_peered":"2024-09-13T14:44:09.174905+0000","last_clean":"2024-09-13T14:44:09.174905+0000","last_became_active":"2024-09-13T14:44:04.111395+0000","last_became_peered":"2024-09-13T14:44:04.111395+0000","last_unstale":"2024-09-13T14:44:09.174905+0000","last_undegraded":"2024-09-13T14:44:09.174905+0000","last_fullsized":"2024-09-13T14:44:09.174905+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:11.511401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":112,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.187028+0000","last_change":"2024-09-13T14:44:04.120876+0000","last_active":"2024-09-13T14:44:09.187028+0000","last_peered":"2024-09-13T14:44:09.187028+0000","last_clean":"2024-09-13T14:44:09.187028+0000","last_became_active":"2024-09-13T14:44:04.118896+0000","last_became_peered":"2024-09-13T14:44:04.118896+0000","last_unstale":"2024-09-13T14:44:09.187028+0000","last_undegraded":"2024-09-13T14:44:09.187028+0000","last_fullsized":"2024-09-13T14:44:09.187028+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:50:33.883900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":116,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.178130+0000","last_change":"2024-09-13T14:44:04.113148+0000","last_active":"2024-09-13T14:44:09.178130+0000","last_peered":"2024-09-13T14:44:09.178130+0000","last_clean":"2024-09-13T14:44:09.178130+0000","last_became_active":"2024-09-13T14:44:04.111669+0000","last_became_peered":"2024-09-13T14:44:04.111669+0000","last_unstale":"2024-09-13T14:44:09.178130+0000","last_undegraded":"2024-09-13T14:44:09.178130+0000","last_fullsized":"2024-09-13T14:44:09.178130+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:50.606259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":94,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.190272+0000","last_change":"2024-09-13T14:44:04.122820+0000","last_active":"2024-09-13T14:44:09.190272+0000","last_peered":"2024-09-13T14:44:09.190272+0000","last_clean":"2024-09-13T14:44:09.190272+0000","last_became_active":"2024-09-13T14:44:04.122467+0000","last_became_peered":"2024-09-13T14:44:04.122467+0000","last_unstale":"2024-09-13T14:44:09.190272+0000","last_undegraded":"2024-09-13T14:44:09.190272+0000","last_fullsized":"2024-09-13T14:44:09.190272+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:50.013331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":125,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.178349+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:44:09.178349+0000","last_peered":"2024-09-13T14:44:09.178349+0000","last_clean":"2024-09-13T14:44:09.178349+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:44:09.178349+0000","last_undegraded":"2024-09-13T14:44:09.178349+0000","last_fullsized":"2024-09-13T14:44:09.178349+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":99,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.186593+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:44:09.186593+0000","last_peered":"2024-09-13T14:44:09.186593+0000","last_clean":"2024-09-13T14:44:09.186593+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:44:09.186593+0000","last_undegraded":"2024-09-13T14:44:09.186593+0000","last_fullsized":"2024-09-13T14:44:09.186593+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":89,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.183213+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:44:09.183213+0000","last_peered":"2024-09-13T14:44:09.183213+0000","last_clean":"2024-09-13T14:44:09.183213+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:44:09.183213+0000","last_undegraded":"2024-09-13T14:44:09.183213+0000","last_fullsized":"2024-09-13T14:44:09.183213+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":108,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.174685+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:44:09.174685+0000","last_peered":"2024-09-13T14:44:09.174685+0000","last_clean":"2024-09-13T14:44:09.174685+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:44:09.174685+0000","last_undegraded":"2024-09-13T14:44:09.174685+0000","last_fullsized":"2024-09-13T14:44:09.174685+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":103,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143411+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:44:07.143411+0000","last_peered":"2024-09-13T14:44:07.143411+0000","last_clean":"2024-09-13T14:44:07.143411+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:44:07.143411+0000","last_undegraded":"2024-09-13T14:44:07.143411+0000","last_fullsized":"2024-09-13T14:44:07.143411+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":128,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.188637+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:44:09.188637+0000","last_peered":"2024-09-13T14:44:09.188637+0000","last_clean":"2024-09-13T14:44:09.188637+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:44:09.188637+0000","last_undegraded":"2024-09-13T14:44:09.188637+0000","last_fullsized":"2024-09-13T14:44:09.188637+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":92,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.194289+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:44:09.194289+0000","last_peered":"2024-09-13T14:44:09.194289+0000","last_clean":"2024-09-13T14:44:09.194289+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:44:09.194289+0000","last_undegraded":"2024-09-13T14:44:09.194289+0000","last_fullsized":"2024-09-13T14:44:09.194289+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":104,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143501+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:44:07.143501+0000","last_peered":"2024-09-13T14:44:07.143501+0000","last_clean":"2024-09-13T14:44:07.143501+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:44:07.143501+0000","last_undegraded":"2024-09-13T14:44:07.143501+0000","last_fullsized":"2024-09-13T14:44:07.143501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":54,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143589+0000","last_change":"2024-09-13T14:44:04.125480+0000","last_active":"2024-09-13T14:44:07.143589+0000","last_peered":"2024-09-13T14:44:07.143589+0000","last_clean":"2024-09-13T14:44:07.143589+0000","last_became_active":"2024-09-13T14:44:04.114399+0000","last_became_peered":"2024-09-13T14:44:04.114399+0000","last_unstale":"2024-09-13T14:44:07.143589+0000","last_undegraded":"2024-09-13T14:44:07.143589+0000","last_fullsized":"2024-09-13T14:44:07.143589+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:01.892842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":101,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143379+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:44:07.143379+0000","last_peered":"2024-09-13T14:44:07.143379+0000","last_clean":"2024-09-13T14:44:07.143379+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:44:07.143379+0000","last_undegraded":"2024-09-13T14:44:07.143379+0000","last_fullsized":"2024-09-13T14:44:07.143379+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":54,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143794+0000","last_change":"2024-09-13T14:44:04.122804+0000","last_active":"2024-09-13T14:44:07.143794+0000","last_peered":"2024-09-13T14:44:07.143794+0000","last_clean":"2024-09-13T14:44:07.143794+0000","last_became_active":"2024-09-13T14:44:04.113000+0000","last_became_peered":"2024-09-13T14:44:04.113000+0000","last_unstale":"2024-09-13T14:44:07.143794+0000","last_undegraded":"2024-09-13T14:44:07.143794+0000","last_fullsized":"2024-09-13T14:44:07.143794+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:55:39.553036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":128,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.175174+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:44:09.175174+0000","last_peered":"2024-09-13T14:44:09.175174+0000","last_clean":"2024-09-13T14:44:09.175174+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:44:09.175174+0000","last_undegraded":"2024-09-13T14:44:09.175174+0000","last_fullsized":"2024-09-13T14:44:09.175174+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":123,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.175292+0000","last_change":"2024-09-13T14:44:04.110168+0000","last_active":"2024-09-13T14:44:09.175292+0000","last_peered":"2024-09-13T14:44:09.175292+0000","last_clean":"2024-09-13T14:44:09.175292+0000","last_became_active":"2024-09-13T14:44:04.109813+0000","last_became_peered":"2024-09-13T14:44:04.109813+0000","last_unstale":"2024-09-13T14:44:09.175292+0000","last_undegraded":"2024-09-13T14:44:09.175292+0000","last_fullsized":"2024-09-13T14:44:09.175292+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:28.575171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":95,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T14:44:07.143158+0000","last_change":"2024-09-13T14:44:04.114340+0000","last_active":"2024-09-13T14:44:07.143158+0000","last_peered":"2024-09-13T14:44:07.143158+0000","last_clean":"2024-09-13T14:44:07.143158+0000","last_became_active":"2024-09-13T14:44:04.113167+0000","last_became_peered":"2024-09-13T14:44:04.113167+0000","last_unstale":"2024-09-13T14:44:07.143158+0000","last_undegraded":"2024-09-13T14:44:07.143158+0000","last_fullsized":"2024-09-13T14:44:07.143158+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:16.843356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":130,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-13T14:44:09.186960+0000","last_change":"2024-09-13T14:44:04.126044+0000","last_active":"2024-09-13T14:44:09.186960+0000","last_peered":"2024-09-13T14:44:09.186960+0000","last_clean":"2024-09-13T14:44:09.186960+0000","last_became_active":"2024-09-13T14:44:04.115674+0000","last_became_peered":"2024-09-13T14:44:04.115674+0000","last_unstale":"2024-09-13T14:44:09.186960+0000","last_undegraded":"2024-09-13T14:44:09.186960+0000","last_fullsized":"2024-09-13T14:44:09.186960+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:50.859922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5933957120,"num_objects":90545,"num_object_clones":0,"num_object_copies":271635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10071,"num_objects_misplaced":8914,"num_objects_unfound":0,"num_objects_dirty":90545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90545,"num_write_kb":5794880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":59244544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17892573184,"data_stored":17892573184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90545,"ondisk_log_size":90545,"up":48,"acting":44,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":6060,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":22722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":12583007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234203,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2282052,"kb_used_data":2210748,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":91463100,"statfs":{"total":95995035648,"available":93658214400,"internally_reserved":0,"allocated":2263805952,"data_stored":2263137672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73005515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568495839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019114,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3649096,"kb_used_data":3563840,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":90096056,"statfs":{"total":95995035648,"available":92258361344,"internally_reserved":0,"allocated":3649372160,"data_stored":3648700153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468002,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1495988,"kb_used_data":1446316,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":92249164,"statfs":{"total":95995035648,"available":94463143936,"internally_reserved":0,"allocated":1481027584,"data_stored":1480370694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":50850980},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631528,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3299580,"kb_used_data":3213748,"kb_used_omap":8,"kb_used_meta":85815,"kb_avail":90445572,"statfs":{"total":95995035648,"available":92616265728,"internally_reserved":0,"allocated":3290877952,"data_stored":3290218063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":87875551},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795055,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2054448,"kb_used_data":1978344,"kb_used_omap":7,"kb_used_meta":76088,"kb_avail":91690704,"statfs":{"total":95995035648,"available":93891280896,"internally_reserved":0,"allocated":2025824256,"data_stored":2025171846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":77914738},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958582,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3791544,"kb_used_data":3701552,"kb_used_omap":7,"kb_used_meta":89976,"kb_avail":89953608,"statfs":{"total":95995035648,"available":92112494592,"internally_reserved":0,"allocated":3790389248,"data_stored":3789729250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":92135911},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3796631552,"data_stored":3796631552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2039873536,"data_stored":2039873536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3310485504,"data_stored":3310485504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1484783616,"data_stored":1484783616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3680567296,"data_stored":3680567296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1303445504,"data_stored":1303445504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2276786176,"data_stored":2276786176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:44:25.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-13T14:44:25.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:25.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:25.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:25.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:25 smithi172 ceph-mon[100696]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:26.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:26.070 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:26.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:26.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:25 smithi154 ceph-mon[102314]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:26.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:25 smithi154 ceph-mon[107038]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:44:26.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:26.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:26.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:26.484 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:26.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 69 16 95637 95621 86.5909 44 0.0118668 0.011541 2024-09-13T14:44:26.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:26.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:26.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:26.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:26.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:26.873 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:26.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:26.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:26.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:26 smithi172 ceph-mon[100696]: pgmap v346: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 53 MiB/s wr, 843 op/s; 10071/279225 objects degraded (3.607%); 8710/279225 objects misplaced (3.119%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:27.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:27.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:27.125 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:26 smithi154 ceph-mon[102314]: pgmap v346: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 53 MiB/s wr, 843 op/s; 10071/279225 objects degraded (3.607%); 8710/279225 objects misplaced (3.119%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:27.125 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:26 smithi154 ceph-mon[107038]: pgmap v346: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 53 MiB/s wr, 843 op/s; 10071/279225 objects degraded (3.607%); 8710/279225 objects misplaced (3.119%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:27.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:27.246 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:27.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:27.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:27.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:27.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:27.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:27.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 70 16 96312 96296 85.9567 42.1875 0.0753703 0.0116259 2024-09-13T14:44:27.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:27.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:27.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:27.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:27.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:27.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:27.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:27.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:28.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:28.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:28.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:28.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:28.332 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:28.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:28.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:28.531 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 71 15 97101 97086 85.4414 49.375 0.0187341 0.0116964 2024-09-13T14:44:28.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:28.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:28.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:28.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:28.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:28.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:28.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:28 smithi154 ceph-mon[102314]: pgmap v347: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 42 MiB/s wr, 671 op/s; 10071/280563 objects degraded (3.590%); 8710/280563 objects misplaced (3.104%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:44:28.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:28.891 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:28 smithi154 ceph-mon[107038]: pgmap v347: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 42 MiB/s wr, 671 op/s; 10071/280563 objects degraded (3.590%); 8710/280563 objects misplaced (3.104%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:44:28.891 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:28.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:28.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:28.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:29.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:29.211 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:29.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:29.239 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:29.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:28 smithi172 ceph-mon[100696]: pgmap v347: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 42 MiB/s wr, 671 op/s; 10071/280563 objects degraded (3.590%); 8710/280563 objects misplaced (3.104%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:44:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:29.245 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:29.248 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:29.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:29.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:29.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:29.532 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 72 16 97751 97735 84.8182 40.5625 0.045518 0.0117835 2024-09-13T14:44:29.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:29.649 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:29.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:29.765 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:44:29.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:29.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:29.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:29.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:29 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3560854357' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:29 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:29 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/280563 objects degraded (3.590%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:29.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:29 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3560854357' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:29.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:29 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:29.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:29 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/280563 objects degraded (3.590%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:30.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:30.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:30.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:30.197 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:30.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:29 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3560854357' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:30.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:29 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:30.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:29 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/280563 objects degraded (3.590%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:30.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:30.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:30.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:30.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:30.533 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 73 16 98512 98496 84.3067 47.5625 0.0559374 0.011855 2024-09-13T14:44:30.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:30.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:30.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:30.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:30.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:30.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:30.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:30.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:30.941 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[102314]: pgmap v348: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 54 MiB/s wr, 867 op/s; 10071/289017 objects degraded (3.485%); 8376/289017 objects misplaced (2.898%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T14:44:30.941 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:30.941 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:30.941 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[102314]: osdmap e127: 8 total, 8 up, 6 in 2024-09-13T14:44:30.942 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[107038]: pgmap v348: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 54 MiB/s wr, 867 op/s; 10071/289017 objects degraded (3.485%); 8376/289017 objects misplaced (2.898%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T14:44:30.942 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:30.942 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:30.942 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:30 smithi154 ceph-mon[107038]: osdmap e127: 8 total, 8 up, 6 in 2024-09-13T14:44:30.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:31.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:31.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:30 smithi172 ceph-mon[100696]: pgmap v348: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 54 MiB/s wr, 867 op/s; 10071/289017 objects degraded (3.485%); 8376/289017 objects misplaced (2.898%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T14:44:31.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:30 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:31.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:30 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:31.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:30 smithi172 ceph-mon[100696]: osdmap e127: 8 total, 8 up, 6 in 2024-09-13T14:44:31.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:31.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:31.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.533 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 74 16 99296 99280 83.8296 49 0.0193391 0.0119228 2024-09-13T14:44:31.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:31.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:31.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:31.828 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:31.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:31.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:31.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:31.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:44:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:44:31] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:44:32.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:32.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:32.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:32.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:32.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:32.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:32.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:32.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:32.533 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 75 16 100064 100048 83.3519 48 0.0148776 0.0119912 2024-09-13T14:44:32.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:32.631 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:32.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:32.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:32.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:32.908 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:32.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:32.997 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:44:32.998 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:33.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:33.030 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:44:33.030 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:33.031 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:32 smithi154 ceph-mon[102314]: pgmap v350: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 714 op/s; 10071/291030 objects degraded (3.460%); 8376/291030 objects misplaced (2.878%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:44:33.031 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:32 smithi154 ceph-mon[107038]: pgmap v350: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 714 op/s; 10071/291030 objects degraded (3.460%); 8376/291030 objects misplaced (2.878%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:44:33.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:33.071 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:44:33.071 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:33.097 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:44:33.097 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:33.113 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:44:33.113 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:33.129 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:44:33.130 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:33.145 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:44:33.146 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:33.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:33.163 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:44:33.163 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:33.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:32 smithi172 ceph-mon[100696]: pgmap v350: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 714 op/s; 10071/291030 objects degraded (3.460%); 8376/291030 objects misplaced (2.878%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:44:33.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:33.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:33.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:33.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:33.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:33.533 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 76 16 100705 100689 82.7824 40.0625 0.00591601 0.012074 2024-09-13T14:44:33.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:33.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:33.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:33.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:33.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:33.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:33.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:33.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:34.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:34.046 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2196315418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:34.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2196315418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:34.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:34.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:34.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:44:34.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:34.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2196315418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:34.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:34.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:34.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:34.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:34.535 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 77 16 101395 101379 82.2674 43.125 0.0111911 0.0121492 2024-09-13T14:44:34.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:34.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:34.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:34.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:34.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:34.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:34.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:34.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:34.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:35.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:35.108 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:35.191 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:34 smithi154 ceph-mon[102314]: pgmap v351: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 712 op/s; 10071/293103 objects degraded (3.436%); 8247/293103 objects misplaced (2.814%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:44:35.192 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:34 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/293103 objects degraded (3.436%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:35.193 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:34 smithi154 ceph-mon[107038]: pgmap v351: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 712 op/s; 10071/293103 objects degraded (3.436%); 8247/293103 objects misplaced (2.814%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:44:35.193 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:34 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/293103 objects degraded (3.436%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:35.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:35.234 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:35.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:34 smithi172 ceph-mon[100696]: pgmap v351: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.0 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 712 op/s; 10071/293103 objects degraded (3.436%); 8247/293103 objects misplaced (2.814%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:44:35.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:34 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/293103 objects degraded (3.436%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:35.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:35.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:35.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:35.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:35.533 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 78 16 102128 102112 81.8 45.8125 0.0222245 0.0122194 2024-09-13T14:44:35.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:35.633 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:35.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:35.759 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:35.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:35.888 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:35.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:35.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:36.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:36.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:36.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:36.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:36.282 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:36.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.408 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:36.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:36.533 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 79 16 102883 102867 81.3619 47.1875 0.00712256 0.0122839 2024-09-13T14:44:36.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:36.640 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:36.641 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:36.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:36.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:36.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:36.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:36.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:36.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:36 smithi154 ceph-mon[102314]: pgmap v352: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 713 op/s; 10071/300642 objects degraded (3.350%); 8032/300642 objects misplaced (2.672%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:36.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:36 smithi154 ceph-mon[107038]: pgmap v352: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 713 op/s; 10071/300642 objects degraded (3.350%); 8032/300642 objects misplaced (2.672%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:36.975 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:36.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:37.041 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:36 smithi172 ceph-mon[100696]: pgmap v352: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 713 op/s; 10071/300642 objects degraded (3.350%); 8032/300642 objects misplaced (2.672%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:37.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:37.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:37.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:37.233 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:37.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:37.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:37.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:37.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:37.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:37.534 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:44:37.534168+0000 min lat: 0.00117109 max lat: 2.26349 avg lat: 0.0123523 2024-09-13T14:44:37.534 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:44:37.534 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 80 16 103616 103600 80.9172 45.8125 0.0392856 0.0123523 2024-09-13T14:44:37.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:37.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:37.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:44:37.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:37.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:37.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:37.938 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:37.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:38.025 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:38.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:38.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:38.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:44:38.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:38.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:38.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:38.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:38.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:38.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:38.534 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 81 16 104248 104232 80.4053 39.5 0.0101943 0.0124307 2024-09-13T14:44:38.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:38.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:38.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:38.774 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 1 pgp_num to 1 2024-09-13T14:44:38.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:38.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:38.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:38.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:39.034 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:39.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:39.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:39.155 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:38 smithi154 ceph-mon[102314]: pgmap v353: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 716 op/s; 10071/302067 objects degraded (3.334%); 8032/302067 objects misplaced (2.659%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:39.155 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1392077597' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:44:39.155 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:38 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:44:39.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:38 smithi154 ceph-mon[107038]: pgmap v353: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 716 op/s; 10071/302067 objects degraded (3.334%); 8032/302067 objects misplaced (2.659%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:39.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1392077597' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:44:39.156 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:38 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:44:39.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:38 smithi172 ceph-mon[100696]: pgmap v353: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 45 MiB/s wr, 716 op/s; 10071/302067 objects degraded (3.334%); 8032/302067 objects misplaced (2.659%); 4.2 MiB/s, 67 objects/s recovering 2024-09-13T14:44:39.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1392077597' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:44:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:38 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T14:44:39.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:39.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:39.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:39.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:39.534 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 82 16 104987 104971 79.9881 46.1875 0.0230952 0.0124959 2024-09-13T14:44:39.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:39.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:39.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:39.764 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:39.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:39.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:39.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T14:44:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[102314]: osdmap e128: 8 total, 8 up, 6 in 2024-09-13T14:44:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[102314]: pgmap v355: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 10071/310647 objects degraded (3.242%); 7680/310647 objects misplaced (2.472%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:44:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/302067 objects degraded (3.334%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:39.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T14:44:39.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[107038]: osdmap e128: 8 total, 8 up, 6 in 2024-09-13T14:44:39.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[107038]: pgmap v355: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 10071/310647 objects degraded (3.242%); 7680/310647 objects misplaced (2.472%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:44:39.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:39 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/302067 objects degraded (3.334%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:40.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:40.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:40.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:40.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:40.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:39 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T14:44:40.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:39 smithi172 ceph-mon[100696]: osdmap e128: 8 total, 8 up, 6 in 2024-09-13T14:44:40.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:39 smithi172 ceph-mon[100696]: pgmap v355: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 48 MiB/s wr, 773 op/s; 10071/310647 objects degraded (3.242%); 7680/310647 objects misplaced (2.472%); 4.8 MiB/s, 76 objects/s recovering 2024-09-13T14:44:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:39 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/302067 objects degraded (3.334%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:40.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:40.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:40.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:40.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:40.430 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:40.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:40.535 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 83 16 105637 105621 79.5138 40.625 0.0181524 0.012568 2024-09-13T14:44:40.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:40.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:40.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:40.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:40.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:40.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:40.975 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:40.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:41.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:41.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:41.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:41.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:41.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.373 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:41.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:41.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:41.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:41.536 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 84 16 106362 106346 79.1056 45.3125 0.0178364 0.0126348 2024-09-13T14:44:41.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:41.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:41.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:41.808 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:41.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:41.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:41.986 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:41 smithi154 ceph-mon[102314]: pgmap v356: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 45 MiB/s wr, 721 op/s; 10071/312696 objects degraded (3.221%); 7680/312696 objects misplaced (2.456%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:44:41.986 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:41 smithi154 ceph-mon[107038]: pgmap v356: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 45 MiB/s wr, 721 op/s; 10071/312696 objects degraded (3.221%); 7680/312696 objects misplaced (2.456%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:44:41.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:41.987 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:44:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:44:41] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:44:41.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:42.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:42.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:42.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:42.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:41 smithi172 ceph-mon[100696]: pgmap v356: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 45 MiB/s wr, 721 op/s; 10071/312696 objects degraded (3.221%); 7680/312696 objects misplaced (2.456%); 4.3 MiB/s, 69 objects/s recovering 2024-09-13T14:44:42.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:42.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:42.378 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:42.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:42.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:42.536 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 85 16 107086 107070 78.7073 45.25 0.0368326 0.0126998 2024-09-13T14:44:42.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:42.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:42.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:42.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:42.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:42.896 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:42.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:42.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:43.024 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:43.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:43.171 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:43.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:43.296 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:43.336 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:43.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:43.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:43.536 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 86 16 107739 107723 78.2667 40.8125 0.0175346 0.0127697 2024-09-13T14:44:43.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:43.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:43.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:43.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:43.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:43.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:43.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:43.894 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:44:43.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:44.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:43 smithi154 ceph-mon[102314]: pgmap v357: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 45 MiB/s wr, 717 op/s; 10071/314628 objects degraded (3.201%); 7536/314628 objects misplaced (2.395%); 4.4 MiB/s, 71 objects/s recovering 2024-09-13T14:44:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3445640943' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:43 smithi154 ceph-mon[107038]: pgmap v357: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 45 MiB/s wr, 717 op/s; 10071/314628 objects degraded (3.201%); 7536/314628 objects misplaced (2.395%); 4.4 MiB/s, 71 objects/s recovering 2024-09-13T14:44:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3445640943' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:43 smithi172 ceph-mon[100696]: pgmap v357: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 45 MiB/s wr, 717 op/s; 10071/314628 objects degraded (3.201%); 7536/314628 objects misplaced (2.395%); 4.4 MiB/s, 71 objects/s recovering 2024-09-13T14:44:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3445640943' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:44:44.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:44.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:44.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:44.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.536 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 87 16 108319 108303 77.7838 36.25 0.0055944 0.0128498 2024-09-13T14:44:44.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.588 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:44:44.588 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:44:44.589 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-13T14:44:44.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd in 6 2024-09-13T14:44:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:44.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:44.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:44.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:44.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:44.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3445640943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[102314]: osdmap e129: 8 total, 8 up, 6 in 2024-09-13T14:44:44.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/314628 objects degraded (3.201%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3445640943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[107038]: osdmap e129: 8 total, 8 up, 6 in 2024-09-13T14:44:44.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/314628 objects degraded (3.201%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:45.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:45.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:45.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:45.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:45.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:44 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:44:45.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3445640943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:44:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:44 smithi172 ceph-mon[100696]: osdmap e129: 8 total, 8 up, 6 in 2024-09-13T14:44:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/314628 objects degraded (3.201%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:45.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:45.309 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:45.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:45.439 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:45.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 88 16 108957 108941 77.353 39.875 0.00566576 0.0129218 2024-09-13T14:44:45.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:45.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:45.659 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:44:45.659 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:44:45.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:45.687 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:45.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:45.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:45.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:45.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:45.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:46.062 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:45 smithi154 ceph-mon[102314]: pgmap v359: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 52 MiB/s wr, 826 op/s; 10071/321921 objects degraded (3.128%); 7319/321921 objects misplaced (2.274%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:44:46.063 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:45 smithi154 ceph-mon[107038]: pgmap v359: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 52 MiB/s wr, 826 op/s; 10071/321921 objects degraded (3.128%); 7319/321921 objects misplaced (2.274%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:44:46.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:46.103 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:46.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:46.231 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:45 smithi172 ceph-mon[100696]: pgmap v359: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 52 MiB/s wr, 826 op/s; 10071/321921 objects degraded (3.128%); 7319/321921 objects misplaced (2.274%); 5.6 MiB/s, 89 objects/s recovering 2024-09-13T14:44:46.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:46.366 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:46.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:46.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:46.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 89 16 109662 109646 76.979 44.0625 0.0205487 0.0129846 2024-09-13T14:44:46.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:46.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:46.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.776 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:46.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:46.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:46.916 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:46.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:47.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:47.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:47.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:47.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:47.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:47.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:47.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:47.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total time run: 90.0208 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total writes made: 110350 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Write size: 65536 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object size: 65536 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Bandwidth (MB/sec): 76.6143 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Bandwidth: 43.4664 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max bandwidth (MB/sec): 231.875 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average IOPS: 1225 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev IOPS: 695.472 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max IOPS: 3710 2024-09-13T14:44:47.537 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min IOPS: 0 2024-09-13T14:44:47.538 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average Latency(s): 0.0130468 2024-09-13T14:44:47.538 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Latency(s): 0.0286788 2024-09-13T14:44:47.538 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max latency(s): 2.26349 2024-09-13T14:44:47.538 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min latency(s): 0.00117109 2024-09-13T14:44:47.544 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:44:47.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:47.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:47.662 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:44:47.662 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:44:47.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:47.705 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:44:47.705 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:44:47.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:47.741 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:44:47.741 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:44:47.776 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:44:47.776 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:44:47.818 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:44:47.819 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:44:47.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:47.851 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:44:47.851 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:44:47.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:47.884 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:44:47.885 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:44:47.919 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:44:47.919 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:44:47.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:47.986 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:47.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:48.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:44:48.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:48.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:48.139 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:47 smithi154 ceph-mon[102314]: pgmap v360: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 36 MiB/s wr, 568 op/s; 10071/323046 objects degraded (3.118%); 7319/323046 objects misplaced (2.266%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:44:48.139 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:47 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2774582150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:48.139 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:47 smithi154 ceph-mon[107038]: pgmap v360: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 36 MiB/s wr, 568 op/s; 10071/323046 objects degraded (3.118%); 7319/323046 objects misplaced (2.266%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:44:48.139 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:47 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2774582150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:48.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:47 smithi172 ceph-mon[100696]: pgmap v360: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 36 MiB/s wr, 568 op/s; 10071/323046 objects degraded (3.118%); 7319/323046 objects misplaced (2.266%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:44:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:47 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2774582150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:44:48.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:48.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:48.389 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:48.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:48.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:48.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:48.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:48.752 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:48.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:48.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:48.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:48.960 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:48.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:49.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.087 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:49.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:49.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:49.244 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:49.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:49.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:44:49.372 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:49.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:49.496 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:49.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:49.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:49.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:49.747 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:49.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:49.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:49.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:49.928 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:50.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:50.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:50.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:50.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:49 smithi154 ceph-mon[102314]: pgmap v361: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 40 MiB/s wr, 685 op/s; 10071/328782 objects degraded (3.063%); 6960/328782 objects misplaced (2.117%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:44:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:49 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/328782 objects degraded (3.063%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:50.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:49 smithi154 ceph-mon[107038]: pgmap v361: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 40 MiB/s wr, 685 op/s; 10071/328782 objects degraded (3.063%); 6960/328782 objects misplaced (2.117%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:44:50.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:49 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/328782 objects degraded (3.063%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:50.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:49 smithi172 ceph-mon[100696]: pgmap v361: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 40 MiB/s wr, 685 op/s; 10071/328782 objects degraded (3.063%); 6960/328782 objects misplaced (2.117%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:44:50.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:49 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/328782 objects degraded (3.063%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:50.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:50.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:50.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:50.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:50.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:50.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:50.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:50.715 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:50.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:44:50.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:50.842 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:50.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:50.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:50.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:51.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:51.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:51.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:51.247 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:51.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:51.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:51.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:51.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:51.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:51.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:51.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:51.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:51.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:51.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:51.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:52.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.045 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:52.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:52.135 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:51 smithi154 ceph-mon[102314]: pgmap v362: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 38 MiB/s wr, 683 op/s; 10071/328809 objects degraded (3.063%); 6960/328809 objects misplaced (2.117%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:44:52.136 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:44:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:44:51] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-13T14:44:52.136 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:51 smithi154 ceph-mon[107038]: pgmap v362: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 38 MiB/s wr, 683 op/s; 10071/328809 objects degraded (3.063%); 6960/328809 objects misplaced (2.117%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:44:52.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.196 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:52.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:52.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:51 smithi172 ceph-mon[100696]: pgmap v362: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 38 MiB/s wr, 683 op/s; 10071/328809 objects degraded (3.063%); 6960/328809 objects misplaced (2.117%); 4.5 MiB/s, 71 objects/s recovering 2024-09-13T14:44:52.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:52.321 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:52.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:52.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:52.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:52.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:52.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:52.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:52.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:52.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:52.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:52.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:52.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:53.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:53.120 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:53.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:53.238 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:53.243 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:53.339 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:53 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T14:44:53.333+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e129 definitely_dead 0 2024-09-13T14:44:53.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:53.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:53.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:53.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:53.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:53.641 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:53.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:53.766 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:53.823 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:44:53.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:53.892 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:53.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:53.945 INFO:teuthology.orchestra.run.smithi154.stderr:marked in osd.6. 2024-09-13T14:44:53.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:54.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:54.075 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:53 smithi154 ceph-mon[102314]: pgmap v363: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 34 MiB/s wr, 685 op/s; 10071/327075 objects degraded (3.079%); 6796/327075 objects misplaced (2.078%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:44:54.075 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:53 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2666160789' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T14:44:54.075 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:53 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T14:44:54.075 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:53 smithi154 ceph-mon[107038]: pgmap v363: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 34 MiB/s wr, 685 op/s; 10071/327075 objects degraded (3.079%); 6796/327075 objects misplaced (2.078%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:44:54.075 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:53 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2666160789' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T14:44:54.075 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:53 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T14:44:54.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:54.145 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:54.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:53 smithi172 ceph-mon[100696]: pgmap v363: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 85 active+clean; 6.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 102 B/s rd, 34 MiB/s wr, 685 op/s; 10071/327075 objects degraded (3.079%); 6796/327075 objects misplaced (2.078%); 4.6 MiB/s, 73 objects/s recovering 2024-09-13T14:44:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:53 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2666160789' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T14:44:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:53 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T14:44:54.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:54.269 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:54.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:54.393 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:54.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:54.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:54.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:54.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:54.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:54.776 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:54.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:54.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:54.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:55.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:55.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:55.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:55.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:55.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T14:44:55.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[102314]: osdmap e130: 8 total, 8 up, 7 in 2024-09-13T14:44:55.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10071/327075 objects degraded (3.079%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:55.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[102314]: osdmap e131: 8 total, 8 up, 7 in 2024-09-13T14:44:55.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T14:44:55.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[107038]: osdmap e130: 8 total, 8 up, 7 in 2024-09-13T14:44:55.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10071/327075 objects degraded (3.079%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:55.207 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:54 smithi154 ceph-mon[107038]: osdmap e131: 8 total, 8 up, 7 in 2024-09-13T14:44:55.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:54 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T14:44:55.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:54 smithi172 ceph-mon[100696]: osdmap e130: 8 total, 8 up, 7 in 2024-09-13T14:44:55.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:54 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10071/327075 objects degraded (3.079%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T14:44:55.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:54 smithi172 ceph-mon[100696]: osdmap e131: 8 total, 8 up, 7 in 2024-09-13T14:44:55.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:55.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:55.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:55.456 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:55.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:55.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:55.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:55.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:55.775 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:44:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:44:55.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:44:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:44:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:28397 5.0 5:0a7067a7:::benchmark_data_smithi154_144140_object28396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:28.389274+0000", 2024-09-13T14:44:55.776 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.382638087000004, 2024-09-13T14:44:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.131948498, 2024-09-13T14:44:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.777 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 28397 2024-09-13T14:44:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.778 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389274+0000", 2024-09-13T14:44:55.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389274+0000", 2024-09-13T14:44:55.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389276+0000", 2024-09-13T14:44:55.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.906e-06 2024-09-13T14:44:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389300+0000", 2024-09-13T14:44:55.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4051000000000001e-05 2024-09-13T14:44:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389318+0000", 2024-09-13T14:44:55.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7966999999999999e-05 2024-09-13T14:44:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389327+0000", 2024-09-13T14:44:55.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7309999999999999e-06 2024-09-13T14:44:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389404+0000", 2024-09-13T14:44:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6691000000000001e-05 2024-09-13T14:44:55.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389546+0000", 2024-09-13T14:44:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000142022 2024-09-13T14:44:55.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.389674+0000", 2024-09-13T14:44:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000128071 2024-09-13T14:44:55.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.391992+0000", 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023187680000000001 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.392067+0000", 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4350999999999993e-05 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.520888+0000", 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.128821556 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.521167+0000", 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000279086 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:28.521223+0000", 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5297999999999998e-05 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:33396 5.0 5:0e3ab43d:::benchmark_data_smithi154_144140_object33395:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.025699+0000", 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.746213706000006, 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.060518319000000001, 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.791 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 33396 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025699+0000", 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025699+0000", 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025700+0000", 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.849e-06 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025724+0000", 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3346999999999999e-05 2024-09-13T14:44:55.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025738+0000", 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.394e-05 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025746+0000", 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0450000000000001e-06 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025805+0000", 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8807999999999997e-05 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.025930+0000", 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012516400000000001 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.026058+0000", 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000127785 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029129+0000", 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030713030000000001 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029162+0000", 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3179000000000002e-05 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086057+0000", 2024-09-13T14:44:55.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.056895183000000002 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086114+0000", 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6619000000000001e-05 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.086217+0000", 2024-09-13T14:44:55.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010309700000000001 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:33403 5.0 5:094967b0:::benchmark_data_smithi154_144140_object33402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:31.029256+0000", 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.742656554999996, 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058615654000000003, 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.799 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 33403 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029256+0000", 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029256+0000", 2024-09-13T14:44:55.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029257+0000", 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.313e-06 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029278+0000", 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0829e-05 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029288+0000", 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0112000000000001e-05 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029294+0000", 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4690000000000002e-06 2024-09-13T14:44:55.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029346+0000", 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.168e-05 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029435+0000", 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8422000000000001e-05 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.029534+0000", 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9531999999999995e-05 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.030753+0000", 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012189779999999999 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.030797+0000", 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4283999999999998e-05 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.087622+0000", 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.056824223 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.087761+0000", 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000139259 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:31.087871+0000", 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110553 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:64812 5.0 5:0b0e492b:::benchmark_data_smithi154_144140_object64811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:50.823900+0000", 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.948012797000004, 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058505176999999998, 2024-09-13T14:44:55.806 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 64812 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.823900+0000", 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.823900+0000", 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.823901+0000", 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.44e-06 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.823922+0000", 2024-09-13T14:44:55.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1430999999999999e-05 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.823933+0000", 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0890000000000001e-05 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.823940+0000", 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3879999999999997e-06 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.824764+0000", 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00082421000000000003 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.824854+0000", 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0137999999999997e-05 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.824942+0000", 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8232000000000005e-05 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.826445+0000", 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001503248 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.827470+0000", 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010246610000000001 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.882211+0000", 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.054741208 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.882307+0000", 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5642000000000003e-05 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:50.882405+0000", 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7689000000000006e-05 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.812 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:94863 5.0 5:031029ad:::benchmark_data_smithi154_144140_object94862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:25.446357+0000", 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.325555138999999, 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047925980999999999, 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.813 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 94863 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.446357+0000", 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.446357+0000", 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.446359+0000", 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9180000000000001e-06 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.446383+0000", 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4026999999999999e-05 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.446403+0000", 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0122000000000001e-05 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.446412+0000", 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8389999999999996e-06 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.448996+0000", 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025842180000000001 2024-09-13T14:44:55.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.449292+0000", 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000295371 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.449675+0000", 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038287100000000001 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.453389+0000", 2024-09-13T14:44:55.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037140950000000002 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.453511+0000", 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012249099999999999 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.493985+0000", 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.040473756999999999 2024-09-13T14:44:55.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.494153+0000", 2024-09-13T14:44:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000168407 2024-09-13T14:44:55.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.494283+0000", 2024-09-13T14:44:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000129865 2024-09-13T14:44:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:95252 5.0 5:0662dd34:::benchmark_data_smithi154_144140_object95251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:44:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:25.962885+0000", 2024-09-13T14:44:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.809027182000001, 2024-09-13T14:44:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.053509503999999999, 2024-09-13T14:44:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.822 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 95252 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.962885+0000", 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.962885+0000", 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.962887+0000", 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.567e-06 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.962913+0000", 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6030999999999999e-05 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.962925+0000", 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2758999999999999e-05 2024-09-13T14:44:55.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.962932+0000", 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6710000000000002e-06 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.963177+0000", 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024506099999999998 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.963328+0000", 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015049599999999999 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.963469+0000", 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014087999999999999 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.966840+0000", 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033713770000000001 2024-09-13T14:44:55.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:25.966929+0000", 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9339999999999995e-05 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.016290+0000", 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049360835999999998 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.016351+0000", 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1166000000000004e-05 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.016395+0000", 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3319999999999999e-05 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.828 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:95782 5.0 5:042a0d0c:::benchmark_data_smithi154_144140_object95781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:26.719145+0000", 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.05276757, 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.042612292000000003, 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 95782 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719145+0000", 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719145+0000", 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719146+0000", 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7269999999999999e-06 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719170+0000", 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3757999999999999e-05 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719186+0000", 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5724000000000001e-05 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719193+0000", 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3440000000000003e-06 2024-09-13T14:44:55.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719247+0000", 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3560000000000002e-05 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719346+0000", 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8876000000000007e-05 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.719437+0000", 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1113999999999995e-05 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.721888+0000", 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002451052 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.722366+0000", 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047816200000000002 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.761440+0000", 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039073937000000003 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.761626+0000", 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185902 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:26.761757+0000", 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013113600000000001 2024-09-13T14:44:55.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:97381 5.0 5:0c71132f:::benchmark_data_smithi154_144140_object97380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:28.935150+0000", 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.836762801999999, 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047031199000000003, 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 97381 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935150+0000", 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935150+0000", 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935152+0000", 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1830000000000001e-06 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935175+0000", 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3368000000000001e-05 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935196+0000", 2024-09-13T14:44:55.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0891999999999999e-05 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935206+0000", 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0013999999999999e-05 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935374+0000", 2024-09-13T14:44:55.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016754699999999999 2024-09-13T14:44:55.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.935671+0000", 2024-09-13T14:44:55.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029776299999999999 2024-09-13T14:44:55.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.936132+0000", 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046033199999999998 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.943680+0000", 2024-09-13T14:44:55.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075481410000000004 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.951076+0000", 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073958899999999996 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.981772+0000", 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030696119000000001 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.982014+0000", 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024220399999999999 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:28.982181+0000", 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016674600000000001 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:97549 5.0 5:085b6106:::benchmark_data_smithi154_144140_object97548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:29.203787+0000", 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.568125402, 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041060549000000002, 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.843 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 97549 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.203787+0000", 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.203787+0000", 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.203789+0000", 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.643e-06 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.203811+0000", 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2923999999999999e-05 2024-09-13T14:44:55.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.203828+0000", 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6861e-05 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.203836+0000", 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8800000000000008e-06 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.212371+0000", 2024-09-13T14:44:55.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085344459999999994 2024-09-13T14:44:55.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.216004+0000", 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003633627 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.216175+0000", 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000170454 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.219512+0000", 2024-09-13T14:44:55.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033376759999999999 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.223476+0000", 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039631199999999997 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.244728+0000", 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021252149000000001 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.244795+0000", 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7354999999999995e-05 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:29.244847+0000", 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2414000000000003e-05 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:99618 5.0 5:04f32433:::benchmark_data_smithi154_144140_object99617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:31.948025+0000", 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.823886988000002, 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.040129275999999998, 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 99618 2024-09-13T14:44:55.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.948025+0000", 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.948025+0000", 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.948027+0000", 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4330000000000001e-06 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.948049+0000", 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2679000000000001e-05 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.948062+0000", 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2595999999999999e-05 2024-09-13T14:44:55.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.948069+0000", 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0809999999999996e-06 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.952567+0000", 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044974669999999998 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.952713+0000", 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014639300000000001 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.952869+0000", 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015610600000000001 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.962953+0000", 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010083572000000001 2024-09-13T14:44:55.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.963936+0000", 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00098300800000000006 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.988041+0000", 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024104863000000001 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.988106+0000", 2024-09-13T14:44:55.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5840000000000007e-05 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:31.988155+0000", 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8238000000000001e-05 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:100089 5.0 5:020e2a0a:::benchmark_data_smithi154_144140_object100088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:32.571475+0000", 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.200437496999999, 2024-09-13T14:44:55.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.040329197999999997, 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 100089 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.571475+0000", 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.571475+0000", 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.571477+0000", 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7099999999999999e-06 2024-09-13T14:44:55.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.571499+0000", 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2932999999999999e-05 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.571517+0000", 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7994000000000001e-05 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.571526+0000", 2024-09-13T14:44:55.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3299999999999999e-06 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.571684+0000", 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015813399999999999 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.572002+0000", 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000317991 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.572664+0000", 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066246499999999999 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.576901+0000", 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042366289999999996 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.580243+0000", 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033423139999999999 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.611314+0000", 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031070632000000001 2024-09-13T14:44:55.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.611650+0000", 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033596100000000002 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.611804+0000", 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000154105 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:100237 5.0 5:0bef09d6:::benchmark_data_smithi154_144140_object100236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:32.761060+0000", 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.010851874, 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039577018999999998, 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.865 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 100237 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.761060+0000", 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.761060+0000", 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.761061+0000", 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3900000000000005e-07 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.761086+0000", 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4641e-05 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.761093+0000", 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0929999999999997e-06 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.761098+0000", 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8899999999999998e-06 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.764685+0000", 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00358786 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.764808+0000", 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122688 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.765101+0000", 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029307299999999998 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.770465+0000", 2024-09-13T14:44:55.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053633220000000002 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.778246+0000", 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077818540000000004 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.800331+0000", 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022084724999999999 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.800504+0000", 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017237900000000001 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:32.800637+0000", 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000133955 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:111547 5.0 5:0a6949cd:::benchmark_data_smithi154_144140_object1194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:49.185045+0000", 2024-09-13T14:44:55.871 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 6.5868674179999998, 2024-09-13T14:44:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.043144824999999998, 2024-09-13T14:44:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.873 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 111547 2024-09-13T14:44:55.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.873 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185045+0000", 2024-09-13T14:44:55.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185045+0000", 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185048+0000", 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7630000000000001e-06 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185055+0000", 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.593e-06 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185078+0000", 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3082999999999999e-05 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185088+0000", 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.577e-06 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185158+0000", 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9611999999999997e-05 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185514+0000", 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035636799999999997 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185979+0000", 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046547 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.193736+0000", 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077566190000000002 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.193788+0000", 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1941000000000003e-05 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.228080+0000", 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034292372000000002 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.228142+0000", 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1476000000000006e-05 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.228190+0000", 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7951000000000002e-05 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.880 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:111549 5.0 5:02e356b8:::benchmark_data_smithi154_144140_object1196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:49.185331+0000", 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 6.5865811050000005, 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.042973790999999997, 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 111549 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185331+0000", 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185331+0000", 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185332+0000", 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.052e-06 2024-09-13T14:44:55.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185337+0000", 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6779999999999999e-06 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185505+0000", 2024-09-13T14:44:55.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000167694 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.185545+0000", 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0272000000000003e-05 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.189927+0000", 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043817359999999998 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.190054+0000", 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012746799999999999 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.190161+0000", 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106805 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.193818+0000", 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003656734 2024-09-13T14:44:55.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.193941+0000", 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000123399 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.228227+0000", 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034286297 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.228271+0000", 2024-09-13T14:44:55.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3537e-05 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.228305+0000", 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4119000000000003e-05 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.887 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:111748 5.0 5:0a846748:::benchmark_data_smithi154_144140_object1395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:49.457261+0000", 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 6.3146512870000002, 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.044867487999999997, 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 111748 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457261+0000", 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457261+0000", 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457263+0000", 2024-09-13T14:44:55.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8929999999999999e-06 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457269+0000", 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3400000000000003e-06 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457284+0000", 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4644000000000001e-05 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457292+0000", 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2120000000000003e-06 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.459356+0000", 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020638779999999999 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.459438+0000", 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1866000000000005e-05 2024-09-13T14:44:55.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.459528+0000", 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0056000000000006e-05 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.466854+0000", 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073256770000000001 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.467248+0000", 2024-09-13T14:44:55.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039423600000000001 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.501652+0000", 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034404399000000002 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.501978+0000", 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032574399999999999 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.502129+0000", 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000150543 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:111749 5.0 5:09b81e4e:::benchmark_data_smithi154_144140_object1396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:49.457636+0000", 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 6.3142761289999996, 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.045037682000000002, 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.894 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 111749 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457636+0000", 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457636+0000", 2024-09-13T14:44:55.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457637+0000", 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0950000000000001e-06 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457642+0000", 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0069999999999997e-06 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457654+0000", 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1294000000000001e-05 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.457661+0000", 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1300000000000003e-06 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.462172+0000", 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045109369999999996 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.462278+0000", 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010631100000000001 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.462385+0000", 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107099 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.467233+0000", 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048480889999999999 2024-09-13T14:44:55.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.468274+0000", 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010403949999999999 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.502312+0000", 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034038018000000003 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.502552+0000", 2024-09-13T14:44:55.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024068099999999999 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.502674+0000", 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121626 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:112077 5.0 5:0f03c9cc:::benchmark_data_smithi154_144140_object1724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:49.908172+0000", 2024-09-13T14:44:55.900 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.8637405229999997, 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.045649132000000002, 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 112077 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908172+0000", 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908172+0000", 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908175+0000", 2024-09-13T14:44:55.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7690000000000001e-06 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908183+0000", 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3990000000000004e-06 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908207+0000", 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3626e-05 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908217+0000", 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0098000000000001e-05 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908293+0000", 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5924e-05 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908444+0000", 2024-09-13T14:44:55.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000151636 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.908584+0000", 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000139806 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.911895+0000", 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033107649999999998 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.912129+0000", 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00023455200000000001 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.953429+0000", 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041299324999999998 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.953674+0000", 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024536499999999999 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:49.953821+0000", 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014686699999999999 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:113995 5.0 5:035f57bf:::benchmark_data_smithi154_144140_object3642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:52.570525+0000", 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.2013875299999999, 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.040993847, 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.907 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 113995 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570525+0000", 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570525+0000", 2024-09-13T14:44:55.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:55.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570526+0000", 2024-09-13T14:44:55.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5400000000000001e-06 2024-09-13T14:44:55.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570532+0000", 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2020000000000003e-06 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570547+0000", 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5237e-05 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570555+0000", 2024-09-13T14:44:55.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0800000000000006e-06 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570657+0000", 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102111 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570754+0000", 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6935000000000006e-05 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.570826+0000", 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1967999999999999e-05 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.573501+0000", 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026754539999999999 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.575781+0000", 2024-09-13T14:44:55.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022797490000000002 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.611246+0000", 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035465216000000001 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.611402+0000", 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000155606 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.611519+0000", 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116749 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:114212 5.0 5:0a9f2995:::benchmark_data_smithi154_144140_object3859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:52.894476+0000", 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.8774367320000001, 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047864227000000002, 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.915 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 114212 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.894476+0000", 2024-09-13T14:44:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.894476+0000", 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.894478+0000", 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.922e-06 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.894484+0000", 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9489999999999999e-06 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.894501+0000", 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6940999999999999e-05 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.894511+0000", 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4679999999999998e-06 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.908925+0000", 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014413812999999999 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.909054+0000", 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000129157 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.909176+0000", 2024-09-13T14:44:55.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012223600000000001 2024-09-13T14:44:55.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.920627+0000", 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011451403000000001 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.920889+0000", 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000261599 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.941980+0000", 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021090658000000002 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.942196+0000", 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021638399999999999 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.942340+0000", 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000143697 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:114214 5.0 5:00bf76bd:::benchmark_data_smithi154_144140_object3861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:52.901287+0000", 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.8706250349999998, 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041245394999999997, 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:44:55.923 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 114214 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.901287+0000", 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.901287+0000", 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.901289+0000", 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.235e-06 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:44:55.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.901302+0000", 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3465000000000001e-05 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.901316+0000", 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4395e-05 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.901326+0000", 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8509999999999994e-06 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.916011+0000", 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014684282 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.916321+0000", 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031005799999999999 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.916663+0000", 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000342126 2024-09-13T14:44:55.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.920815+0000", 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004152251 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.920912+0000", 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6922999999999993e-05 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.942438+0000", 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021526503999999998 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.942493+0000", 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4432000000000003e-05 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:52.942533+0000", 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9873000000000003e-05 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.929 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.930 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.930 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:44:55.930 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:44:55.930 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:44:55.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:55.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:56.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:56.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:56.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.216 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:56.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:56.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:56.355 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:56.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:56.488 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:56.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:56.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:56.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:56.768 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:56.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:56.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:56.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:56.954 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:56 smithi154 ceph-mon[102314]: pgmap v366: 89 pgs: 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 79 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 127 B/s rd, 21 MiB/s wr, 814 op/s; 10071/319668 objects degraded (3.150%); 6574/319668 objects misplaced (2.057%); 5.8 MiB/s, 93 objects/s recovering 2024-09-13T14:44:56.955 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:56 smithi154 ceph-mon[102314]: osdmap e132: 8 total, 8 up, 7 in 2024-09-13T14:44:56.955 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:56 smithi154 ceph-mon[107038]: pgmap v366: 89 pgs: 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 79 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 127 B/s rd, 21 MiB/s wr, 814 op/s; 10071/319668 objects degraded (3.150%); 6574/319668 objects misplaced (2.057%); 5.8 MiB/s, 93 objects/s recovering 2024-09-13T14:44:56.955 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:56 smithi154 ceph-mon[107038]: osdmap e132: 8 total, 8 up, 7 in 2024-09-13T14:44:56.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:56 smithi172 ceph-mon[100696]: pgmap v366: 89 pgs: 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 79 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 127 B/s rd, 21 MiB/s wr, 814 op/s; 10071/319668 objects degraded (3.150%); 6574/319668 objects misplaced (2.057%); 5.8 MiB/s, 93 objects/s recovering 2024-09-13T14:44:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:56 smithi172 ceph-mon[100696]: osdmap e132: 8 total, 8 up, 7 in 2024-09-13T14:44:57.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:57.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:57.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:57.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:57.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:57.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:44:57.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:44:57.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:57.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:57.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:57.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:57.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:57.803 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:44:57.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:57.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:57.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:57.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:57.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:57.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1923738654' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:57.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:57 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:58.002 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1923738654' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:58.002 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:57 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:58.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1923738654' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:58.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:57 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:44:58.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:58.104 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:58.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:58.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:58.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:58.355 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:58.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:58.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:58.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:44:58.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:44:58.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:58.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:58.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:58.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:58.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:58.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:58 smithi172 ceph-mon[100696]: pgmap v368: 89 pgs: 1 remapped+peering, 1 active+recovery_wait+degraded, 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 77 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 458 KiB/s wr, 589 op/s; 10079/318456 objects degraded (3.165%); 6574/318456 objects misplaced (2.064%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:44:58.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:58 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:58.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:58.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:58 smithi172 ceph-mon[100696]: osdmap e133: 8 total, 8 up, 7 in 2024-09-13T14:44:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:59.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:59.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:59.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:59.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[107038]: pgmap v368: 89 pgs: 1 remapped+peering, 1 active+recovery_wait+degraded, 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 77 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 458 KiB/s wr, 589 op/s; 10079/318456 objects degraded (3.165%); 6574/318456 objects misplaced (2.064%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[107038]: osdmap e133: 8 total, 8 up, 7 in 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[102314]: pgmap v368: 89 pgs: 1 remapped+peering, 1 active+recovery_wait+degraded, 6 peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 77 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 458 KiB/s wr, 589 op/s; 10079/318456 objects degraded (3.165%); 6574/318456 objects misplaced (2.064%); 4.0 MiB/s, 64 objects/s recovering 2024-09-13T14:44:59.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:44:59.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:44:59.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[102314]: osdmap e133: 8 total, 8 up, 7 in 2024-09-13T14:44:59.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:44:59.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:44:59.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:44:59.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:44:59.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:44:59.443 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:44:59.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:44:59.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:44:59.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:44:59.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:44:59.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:44:59.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:44:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:44:59.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:44:59.972 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:44:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:44:59.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:44:59.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:44:59 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 10079/318456 objects degraded (3.165%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:00.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:00.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:00.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:00.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:44:59 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 10079/318456 objects degraded (3.165%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:00.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:44:59 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 10079/318456 objects degraded (3.165%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:00.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:00.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:00.348 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:00.369 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:00.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:00.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:00.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:00.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:00.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:00.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:00.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:00.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:00.900 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:00.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:00.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:00 smithi172 ceph-mon[100696]: pgmap v370: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+recovery_wait+degraded, 6 peering, 5 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 688 op/s; 6626/310122 objects degraded (2.137%); 1640/310122 objects misplaced (0.529%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:45:01.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:01.032 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:00 smithi154 ceph-mon[107038]: pgmap v370: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+recovery_wait+degraded, 6 peering, 5 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 688 op/s; 6626/310122 objects degraded (2.137%); 1640/310122 objects misplaced (0.529%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:45:01.033 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:00 smithi154 ceph-mon[102314]: pgmap v370: 89 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+recovery_wait+degraded, 6 peering, 5 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 688 op/s; 6626/310122 objects degraded (2.137%); 1640/310122 objects misplaced (0.529%); 5.1 MiB/s, 82 objects/s recovering 2024-09-13T14:45:01.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:01.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:01.170 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:01.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:01.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:01.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:01.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:01.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:01.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:01.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:01.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:01.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:01.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:01.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:01.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:45:01.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:01.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:01.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.015 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-13T14:45:02.091 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:45:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:45:01] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:45:02.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:02.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:02.189 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:02.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:02.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:02.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:02.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:02.546 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:02.681 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:02.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:02.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:02.879 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:02 smithi172 ceph-mon[100696]: pgmap v371: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 6632/307932 objects degraded (2.154%); 1640/307932 objects misplaced (0.533%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:45:02.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:02.951 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:02.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:03.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:03.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:03.087 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:02 smithi154 ceph-mon[102314]: pgmap v371: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 6632/307932 objects degraded (2.154%); 1640/307932 objects misplaced (0.533%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:45:03.087 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:02 smithi154 ceph-mon[107038]: pgmap v371: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 6632/307932 objects degraded (2.154%); 1640/307932 objects misplaced (0.533%); 4.1 MiB/s, 65 objects/s recovering 2024-09-13T14:45:03.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:03.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:03.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:03.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:03.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:03.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:45:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:03.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:03.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:03.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:45:03.693 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:45:03.733 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:45:03.769 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:03.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:03.805 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:45:03.805 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:03.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:03.823 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:45:03.824 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:03.841 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:03.851 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/571128505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:03.851 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/571128505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:03.851 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:45:03.851 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:03.880 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:45:03.881 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:03.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:03.920 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:45:03.920 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:03.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:03.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:04.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:04.064 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/571128505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:04.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:04.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:04.219 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:04.220 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.232 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:04.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:04.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:04.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:04.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:04.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:04.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:04.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:04.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:04.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:04.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:04.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:04.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:04.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:45:05.014 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:04 smithi154 ceph-mon[102314]: pgmap v372: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 588 op/s; 6632/305526 objects degraded (2.171%); 1640/305526 objects misplaced (0.537%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:45:05.014 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:04 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6632/305526 objects degraded (2.171%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:05.014 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:04 smithi154 ceph-mon[107038]: pgmap v372: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 588 op/s; 6632/305526 objects degraded (2.171%); 1640/305526 objects misplaced (0.537%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:45:05.014 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:04 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6632/305526 objects degraded (2.171%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:05.015 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:04 smithi172 ceph-mon[100696]: pgmap v372: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 588 op/s; 6632/305526 objects degraded (2.171%); 1640/305526 objects misplaced (0.537%); 3.1 MiB/s, 50 objects/s recovering 2024-09-13T14:45:05.015 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:04 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6632/305526 objects degraded (2.171%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:05.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:05.066 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:05.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:05.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.326 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:05.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:05.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.366 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:05.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:05.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:05.505 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:05.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.652 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:05.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:05.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:05.784 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:05.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:05.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:05.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:06.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:06.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:06.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:06.215 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:06.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:06.355 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:06.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:06.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:06.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:06.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:06.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:06.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:06.800 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:06 smithi154 ceph-mon[102314]: pgmap v373: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 813 op/s; 6632/296199 objects degraded (2.239%); 1182/296199 objects misplaced (0.399%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:45:06.800 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:06 smithi154 ceph-mon[107038]: pgmap v373: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 813 op/s; 6632/296199 objects degraded (2.239%); 1182/296199 objects misplaced (0.399%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:45:06.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:06.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:06.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:06.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:06 smithi172 ceph-mon[100696]: pgmap v373: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 813 op/s; 6632/296199 objects degraded (2.239%); 1182/296199 objects misplaced (0.399%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:45:07.015 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7, 6] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:45:07.015 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:45:07.016 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T14:45:07.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T14:45:07.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:07.060 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:07.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:07.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:07.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:07.350 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:07.400 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:07.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:07.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:07.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:07.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:07.701 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:07.701 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:07.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:07.790 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:45:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:07.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:07.812 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2229060847' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:07.812 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:07 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:07.813 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2229060847' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:07.813 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:07 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:07.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:07.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:07.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:07.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2229060847' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:07 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:45:08.027 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:08.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:08.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:08.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:08.190 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:08.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:08.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:08.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:45:08.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.475 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:08.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:08.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:08.614 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:08.673 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:08.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:08.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:08.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:08.908 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:08.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:09.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:09.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[102314]: pgmap v374: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 6632/294387 objects degraded (2.253%); 1182/294387 objects misplaced (0.402%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[102314]: osdmap e134: 8 total, 8 up, 7 in 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[107038]: pgmap v374: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 6632/294387 objects degraded (2.253%); 1182/294387 objects misplaced (0.402%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:09.058 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:08 smithi154 ceph-mon[107038]: osdmap e134: 8 total, 8 up, 7 in 2024-09-13T14:45:09.093 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:08 smithi172 ceph-mon[100696]: pgmap v374: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 6632/294387 objects degraded (2.253%); 1182/294387 objects misplaced (0.402%); 5.4 MiB/s, 85 objects/s recovering 2024-09-13T14:45:09.094 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:08 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:09.094 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:08 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:45:09.094 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:08 smithi172 ceph-mon[100696]: osdmap e134: 8 total, 8 up, 7 in 2024-09-13T14:45:09.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:09.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:09.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:09.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:09.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:09.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:09.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:09.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:09.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:09.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:09.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:09.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:09.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:09.882 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:09 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6632/294387 objects degraded (2.253%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:09 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6632/294387 objects degraded (2.253%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:09.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:09 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6632/294387 objects degraded (2.253%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:09.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:10.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:10.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:10.127 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:10.229 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:10.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:10.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:10.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:10.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:10.526 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:10.527 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:10.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:10.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:10.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:10.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:10.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:10.873 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:10.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:10.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:11.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:11.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:11.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:10 smithi154 ceph-mon[102314]: pgmap v376: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 6632/283569 objects degraded (2.339%); 701/283569 objects misplaced (0.247%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:11.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:10 smithi154 ceph-mon[107038]: pgmap v376: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 6632/283569 objects degraded (2.339%); 701/283569 objects misplaced (0.247%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:11.229 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:10 smithi172 ceph-mon[100696]: pgmap v376: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 6632/283569 objects degraded (2.339%); 701/283569 objects misplaced (0.247%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:11.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:11.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:45:11.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:45:11.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:11.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-13T14:45:11.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:11.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:11.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:11.551 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:11.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:11.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:11.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:11.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:11.872 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:11 smithi154 ceph-mon[102314]: pgmap v377: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 896 op/s; 6632/281025 objects degraded (2.360%); 701/281025 objects misplaced (0.249%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:11.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:11 smithi154 ceph-mon[107038]: pgmap v377: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 896 op/s; 6632/281025 objects degraded (2.360%); 701/281025 objects misplaced (0.249%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:11.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:11.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:11.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:11.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:11 smithi172 ceph-mon[100696]: pgmap v377: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 896 op/s; 6632/281025 objects degraded (2.360%); 701/281025 objects misplaced (0.249%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:12.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:12.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:12.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:45:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:45:11] "GET /metrics HTTP/1.1" 200 36549 "" "Prometheus/2.51.0" 2024-09-13T14:45:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:12.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:12.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:12.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:12.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:12.524 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:12.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:12.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:12.773 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:12.774 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:12.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:12.923 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:12.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:13.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:13.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:13.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:13.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:13.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:13.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:13.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.485 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:13.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:13.486 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:13.486 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:13.587 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:13.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:13.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:13.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:13.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:13.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:13.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:13.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:14.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:14.130 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:13 smithi172 ceph-mon[100696]: pgmap v378: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 6632/278670 objects degraded (2.380%); 701/278670 objects misplaced (0.252%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:14.130 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:14.130 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:14.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:14.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:45:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.175 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:14.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:13 smithi154 ceph-mon[102314]: pgmap v378: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 6632/278670 objects degraded (2.380%); 701/278670 objects misplaced (0.252%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:13 smithi154 ceph-mon[107038]: pgmap v378: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 894 op/s; 6632/278670 objects degraded (2.380%); 701/278670 objects misplaced (0.252%); 5.9 MiB/s, 93 objects/s recovering 2024-09-13T14:45:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:14.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:14.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:14.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:14.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:14.506 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:14.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:14.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:14.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:14.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:14.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:14.873 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:14.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:14.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:15.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:15.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:15.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:14 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6632/278670 objects degraded (2.380%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:14 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6632/278670 objects degraded (2.380%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:14 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6632/278670 objects degraded (2.380%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:15.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:15.285 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:15.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:15.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:15.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:15.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:15.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:15.665 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:15.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.797 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:15.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:15.834 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:45:15.834 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:15.835 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:45:15.835 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:15.836 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:45:15.836 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:15.837 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:45:15.837 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:15.837 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:45:15.838 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:15.838 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:45:15.839 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:15.863 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:45:15.864 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:15.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:15.900 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:45:15.900 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:15.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:15.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:15.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:15 smithi172 ceph-mon[100696]: pgmap v379: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 6632/269607 objects degraded (2.460%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:45:15.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2199159629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:16.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:16.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:16.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:16.201 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:15 smithi154 ceph-mon[102314]: pgmap v379: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 6632/269607 objects degraded (2.460%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:45:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2199159629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:16.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:15 smithi154 ceph-mon[107038]: pgmap v379: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 6632/269607 objects degraded (2.460%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:45:16.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2199159629' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:16.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:16.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:16.344 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:45:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:45:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:45:16.344 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:45:16.344 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:45:16.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:16.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:16.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:16.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:16.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-13T14:45:16.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:16.706 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:16.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:16.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:16.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:16.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:16.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:17.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:17.084 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:16 smithi172 ceph-mon[100696]: osdmap e135: 8 total, 8 up, 7 in 2024-09-13T14:45:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.088 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:17.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:17.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:45:17.189 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:17.197 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:16 smithi154 ceph-mon[102314]: osdmap e135: 8 total, 8 up, 7 in 2024-09-13T14:45:17.198 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:16 smithi154 ceph-mon[107038]: osdmap e135: 8 total, 8 up, 7 in 2024-09-13T14:45:17.233 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:17.330 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:17.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:17.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:17.454 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:17.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:17.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:17.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:17.658 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:17.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:17.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:17.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:17.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:17.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:17.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:18.028 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:17 smithi154 ceph-mon[102314]: pgmap v381: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 6632/267891 objects degraded (2.476%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:45:18.028 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:17 smithi154 ceph-mon[102314]: osdmap e136: 8 total, 8 up, 7 in 2024-09-13T14:45:18.029 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:17 smithi154 ceph-mon[107038]: pgmap v381: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 6632/267891 objects degraded (2.476%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:45:18.029 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:17 smithi154 ceph-mon[107038]: osdmap e136: 8 total, 8 up, 7 in 2024-09-13T14:45:18.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:18.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:18.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:18.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:18.224 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:17 smithi172 ceph-mon[100696]: pgmap v381: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.5 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 6632/267891 objects degraded (2.476%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:45:18.224 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:17 smithi172 ceph-mon[100696]: osdmap e136: 8 total, 8 up, 7 in 2024-09-13T14:45:18.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:18.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:18.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:18.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:18.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:18.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:18.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:18.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:18.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:18.878 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:18.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:18.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:19.003 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:45:19.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:19.091 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4286084288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:19.091 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:19 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:19.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.164 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:19.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:19.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4286084288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:19.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:19 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4286084288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:19 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:19.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:19.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:19.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:19.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:19.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:19.569 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:19.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:19.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34119.0:9 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:37:12.970149+0000", 2024-09-13T14:45:19.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 486.74238636600001, 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033165799999999997, 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34119", 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3152436209", 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:19.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970149+0000", 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970149+0000", 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970146+0000", 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999971 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970184+0000", 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8099000000000002e-05 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970220+0000", 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5389000000000003e-05 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970237+0000", 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6986e-05 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970366+0000", 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129281 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:37:12.970481+0000", 2024-09-13T14:45:19.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114953 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34119.0:11 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:38:12.973762+0000", 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 426.738773885, 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026809499999999999, 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:45:19.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34119", 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3152436209", 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 11 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.973762+0000", 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.973762+0000", 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.973760+0000", 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.973769+0000", 2024-09-13T14:45:19.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7770000000000008e-06 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.973785+0000", 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6501999999999999e-05 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.973794+0000", 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5750000000000001e-06 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.973888+0000", 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3938999999999996e-05 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:38:12.974030+0000", 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142108 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.723 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34119.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:39:12.982037+0000", 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 366.73049916899998, 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027630599999999998, 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34119", 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3152436209", 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 13 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:19.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982037+0000", 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982037+0000", 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982034+0000", 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:19.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982044+0000", 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2099999999999999e-06 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982061+0000", 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7011e-05 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:19.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982075+0000", 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4163e-05 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982168+0000", 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3061999999999998e-05 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:39:12.982313+0000", 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014494 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:19.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34119.0:16 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:40:12.994913+0000", 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 306.717622694, 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035662600000000002, 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34119", 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3152436209", 2024-09-13T14:45:19.728 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 16 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.994913+0000", 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.994913+0000", 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.994910+0000", 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999967 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.994929+0000", 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9485999999999999e-05 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.994968+0000", 2024-09-13T14:45:19.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9059e-05 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.994994+0000", 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6251999999999999e-05 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.995134+0000", 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139314 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:40:12.995270+0000", 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013601499999999999 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34119.0:18 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:41:13.002930+0000", 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 246.70960557699999, 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000334803, 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:19.732 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34119", 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3152436209", 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.002930+0000", 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.002930+0000", 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.002927+0000", 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999962 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.002947+0000", 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0475999999999998e-05 2024-09-13T14:45:19.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.003003+0000", 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5492000000000003e-05 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.003012+0000", 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6639999999999993e-06 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:19.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.003128+0000", 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011615100000000001 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:41:13.003265+0000", 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136606 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34119.0:19 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e105)", 2024-09-13T14:45:19.736 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:42:13.009216+0000", 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 186.70332015599999, 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028271500000000001, 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34119", 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3152436209", 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 19 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009216+0000", 2024-09-13T14:45:19.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009216+0000", 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009213+0000", 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999971 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009228+0000", 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.552e-05 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009261+0000", 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2765000000000002e-05 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009286+0000", 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5230000000000001e-05 2024-09-13T14:45:19.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009403+0000", 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011737499999999999 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:42:13.009498+0000", 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4797999999999996e-05 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.740 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:19.741 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:19.741 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:45:19.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:19.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:19.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:19.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:19.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:20.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:20.080 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[102314]: pgmap v383: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 4432/258912 objects degraded (1.712%); 1571/258912 objects misplaced (0.607%); 8.1 MiB/s, 164 objects/s recovering 2024-09-13T14:45:20.081 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:20.081 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:20.081 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[102314]: osdmap e137: 8 total, 8 up, 7 in 2024-09-13T14:45:20.081 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4432/258912 objects degraded (1.712%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:20.081 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[107038]: pgmap v383: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 4432/258912 objects degraded (1.712%); 1571/258912 objects misplaced (0.607%); 8.1 MiB/s, 164 objects/s recovering 2024-09-13T14:45:20.081 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:20.081 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:20.082 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[107038]: osdmap e137: 8 total, 8 up, 7 in 2024-09-13T14:45:20.082 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4432/258912 objects degraded (1.712%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:20.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.132 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:20.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:20.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:20.263 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:20.314 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:20 smithi172 ceph-mon[100696]: pgmap v383: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 4432/258912 objects degraded (1.712%); 1571/258912 objects misplaced (0.607%); 8.1 MiB/s, 164 objects/s recovering 2024-09-13T14:45:20.314 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:20 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:20.314 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:20.314 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:20 smithi172 ceph-mon[100696]: osdmap e137: 8 total, 8 up, 7 in 2024-09-13T14:45:20.314 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4432/258912 objects degraded (1.712%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:20.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:20.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:20.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:45:20.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:20.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:20.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.699 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:20.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:20.763 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:20.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:20.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:20.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:20.962 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:20.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:21.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:21.064 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.094 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:21.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:21.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:21.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:21.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.377 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:21.507 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:21.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:21.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:21.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:21.764 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:21.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:21.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:21.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:21.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:45:22.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:22.036 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:22.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:22.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:22 smithi154 ceph-mon[102314]: pgmap v385: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 705 op/s; 4432/256905 objects degraded (1.725%); 1571/256905 objects misplaced (0.612%); 3.4 MiB/s, 100 objects/s recovering 2024-09-13T14:45:22.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:22.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:45:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:45:21] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:45:22.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:22 smithi154 ceph-mon[107038]: pgmap v385: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 705 op/s; 4432/256905 objects degraded (1.725%); 1571/256905 objects misplaced (0.612%); 3.4 MiB/s, 100 objects/s recovering 2024-09-13T14:45:22.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:22.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:22.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:22.319 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:22 smithi172 ceph-mon[100696]: pgmap v385: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 705 op/s; 4432/256905 objects degraded (1.725%); 1571/256905 objects misplaced (0.612%); 3.4 MiB/s, 100 objects/s recovering 2024-09-13T14:45:22.319 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:45:22.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:22.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:22.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T14:45:22.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:22.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:22.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:22.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:22.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:22.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:22.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:22.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:23.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:23.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:23.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:23.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.332 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:23.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:23.387 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:45:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:45:23.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(5.9 137/130 [PushOp(5:9c145819:::benchmark_data_smithi154_144140_object91027:head, version: 126'5590, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:9c145819:::benchmark_data_smithi154_144140_object91027:head@126'5590, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-13T14:45:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:23.383893+0000", 2024-09-13T14:45:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0018332520000000001, 2024-09-13T14:45:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001879405, 2024-09-13T14:45:23.388 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:23.389 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T14:45:23.389 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:23.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:23.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.383893+0000", 2024-09-13T14:45:23.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:23.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:23.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:23.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.383893+0000", 2024-09-13T14:45:23.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:23.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:23.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:23.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.383891+0000", 2024-09-13T14:45:23.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:23.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:23.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:23.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.383929+0000", 2024-09-13T14:45:23.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7595999999999997e-05 2024-09-13T14:45:23.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:23.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.383956+0000", 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6679999999999999e-05 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.383972+0000", 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6614000000000001e-05 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.384086+0000", 2024-09-13T14:45:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113403 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:23.384102+0000", 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5869999999999999e-05 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-13T14:45:23.395 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:45:23.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:23.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:23.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.606 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:23.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:23.705 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:23.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:23.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:23.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:23.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:23.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:24.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:24.069 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:24.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:45:24.157 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:24 smithi154 ceph-mon[102314]: pgmap v386: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 4432/254454 objects degraded (1.742%); 1571/254454 objects misplaced (0.617%); 3.0 MiB/s, 88 objects/s recovering 2024-09-13T14:45:24.157 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:24 smithi154 ceph-mon[107038]: pgmap v386: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 4432/254454 objects degraded (1.742%); 1571/254454 objects misplaced (0.617%); 3.0 MiB/s, 88 objects/s recovering 2024-09-13T14:45:24.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:24.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:24.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:24.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:24.378 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:24 smithi172 ceph-mon[100696]: pgmap v386: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 4432/254454 objects degraded (1.742%); 1571/254454 objects misplaced (0.617%); 3.0 MiB/s, 88 objects/s recovering 2024-09-13T14:45:24.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.480 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:24.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:24.514 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:24.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:24.591 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:45:24.591 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:24.608 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:24.644 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:45:24.644 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:24.684 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:45:24.684 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:24.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:24.717 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:45:24.717 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:24.738 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:24.751 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:45:24.751 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:24.785 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:45:24.786 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:24.814 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:45:24.814 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:24.835 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:45:24.835 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:24.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:24.874 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:24.896 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:24.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:24.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:25.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:25.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:25.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:25.282 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:25 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3736984755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:25.282 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4432/254454 objects degraded (1.742%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:25.289 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:25 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3736984755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:25.289 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4432/254454 objects degraded (1.742%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:25.289 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:25 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3736984755' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:25.289 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4432/254454 objects degraded (1.742%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:25.306 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:25.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:25.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:25.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:25.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.728 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:25.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:25.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:25.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:25.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:25.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:25.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:26.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:26.118 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:26.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:26.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:26.345 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:26 smithi172 ceph-mon[100696]: pgmap v387: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 4432/245619 objects degraded (1.804%); 1176/245619 objects misplaced (0.479%); 5.7 MiB/s, 125 objects/s recovering 2024-09-13T14:45:26.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:26.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:26.457 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:26 smithi154 ceph-mon[102314]: pgmap v387: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 4432/245619 objects degraded (1.804%); 1176/245619 objects misplaced (0.479%); 5.7 MiB/s, 125 objects/s recovering 2024-09-13T14:45:26.458 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:26 smithi154 ceph-mon[107038]: pgmap v387: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 4432/245619 objects degraded (1.804%); 1176/245619 objects misplaced (0.479%); 5.7 MiB/s, 125 objects/s recovering 2024-09-13T14:45:26.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:26.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:26.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.678 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:26.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:26.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:26.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:26.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:26.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:26.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:27.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:27.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:27.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:27.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:27.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:27.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:27.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:27.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:27.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:27.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:27.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:27.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:27.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:27.878 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:27.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:27.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:28.009 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:28.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.144 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:28.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:28.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:28.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:28.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:28.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:28 smithi154 ceph-mon[102314]: pgmap v388: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 805 op/s; 4432/244095 objects degraded (1.816%); 1176/244095 objects misplaced (0.482%); 4.6 MiB/s, 101 objects/s recovering 2024-09-13T14:45:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:28 smithi154 ceph-mon[107038]: pgmap v388: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 805 op/s; 4432/244095 objects degraded (1.816%); 1176/244095 objects misplaced (0.482%); 4.6 MiB/s, 101 objects/s recovering 2024-09-13T14:45:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:28 smithi172 ceph-mon[100696]: pgmap v388: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 805 op/s; 4432/244095 objects degraded (1.816%); 1176/244095 objects misplaced (0.482%); 4.6 MiB/s, 101 objects/s recovering 2024-09-13T14:45:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:28.500 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:45:28.500 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:45:28.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:28.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:28.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:28.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:28.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:28.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:28.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:28.944 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:28.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:29.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:29.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:29.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:29.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:29.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:29.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:29.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:29.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:29.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:29.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:29.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:29.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:29.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:29.893 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:29.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:29.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:30.027 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:30.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:30.157 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:30.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:30.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:30.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:30.430 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:30 smithi154 ceph-mon[102314]: pgmap v389: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 703 op/s; 4432/237801 objects degraded (1.864%); 1176/237801 objects misplaced (0.495%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4432/237801 objects degraded (1.864%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:30.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:30 smithi154 ceph-mon[107038]: pgmap v389: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 703 op/s; 4432/237801 objects degraded (1.864%); 1176/237801 objects misplaced (0.495%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:30.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4432/237801 objects degraded (1.864%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:30.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:30 smithi172 ceph-mon[100696]: pgmap v389: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 703 op/s; 4432/237801 objects degraded (1.864%); 1176/237801 objects misplaced (0.495%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:30.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4432/237801 objects degraded (1.864%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:30.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:30.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:30.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:30.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:30.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:30.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:30.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:30.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:30.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:31.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:31.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:31.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:31.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:31.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:31.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:31.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:31.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:31.617 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:31.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:31.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:31.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:31.877 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:31.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:31.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:32.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:32.072 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:45:32.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:32.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:32 smithi154 ceph-mon[102314]: pgmap v390: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 767 op/s; 4432/231654 objects degraded (1.913%); 708/231654 objects misplaced (0.306%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:45:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/457672615' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:45:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:45:31] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:45:32.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:32 smithi154 ceph-mon[107038]: pgmap v390: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 767 op/s; 4432/231654 objects degraded (1.913%); 708/231654 objects misplaced (0.306%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:45:32.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/457672615' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:32.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:32.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:32 smithi172 ceph-mon[100696]: pgmap v390: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 767 op/s; 4432/231654 objects degraded (1.913%); 708/231654 objects misplaced (0.306%); 4.6 MiB/s, 72 objects/s recovering 2024-09-13T14:45:32.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/457672615' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:45:32.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:32.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:32.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:32.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:32.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:32.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:32.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:33.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:33.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:33.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:33.178 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:33.280 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:33.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:33.389 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:33 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:33.389 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/457672615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:33.389 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:33 smithi154 ceph-mon[102314]: osdmap e138: 8 total, 8 up, 7 in 2024-09-13T14:45:33.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:33 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:33.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/457672615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:33.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:33 smithi154 ceph-mon[107038]: osdmap e138: 8 total, 8 up, 7 in 2024-09-13T14:45:33.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:33.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:33.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:33 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:45:33.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/457672615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:45:33.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:33 smithi172 ceph-mon[100696]: osdmap e138: 8 total, 8 up, 7 in 2024-09-13T14:45:33.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:33.577 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:33.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:33.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:33.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:33.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:33.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:33.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:33.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:34.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:34.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:34.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:34.251 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:34.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:34.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:34.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:34 smithi154 ceph-mon[102314]: pgmap v392: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 4432/229482 objects degraded (1.931%); 708/229482 objects misplaced (0.309%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:45:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:34 smithi154 ceph-mon[107038]: pgmap v392: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 4432/229482 objects degraded (1.931%); 708/229482 objects misplaced (0.309%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:45:34.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:34 smithi172 ceph-mon[100696]: pgmap v392: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 4432/229482 objects degraded (1.931%); 708/229482 objects misplaced (0.309%); 5.4 MiB/s, 86 objects/s recovering 2024-09-13T14:45:34.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:34.543 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:34.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:34.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:34.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:34.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:34.816 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:34.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:34.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:45:34.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:34.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:35.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-13T14:45:35.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:35.132 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:35.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:35.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4432/229482 objects degraded (1.931%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:35.296 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4432/229482 objects degraded (1.931%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:35.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4432/229482 objects degraded (1.931%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:35.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:35.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:35.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:35.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.504 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:35.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:35.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:35.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:35.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:35.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:35.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:35.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:35.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:35.956 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:35.990 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:36.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:36.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:36.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.170 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:36.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:36.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:45:36.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:36.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:36.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:36.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:36.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:36.581 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:36.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:36.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:36.827 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:36 smithi154 ceph-mon[102314]: pgmap v393: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 4432/221544 objects degraded (2.001%); 137/221544 objects misplaced (0.062%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:45:36.827 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:36 smithi154 ceph-mon[107038]: pgmap v393: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 4432/221544 objects degraded (2.001%); 137/221544 objects misplaced (0.062%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:45:36.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:36.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:36.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:36.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:36.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:36 smithi172 ceph-mon[100696]: pgmap v393: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 802 op/s; 4432/221544 objects degraded (2.001%); 137/221544 objects misplaced (0.062%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:45:37.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:37.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:37.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:37.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:37.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:37.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:37.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:37.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:37.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:37.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:37.633 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:37.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:37.754 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:37 smithi172 ceph-mon[100696]: osdmap e139: 8 total, 8 up, 7 in 2024-09-13T14:45:37.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:37.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:37.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:37.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:37.908 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:37 smithi154 ceph-mon[102314]: osdmap e139: 8 total, 8 up, 7 in 2024-09-13T14:45:37.908 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:37 smithi154 ceph-mon[107038]: osdmap e139: 8 total, 8 up, 7 in 2024-09-13T14:45:38.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:38.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:38.060 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:38.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.165 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:38.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:38.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:38.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:38.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:38.474 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:38.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:38.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:38.641 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:45:38.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:38.677 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:45:38.677 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:38.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:38.712 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:45:38.713 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.745 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:38.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:38.753 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:45:38.753 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:38.787 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:45:38.787 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:38.821 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:45:38.821 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:38.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:38.857 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:45:38.857 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:15159 5.2 5:462b06ea:::benchmark_data_smithi154_144140_object15158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:22.320707+0000", 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 136.539556275, 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.046455214000000002, 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.863 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 15159 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.320707+0000", 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.320707+0000", 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.320706+0000", 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.320821+0000", 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115596 2024-09-13T14:45:38.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.320835+0000", 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3868e-05 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.320842+0000", 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0639999999999996e-06 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.320905+0000", 2024-09-13T14:45:38.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3115999999999997e-05 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321036+0000", 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013039799999999999 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321182+0000", 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146778 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.322516+0000", 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013337329999999999 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.324219+0000", 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001703277 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.367061+0000", 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042841733999999999 2024-09-13T14:45:38.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.367116+0000", 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4882000000000001e-05 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.367162+0000", 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6023999999999998e-05 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.869 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:15162 5.2 5:43ac85c1:::benchmark_data_smithi154_144140_object15161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:22.321221+0000", 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 136.539042034, 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.04607377, 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 15162 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.870 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321221+0000", 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321221+0000", 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321220+0000", 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321280+0000", 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0197e-05 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321292+0000", 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1287999999999999e-05 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321299+0000", 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8950000000000001e-06 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321803+0000", 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050488200000000001 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.321937+0000", 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013314199999999999 2024-09-13T14:45:38.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.322066+0000", 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012973300000000001 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.322603+0000", 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053710699999999999 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.324255+0000", 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016519460000000001 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.367215+0000", 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042959259999999999 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.367262+0000", 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7750999999999997e-05 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:22.367295+0000", 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2496999999999997e-05 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:24657 5.2 5:4ef2e9a9:::benchmark_data_smithi154_144140_object24656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:26.369059+0000", 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 132.49120463700001, 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.056753895999999998, 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.876 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 24657 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369059+0000", 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369059+0000", 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369056+0000", 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369115+0000", 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8186000000000002e-05 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369141+0000", 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6716e-05 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369156+0000", 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4686e-05 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369236+0000", 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9927999999999998e-05 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369401+0000", 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016487000000000001 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.369493+0000", 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2029999999999998e-05 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.370285+0000", 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079180299999999999 2024-09-13T14:45:38.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.424504+0000", 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.054219006 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.425687+0000", 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011837740000000001 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.425759+0000", 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1563000000000006e-05 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:26.425812+0000", 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3378000000000003e-05 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:26004 5.2 5:41edf2d1:::benchmark_data_smithi154_144140_object26003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.883 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.029092+0000", 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.83117101100001, 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.086725410000000003, 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26004 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029092+0000", 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029092+0000", 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029091+0000", 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029145+0000", 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4287999999999999e-05 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029160+0000", 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4435e-05 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029171+0000", 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1314e-05 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029222+0000", 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1351999999999999e-05 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029355+0000", 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013317 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.029505+0000", 2024-09-13T14:45:38.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014950900000000001 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.030437+0000", 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093172100000000005 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036181+0000", 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057447560000000002 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115763+0000", 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079581726000000005 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115795+0000", 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1918000000000002e-05 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115818+0000", 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2509000000000001e-05 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.889 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:26011 5.2 5:4a25616d:::benchmark_data_smithi154_144140_object26010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.031355+0000", 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.82890845, 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084553675999999994, 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.890 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26011 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031355+0000", 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031355+0000", 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031354+0000", 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031446+0000", 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2757000000000007e-05 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031459+0000", 2024-09-13T14:45:38.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2206e-05 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031466+0000", 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5939999999999996e-06 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031510+0000", 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3374000000000002e-05 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031629+0000", 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119319 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.031779+0000", 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149956 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.032694+0000", 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091564499999999998 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036208+0000", 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003513057 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115855+0000", 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079647127999999998 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115886+0000", 2024-09-13T14:45:38.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1241000000000001e-05 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115908+0000", 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2481e-05 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:26013 5.2 5:42a23d5b:::benchmark_data_smithi154_144140_object26012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.033393+0000", 2024-09-13T14:45:38.896 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.826869794, 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082592766999999997, 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26013 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033393+0000", 2024-09-13T14:45:38.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033393+0000", 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033392+0000", 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033442+0000", 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9990000000000001e-05 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033453+0000", 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0902e-05 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033459+0000", 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.215e-06 2024-09-13T14:45:38.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033487+0000", 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8768e-05 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033576+0000", 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8790999999999997e-05 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.033659+0000", 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3158999999999994e-05 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.034257+0000", 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00059773499999999995 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036246+0000", 2024-09-13T14:45:38.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019887310000000001 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115934+0000", 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079688456000000005 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115960+0000", 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6177999999999999e-05 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.115986+0000", 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5720000000000001e-05 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:26015 5.2 5:409e6d64:::benchmark_data_smithi154_144140_object26014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.036444+0000", 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.823819136, 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082970211000000002, 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26015 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036444+0000", 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036444+0000", 2024-09-13T14:45:38.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036443+0000", 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036500+0000", 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6657999999999999e-05 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036509+0000", 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9840000000000007e-06 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036513+0000", 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3889999999999998e-06 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036538+0000", 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5004999999999999e-05 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036621+0000", 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3362999999999995e-05 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036710+0000", 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8953e-05 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.038190+0000", 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014795170000000001 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.038772+0000", 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00058210200000000003 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.119344+0000", 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080571606000000004 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.119378+0000", 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4888000000000002e-05 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.119414+0000", 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5788999999999999e-05 2024-09-13T14:45:38.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:26017 5.2 5:45922fee:::benchmark_data_smithi154_144140_object26016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.036573+0000", 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.82368969199999, 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084782462000000003, 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.910 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 26017 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036573+0000", 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036573+0000", 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036573+0000", 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036668+0000", 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5534999999999999e-05 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036680+0000", 2024-09-13T14:45:38.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1674999999999999e-05 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.036685+0000", 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9250000000000003e-06 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.037280+0000", 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00059454699999999996 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.037418+0000", 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013819599999999999 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.037555+0000", 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013716499999999999 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.038300+0000", 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00074508500000000004 2024-09-13T14:45:38.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.038847+0000", 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00054652100000000001 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.121274+0000", 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082427882999999993 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.121324+0000", 2024-09-13T14:45:38.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9372000000000001e-05 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.121356+0000", 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2121e-05 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:27229 5.2 5:47a3bf38:::benchmark_data_smithi154_144140_object27228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.669459+0000", 2024-09-13T14:45:38.916 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.19080437599999, 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.16906344300000001, 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 27229 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669459+0000", 2024-09-13T14:45:38.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669459+0000", 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669457+0000", 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669491+0000", 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3810000000000003e-05 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669500+0000", 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9339999999999995e-06 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669506+0000", 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0839999999999997e-06 2024-09-13T14:45:38.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669547+0000", 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0438999999999997e-05 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669666+0000", 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011913600000000001 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669880+0000", 2024-09-13T14:45:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000214195 2024-09-13T14:45:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.671121+0000", 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001240653 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.672401+0000", 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012802569999999999 2024-09-13T14:45:38.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838431+0000", 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.16602999700000001 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838482+0000", 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0695999999999997e-05 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838522+0000", 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0590999999999998e-05 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:27239 5.2 5:4246bfbf:::benchmark_data_smithi154_144140_object27238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.674859+0000", 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.185404613, 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.163782447, 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.925 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 27239 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674859+0000", 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674859+0000", 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674857+0000", 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674958+0000", 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101588 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674974+0000", 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5661000000000001e-05 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674982+0000", 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8970000000000008e-06 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.675031+0000", 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9191999999999999e-05 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.675172+0000", 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140345 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.675332+0000", 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160089 2024-09-13T14:45:38.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676497+0000", 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011657250000000001 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.677801+0000", 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013040370000000001 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838574+0000", 2024-09-13T14:45:38.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.160772633 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838607+0000", 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2879000000000001e-05 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838641+0000", 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4063000000000002e-05 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:27242 5.2 5:482bba7a:::benchmark_data_smithi154_144140_object27241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.676639+0000", 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.183624425, 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.162555276, 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 27242 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676639+0000", 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676639+0000", 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676638+0000", 2024-09-13T14:45:38.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676723+0000", 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4969999999999995e-05 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676734+0000", 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1841000000000001e-05 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676742+0000", 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676790+0000", 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8687999999999999e-05 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676933+0000", 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000143008 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.677076+0000", 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014246899999999999 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.677906+0000", 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083029999999999996 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.678301+0000", 2024-09-13T14:45:38.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000394607 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.839143+0000", 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.160842136 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.839173+0000", 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0505999999999999e-05 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.839194+0000", 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0752999999999999e-05 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:28332 5.2 5:46437a7f:::benchmark_data_smithi154_144140_object28331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:28.296646+0000", 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.56361704299999, 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.051916703000000002, 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.938 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28332 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296646+0000", 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296646+0000", 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296645+0000", 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296699+0000", 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3922000000000003e-05 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296710+0000", 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1097000000000001e-05 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296717+0000", 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3710000000000002e-06 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296762+0000", 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5278000000000003e-05 2024-09-13T14:45:38.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.296879+0000", 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116068 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.297014+0000", 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013538100000000001 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.298088+0000", 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001074204 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.299418+0000", 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013296950000000001 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.348485+0000", 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049066785000000002 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.348529+0000", 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4679999999999999e-05 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.348563+0000", 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3504999999999999e-05 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.944 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:28633 5.2 5:4a721b5f:::benchmark_data_smithi154_144140_object28632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:28.605016+0000", 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.255246825, 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.074521778999999996, 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 28633 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605016+0000", 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605016+0000", 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605015+0000", 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605059+0000", 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4379000000000003e-05 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605072+0000", 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2128e-05 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605078+0000", 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9689999999999999e-06 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605121+0000", 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3392000000000001e-05 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605206+0000", 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5115000000000001e-05 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.605291+0000", 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5222999999999994e-05 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.606165+0000", 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087399699999999995 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.667634+0000", 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061469039000000003 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.679447+0000", 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011812193 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.679498+0000", 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1913999999999998e-05 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:28.679538+0000", 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9629999999999998e-05 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:31415 5.2 5:4f8fa0e5:::benchmark_data_smithi154_144140_object31414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:29.988205+0000", 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.87205776799999, 2024-09-13T14:45:38.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064867896999999994, 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31415 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988205+0000", 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988205+0000", 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988204+0000", 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988245+0000", 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1393000000000002e-05 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988256+0000", 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0259e-05 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988262+0000", 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9599999999999997e-06 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988308+0000", 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6261999999999999e-05 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988421+0000", 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011326400000000001 2024-09-13T14:45:38.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.988578+0000", 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015688900000000001 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.989849+0000", 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001271187 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.046252+0000", 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.056402423 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.052980+0000", 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067282779999999999 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.053036+0000", 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5847000000000002e-05 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.053073+0000", 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7456000000000002e-05 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:31424 5.2 5:41fe7bee:::benchmark_data_smithi154_144140_object31423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:29.990537+0000", 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 128.869726638, 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062653464000000006, 2024-09-13T14:45:38.959 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31424 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990537+0000", 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990537+0000", 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990535+0000", 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990587+0000", 2024-09-13T14:45:38.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1836999999999999e-05 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990600+0000", 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3189e-05 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990606+0000", 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6590000000000004e-06 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990650+0000", 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4292999999999998e-05 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990736+0000", 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6234000000000001e-05 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.990880+0000", 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014341700000000001 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:29.991885+0000", 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001004916 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.048722+0000", 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.056837367999999999 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.053112+0000", 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043898720000000004 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.053156+0000", 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3718999999999999e-05 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:30.053190+0000", 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4369999999999998e-05 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:33386 5.2 5:45c88780:::benchmark_data_smithi154_144140_object33385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:31.021828+0000", 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.83843495799999, 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.108195734, 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 33386 2024-09-13T14:45:38.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021828+0000", 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021828+0000", 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021827+0000", 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021872+0000", 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5302000000000001e-05 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021888+0000", 2024-09-13T14:45:38.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5984000000000001e-05 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021897+0000", 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5499999999999995e-06 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021935+0000", 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8494000000000001e-05 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.022059+0000", 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012427299999999999 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.022200+0000", 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014108999999999999 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.023483+0000", 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001283001 2024-09-13T14:45:38.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.027180+0000", 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036963 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.129924+0000", 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102744157 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.129979+0000", 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5599000000000001e-05 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.130024+0000", 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4467e-05 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:33402 5.2 5:402a2cd5:::benchmark_data_smithi154_144140_object33401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:31.029058+0000", 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.83120486200001, 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.101244691, 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 33402 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.973 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029058+0000", 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029058+0000", 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029057+0000", 2024-09-13T14:45:38.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029127+0000", 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0009e-05 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029143+0000", 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6594000000000001e-05 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029152+0000", 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7260000000000008e-06 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029222+0000", 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9418000000000005e-05 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029348+0000", 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012644699999999999 2024-09-13T14:45:38.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.029453+0000", 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104954 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.030489+0000", 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010357280000000001 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.030984+0000", 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049547100000000004 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.130219+0000", 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.099235136000000002 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.130266+0000", 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7136999999999999e-05 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.130303+0000", 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6566999999999997e-05 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:65177 5.2 5:42abcda4:::benchmark_data_smithi154_144140_object65176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:51.109599+0000", 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 107.750664105, 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066504613000000004, 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.979 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 65177 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109599+0000", 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109599+0000", 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109598+0000", 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109638+0000", 2024-09-13T14:45:38.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0856999999999999e-05 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109656+0000", 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7691999999999999e-05 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109664+0000", 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3299999999999999e-06 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109723+0000", 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8736999999999997e-05 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109858+0000", 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013460499999999999 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.109990+0000", 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013238200000000001 2024-09-13T14:45:38.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.112325+0000", 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023349109999999998 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.113526+0000", 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012010020000000001 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.176023+0000", 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.062496755000000001 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.176067+0000", 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4004000000000001e-05 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:51.176104+0000", 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6847999999999997e-05 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:69913 5.2 5:402e32da:::benchmark_data_smithi154_144140_object69912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:55.097423+0000", 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 103.762839919, 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084498371000000003, 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 69913 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.986 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097423+0000", 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097423+0000", 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097421+0000", 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097468+0000", 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6748e-05 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097484+0000", 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6056999999999999e-05 2024-09-13T14:45:38.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097490+0000", 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0730000000000001e-06 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097840+0000", 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034929299999999998 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.097972+0000", 2024-09-13T14:45:38.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132683 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.098123+0000", 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150989 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.100464+0000", 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406659999999999 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.107556+0000", 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070918939999999996 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.181830+0000", 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.074274218000000003 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.181877+0000", 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6739000000000001e-05 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.181922+0000", 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4742e-05 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:69925 5.2 5:4b57db6a:::benchmark_data_smithi154_144140_object69924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:55.104825+0000", 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 103.75543820999999, 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083834521999999995, 2024-09-13T14:45:38.992 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 69925 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.104825+0000", 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.104825+0000", 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.104823+0000", 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.104866+0000", 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3343999999999997e-05 2024-09-13T14:45:38.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.104884+0000", 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.789e-05 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.104892+0000", 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9270000000000005e-06 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:38.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.104939+0000", 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6211000000000002e-05 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.105062+0000", 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123251 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.105198+0000", 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013635899999999999 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.107348+0000", 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002150211 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.109624+0000", 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022754089999999999 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:38.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.188588+0000", 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.078964088000000002 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.188624+0000", 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5676999999999997e-05 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:55.188659+0000", 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5939999999999998e-05 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.998 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:38.999 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:38.999 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:45:38.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:39.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:39.000 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:45:39.000 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:39.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:38 smithi154 ceph-mon[102314]: pgmap v395: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 4432/220134 objects degraded (2.013%); 137/220134 objects misplaced (0.062%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:45:39.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:38 smithi154 ceph-mon[102314]: osdmap e140: 8 total, 8 up, 7 in 2024-09-13T14:45:39.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/770306103' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:39.001 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:38 smithi154 ceph-mon[107038]: pgmap v395: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 4432/220134 objects degraded (2.013%); 137/220134 objects misplaced (0.062%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:45:39.001 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:38 smithi154 ceph-mon[107038]: osdmap e140: 8 total, 8 up, 7 in 2024-09-13T14:45:39.001 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/770306103' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:39.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:38 smithi172 ceph-mon[100696]: pgmap v395: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 4432/220134 objects degraded (2.013%); 137/220134 objects misplaced (0.062%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:45:39.001 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:38 smithi172 ceph-mon[100696]: osdmap e140: 8 total, 8 up, 7 in 2024-09-13T14:45:39.002 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/770306103' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:39.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:39.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:39.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:39.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:39.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:39.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:39.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:39.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:39.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:39.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:39.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:39.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:39.909 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:39 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4432/220134 objects degraded (2.013%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:39.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:39.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:39.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:40.009 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:39 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4432/220134 objects degraded (2.013%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:40.009 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:39 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4432/220134 objects degraded (2.013%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:40.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:40.134 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:40.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:40.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:40.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:40.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.428 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:40.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:40.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:40.567 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:40.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:45:40.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:40.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:40.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:40.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:40.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:40.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:40.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:40.951 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:40.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:40.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:40 smithi172 ceph-mon[100696]: pgmap v397: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 2274/214839 objects degraded (1.058%); 1929/214839 objects misplaced (0.898%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:45:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:40 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:40 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:41.065 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:40 smithi154 ceph-mon[102314]: pgmap v397: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 2274/214839 objects degraded (1.058%); 1929/214839 objects misplaced (0.898%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:45:41.065 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:41.065 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:41.065 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:40 smithi154 ceph-mon[107038]: pgmap v397: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 826 op/s; 2274/214839 objects degraded (1.058%); 1929/214839 objects misplaced (0.898%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:45:41.065 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:41.065 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:41.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:41.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:41.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:41.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:41.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:41.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:41.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:41.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:41.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:41.659 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:41.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:41.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:45:41.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:41.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:41.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:41.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:42.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.106 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:42.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:42.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:42.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:45:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:45:41] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:45:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.237 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:42.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:42.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:42.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:42.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:42.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:42.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:42.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-13T14:45:42.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.34620.0:153932 5.b e140/111 5:d8bbf4bc:::benchmark_data_smithi154_144140_object43579:head v 140'9833, mlcod=140'9828)", 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:42.701905+0000", 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.00355463, 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035751200000000002, 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701905+0000", 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701905+0000", 2024-09-13T14:45:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701904+0000", 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701911+0000", 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0489999999999998e-06 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701923+0000", 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2357999999999999e-05 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701930+0000", 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6069999999999996e-06 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701975+0000", 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4903e-05 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.701995+0000", 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0438999999999999e-05 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.34620.0:153933 5.6 e140/111 5:60469e58:::benchmark_data_smithi154_144140_object43580:head v 140'9530, mlcod=140'9528)", 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:42.704520+0000", 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.00093954599999999998, 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010033589999999999, 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704520+0000", 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704520+0000", 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704519+0000", 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704526+0000", 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2759999999999997e-06 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704540+0000", 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4669e-05 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704547+0000", 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0840000000000003e-06 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704590+0000", 2024-09-13T14:45:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2858000000000002e-05 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704609+0000", 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8787000000000002e-05 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.34620.0:153939 5.1 e140/110 5:8ff9589d:::benchmark_data_smithi154_144140_object43586:head v 140'9464, mlcod=140'9462)", 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:42.704817+0000", 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.00064270400000000002, 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00073757399999999996, 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704817+0000", 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704817+0000", 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704816+0000", 2024-09-13T14:45:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704822+0000", 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9920000000000004e-06 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704833+0000", 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1231e-05 2024-09-13T14:45:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704839+0000", 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8699999999999997e-06 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704875+0000", 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5853999999999999e-05 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:42.704897+0000", 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1617999999999999e-05 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 3 2024-09-13T14:45:42.723 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:45:42.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:42.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:42.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:42.903 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:42.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:42 smithi154 ceph-mon[102314]: pgmap v398: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 2274/209682 objects degraded (1.084%); 1792/209682 objects misplaced (0.855%); 6.3 MiB/s, 137 objects/s recovering 2024-09-13T14:45:42.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:42 smithi154 ceph-mon[107038]: pgmap v398: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 2274/209682 objects degraded (1.084%); 1792/209682 objects misplaced (0.855%); 6.3 MiB/s, 137 objects/s recovering 2024-09-13T14:45:42.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:42 smithi172 ceph-mon[100696]: pgmap v398: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 2274/209682 objects degraded (1.084%); 1792/209682 objects misplaced (0.855%); 6.3 MiB/s, 137 objects/s recovering 2024-09-13T14:45:43.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:43.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:43.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:43.189 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:43.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:43.265 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:43.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:43.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:43.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:45:43.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:43.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:43.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:43.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:43.701 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:43.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:43.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.752 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:43.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:43.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:43.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:43.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:43.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:44.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:44.088 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:44.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:44.171 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:44.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:44.300 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:44.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:44.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:44.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:44.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:44.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:44.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:44.798 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:44 smithi154 ceph-mon[102314]: pgmap v399: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 577 op/s; 2274/207672 objects degraded (1.095%); 1554/207672 objects misplaced (0.748%); 3.7 MiB/s, 95 objects/s recovering 2024-09-13T14:45:44.798 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:44 smithi154 ceph-mon[107038]: pgmap v399: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 577 op/s; 2274/207672 objects degraded (1.095%); 1554/207672 objects misplaced (0.748%); 3.7 MiB/s, 95 objects/s recovering 2024-09-13T14:45:44.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:44.842 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:44.868 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:44 smithi172 ceph-mon[100696]: pgmap v399: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 577 op/s; 2274/207672 objects degraded (1.095%); 1554/207672 objects misplaced (0.748%); 3.7 MiB/s, 95 objects/s recovering 2024-09-13T14:45:44.868 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2274/207672 objects degraded (1.095%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:44.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:44.985 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:44.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:45.060 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2274/207672 objects degraded (1.095%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:45.061 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2274/207672 objects degraded (1.095%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:45.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:45.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:45.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.246 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:45.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:45.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:45.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:45.474 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:45:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:45:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:45:45.474 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:45:45.474 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:45:45.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:45.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:45.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:45.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:45.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:45.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:45.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:45.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:45.931 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:45.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:46.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:46.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:46.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:46.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:46.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:45:46.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:46.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:46.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:46.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:46.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:46 smithi172 ceph-mon[100696]: pgmap v400: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 842 op/s; 2274/199617 objects degraded (1.139%); 1554/199617 objects misplaced (0.778%); 3.6 MiB/s, 94 objects/s recovering 2024-09-13T14:45:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.509 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:45:46.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:46.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:46.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[102314]: pgmap v400: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 842 op/s; 2274/199617 objects degraded (1.139%); 1554/199617 objects misplaced (0.778%); 3.6 MiB/s, 94 objects/s recovering 2024-09-13T14:45:46.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:46.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:46.653 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[107038]: pgmap v400: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 842 op/s; 2274/199617 objects degraded (1.139%); 1554/199617 objects misplaced (0.778%); 3.6 MiB/s, 94 objects/s recovering 2024-09-13T14:45:46.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:45:46.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:45:46.654 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:46.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:46.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:46.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:46.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:46.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:46.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:47.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:47.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:47.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:47.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:47.301 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:47.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:47.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:47.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:47.449 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:45:47.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:47.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:47.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:47.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:47.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:47 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2402273645' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:47.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:47 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:47.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:47 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2402273645' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:47.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:47 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:47.729 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:47 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2402273645' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:47.729 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:47 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:45:47.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:47.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:47.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:47.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:47.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:48.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:48.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:48.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:48.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:95602 5.4 5:2f361fea:::benchmark_data_smithi154_144140_object95601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:26.435896+0000", 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.857536963000001, 2024-09-13T14:45:48.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081072531000000003, 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 95602 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.435896+0000", 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.435896+0000", 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.435895+0000", 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.435960+0000", 2024-09-13T14:45:48.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5837e-05 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.435972+0000", 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1382e-05 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.435979+0000", 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0049999999999998e-06 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436121+0000", 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014220899999999999 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436224+0000", 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103112 2024-09-13T14:45:48.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436348+0000", 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124144 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.441940+0000", 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055921549999999997 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.445817+0000", 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003876904 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.516870+0000", 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.071052757999999994 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.516923+0000", 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3146999999999998e-05 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.516969+0000", 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5769999999999997e-05 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:95606 5.4 5:246ce32b:::benchmark_data_smithi154_144140_object95605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:26.436438+0000", 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.856995220000002, 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080687274000000003, 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.303 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 95606 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436438+0000", 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436438+0000", 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436437+0000", 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436499+0000", 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1715000000000001e-05 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436510+0000", 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0997e-05 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.436517+0000", 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9659999999999999e-06 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.439340+0000", 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028227450000000002 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.439467+0000", 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127546 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.439596+0000", 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012903 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.442063+0000", 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024670629999999998 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.445845+0000", 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00378152 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.517017+0000", 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.071172492000000004 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.517090+0000", 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2478000000000006e-05 2024-09-13T14:45:48.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:26.517125+0000", 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5771e-05 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:96127 5.4 5:22d5a6b9:::benchmark_data_smithi154_144140_object96126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:27.208409+0000", 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 81.085024415000007, 2024-09-13T14:45:48.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084314993000000005, 2024-09-13T14:45:48.310 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.310 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 96127 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208409+0000", 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208409+0000", 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208407+0000", 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208452+0000", 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4978999999999997e-05 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208468+0000", 2024-09-13T14:45:48.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6073999999999999e-05 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208477+0000", 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0089999999999996e-06 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208539+0000", 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2458000000000004e-05 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208663+0000", 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123344 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.208774+0000", 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111117 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.211465+0000", 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002690797 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.219639+0000", 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081741230000000002 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.292668+0000", 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.073029307000000002 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.292702+0000", 2024-09-13T14:45:48.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3875999999999999e-05 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:27.292724+0000", 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1982000000000001e-05 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:97375 5.9 5:9dd925ab:::benchmark_data_smithi154_144140_object97374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:45:48.316 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.363266096999993, 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.11292851499999999, 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 97375 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:45:48.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930165+0000", 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975030+0000", 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044864996999999997 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975066+0000", 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5460999999999998e-05 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975076+0000", 2024-09-13T14:45:48.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0397999999999999e-05 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.979893+0000", 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048163379999999999 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.980059+0000", 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016674300000000001 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:45:48.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.980241+0000", 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018151899999999999 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.988970+0000", 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087295719999999997 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.989030+0000", 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9859999999999999e-05 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.040919+0000", 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.051888722999999998 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.040974+0000", 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4549999999999998e-05 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.043096+0000", 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002122199 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.322 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:113747 5.9 5:91f8fbcd:::benchmark_data_smithi154_144140_object3394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.052810608999998, 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.091457920999999998, 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113747 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.323 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240622+0000", 2024-09-13T14:45:48.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240625+0000", 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7530000000000002e-06 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240630+0000", 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2979999999999999e-06 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240635+0000", 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3989999999999997e-06 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247577+0000", 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0069425770000000001 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247719+0000", 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014185900000000001 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247864+0000", 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014463900000000001 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.250756+0000", 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028924279999999998 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.251141+0000", 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038421899999999999 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.330437+0000", 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079296273 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.330474+0000", 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7418e-05 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.332081+0000", 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001606447 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:113964 5.9 5:9bdfc14e:::benchmark_data_smithi154_144140_object3611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:52.521039+0000", 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 55.772394302999999, 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084537020000000004, 2024-09-13T14:45:48.329 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113964 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521039+0000", 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521039+0000", 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521038+0000", 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521043+0000", 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4620000000000004e-06 2024-09-13T14:45:48.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521054+0000", 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1032999999999999e-05 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521061+0000", 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3049999999999998e-06 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.523325+0000", 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002264028 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.523450+0000", 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000125437 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.523574+0000", 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012355600000000001 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.529983+0000", 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064089610000000003 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.530082+0000", 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9479000000000001e-05 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.555024+0000", 2024-09-13T14:45:48.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024942114000000001 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.555055+0000", 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0746e-05 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.605576+0000", 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.050521005000000001 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:113965 5.9 5:9051647f:::benchmark_data_smithi154_144140_object3612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:52.521836+0000", 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 55.771597196000002, 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.085882313000000002, 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113965 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.336 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521836+0000", 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521836+0000", 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521835+0000", 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521840+0000", 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3309999999999997e-06 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521849+0000", 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1009999999999998e-06 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.521854+0000", 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6380000000000001e-06 2024-09-13T14:45:48.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.526916+0000", 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00506136 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.527053+0000", 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013733899999999999 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.527178+0000", 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000125282 2024-09-13T14:45:48.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.530099+0000", 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029203279999999998 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.530215+0000", 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011662099999999999 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.605629+0000", 2024-09-13T14:45:48.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.075413732999999997 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.605691+0000", 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2417999999999998e-05 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.607718+0000", 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002027025 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:114992 5.9 5:9bf9d9df:::benchmark_data_smithi154_144140_object4639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 54.308778562000001, 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.85779442800000005, 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114992 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984654+0000", 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984658+0000", 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7450000000000001e-06 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984665+0000", 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5020000000000003e-06 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984675+0000", 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.132e-06 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984715+0000", 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701000000000002e-05 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984723+0000", 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6140000000000003e-06 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824713+0000", 2024-09-13T14:45:48.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.83998967899999999 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824847+0000", 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000134356 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824939+0000", 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1880999999999997e-05 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830175+0000", 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052358680000000003 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.833865+0000", 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003690189 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.840699+0000", 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00683411 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.840754+0000", 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5126e-05 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842449+0000", 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016951170000000001 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:115007 5.9 5:93772757:::benchmark_data_smithi154_144140_object4654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 54.280766180999997, 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.83119907900000001, 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115007 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.349 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012666+0000", 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012675+0000", 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6880000000000005e-06 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012685+0000", 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.696e-06 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012692+0000", 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8549999999999996e-06 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012728+0000", 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6328000000000002e-05 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012736+0000", 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.011e-06 2024-09-13T14:45:48.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:48.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830212+0000", 2024-09-13T14:45:48.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81747611099999995 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830363+0000", 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015098299999999999 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830536+0000", 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173227 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834179+0000", 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003643178 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838231+0000", 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040517900000000004 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842523+0000", 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042920570000000002 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842581+0000", 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7868999999999998e-05 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843866+0000", 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001285071 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:115010 5.9 5:9c0819f0:::benchmark_data_smithi154_144140_object4657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 54.276255597000002, 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.82826413700000001, 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115010 2024-09-13T14:45:48.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017177+0000", 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017181+0000", 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6619999999999998e-06 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017188+0000", 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0960000000000003e-06 2024-09-13T14:45:48.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017192+0000", 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0330000000000002e-06 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017219+0000", 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7807999999999999e-05 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:45:48.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017226+0000", 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4629999999999998e-06 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834282+0000", 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81705591399999999 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834452+0000", 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170314 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834602+0000", 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149732 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838287+0000", 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00368491 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838568+0000", 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028085099999999998 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843931+0000", 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053631590000000002 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843991+0000", 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0174999999999999e-05 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.845442+0000", 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001450914 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.364 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117184 5.4 5:285a1434:::benchmark_data_smithi154_144140_object6831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.123077645000002, 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15315772, 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117184 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.365 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170355+0000", 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170358+0000", 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.19e-06 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170365+0000", 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9419999999999998e-06 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170371+0000", 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7899999999999996e-06 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174205+0000", 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038343880000000002 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174315+0000", 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109225 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174416+0000", 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010148 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.177943+0000", 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035270409999999999 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.183599+0000", 2024-09-13T14:45:48.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056554109999999999 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323412+0000", 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.13981297000000001 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323466+0000", 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4549000000000002e-05 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323513+0000", 2024-09-13T14:45:48.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7312999999999999e-05 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117538 5.4 5:2cd2fa81:::benchmark_data_smithi154_144140_object7185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.654457938999997, 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.140160653, 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117538 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638974+0000", 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638981+0000", 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3900000000000004e-06 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638999+0000", 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7948000000000002e-05 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639007+0000", 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9670000000000001e-06 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639049+0000", 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1593e-05 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639234+0000", 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185253 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639344+0000", 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109447 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648634+0000", 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092907010000000002 2024-09-13T14:45:48.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648647+0000", 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3182e-05 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779063+0000", 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.130415064 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779100+0000", 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7110000000000002e-05 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779136+0000", 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6436999999999997e-05 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117539 5.4 5:27e8a4d2:::benchmark_data_smithi154_144140_object7186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:45:48.377 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.654250740999998, 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.140068675, 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117539 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639182+0000", 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639187+0000", 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.758e-06 2024-09-13T14:45:48.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639198+0000", 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1674999999999999e-05 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639205+0000", 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5409999999999997e-06 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644291+0000", 2024-09-13T14:45:48.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050859499999999997 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644444+0000", 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015357700000000001 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644545+0000", 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010090999999999999 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655177+0000", 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010632157 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.776212+0000", 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.121034903 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779189+0000", 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029772129999999998 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779215+0000", 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5933999999999999e-05 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779251+0000", 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5936000000000003e-05 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117544 5.4 5:239bf146:::benchmark_data_smithi154_144140_object7191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.652548426000003, 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.138439275, 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117544 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.384 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640884+0000", 2024-09-13T14:45:48.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640889+0000", 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7640000000000002e-06 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640902+0000", 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3046e-05 2024-09-13T14:45:48.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640909+0000", 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1840000000000004e-06 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648692+0000", 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077836470000000003 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648798+0000", 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010568700000000001 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648898+0000", 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100207 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655188+0000", 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062894869999999999 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.776255+0000", 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12106747599999999 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779281+0000", 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030257999999999999 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779305+0000", 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4136000000000001e-05 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779324+0000", 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9199999999999999e-05 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117548 5.4 5:28e11507:::benchmark_data_smithi154_144140_object7195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.646238783000001, 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15636461800000001, 2024-09-13T14:45:48.390 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117548 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647193+0000", 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647199+0000", 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4439999999999999e-06 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647209+0000", 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0275000000000001e-05 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647215+0000", 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1040000000000003e-06 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652094+0000", 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048785010000000004 2024-09-13T14:45:48.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652244+0000", 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015033599999999999 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652372+0000", 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012824700000000001 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.658282+0000", 2024-09-13T14:45:48.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059094480000000003 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.783415+0000", 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12513291300000001 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803465+0000", 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020050179000000001 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803514+0000", 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8661e-05 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803559+0000", 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5626e-05 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117551 5.4 5:28c6b1cf:::benchmark_data_smithi154_144140_object7198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.644732349999998, 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15501442800000001, 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117551 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.397 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648700+0000", 2024-09-13T14:45:48.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648704+0000", 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4279999999999998e-06 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648712+0000", 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0320000000000003e-06 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648718+0000", 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5620000000000003e-06 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655217+0000", 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064988140000000003 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655329+0000", 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112169 2024-09-13T14:45:48.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655446+0000", 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116567 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.658387+0000", 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029411540000000001 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.783571+0000", 2024-09-13T14:45:48.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12518472899999999 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803616+0000", 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020044480999999999 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803682+0000", 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6157999999999999e-05 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803715+0000", 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3432000000000001e-05 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117857 5.4 5:20ac363d:::benchmark_data_smithi154_144140_object7504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.192744613999999, 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12765149100000001, 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117857 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:45:48.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100687+0000", 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100696+0000", 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100716+0000", 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9788000000000001e-05 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100724+0000", 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4519999999999997e-06 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100794+0000", 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9370999999999997e-05 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100907+0000", 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113435 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.101008+0000", 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100407 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107808+0000", 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068003430000000004 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107856+0000", 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7775000000000003e-05 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228218+0000", 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12036281 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228302+0000", 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3688000000000002e-05 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228340+0000", 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090999999999998e-05 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117860 5.4 5:29f284c5:::benchmark_data_smithi154_144140_object7507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.189600773000002, 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.124665336, 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.409 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117860 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:45:48.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103831+0000", 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103839+0000", 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8129999999999996e-06 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103855+0000", 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6900000000000001e-05 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103864+0000", 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000003e-06 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104514+0000", 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000650856 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104600+0000", 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5352999999999994e-05 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104680+0000", 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0687999999999997e-05 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107887+0000", 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032067060000000001 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228173+0000", 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12028613 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228378+0000", 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020511699999999999 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228456+0000", 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7156000000000005e-05 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228498+0000", 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2373000000000002e-05 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:119604 5.4 5:24a0ff13:::benchmark_data_smithi154_144140_object9251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:00.328079+0000", 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.96535411, 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079768361999999995, 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.415 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 119604 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.328079+0000", 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.328079+0000", 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.328078+0000", 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.328085+0000", 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9319999999999999e-06 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.328098+0000", 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3457e-05 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.328106+0000", 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1589999999999993e-06 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.329880+0000", 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017732780000000001 2024-09-13T14:45:48.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.329991+0000", 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011099600000000001 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.330069+0000", 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8483000000000001e-05 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.333497+0000", 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034280600000000001 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.333542+0000", 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4765999999999998e-05 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.407758+0000", 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.074215924000000003 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.407809+0000", 2024-09-13T14:45:48.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1178999999999999e-05 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:00.407848+0000", 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.854e-05 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:154507 5.4 5:2764ce32:::benchmark_data_smithi154_144140_object44154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:43.497642+0000", 2024-09-13T14:45:48.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.795790963, 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082769283999999999, 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 154507 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.497642+0000", 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.497642+0000", 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.497641+0000", 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:45:48.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.497647+0000", 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2079999999999999e-06 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.497657+0000", 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190999999999999e-05 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.497664+0000", 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7649999999999996e-06 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.501070+0000", 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034065319999999999 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.501159+0000", 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8887000000000005e-05 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.501233+0000", 2024-09-13T14:45:48.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3677000000000006e-05 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.504433+0000", 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031998840000000001 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.511605+0000", 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071723029999999997 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.580301+0000", 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.068695918999999994 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.580361+0000", 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0272999999999997e-05 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:43.580412+0000", 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0671999999999999e-05 2024-09-13T14:45:48.427 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.428 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:45:48.428 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:45:48.428 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:45:48.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:48.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:48.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:48.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:48.699 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[102314]: pgmap v401: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 2274/198096 objects degraded (1.148%); 1554/198096 objects misplaced (0.784%); 2.9 MiB/s, 76 objects/s recovering 2024-09-13T14:45:48.699 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:48.699 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:48.699 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[102314]: osdmap e141: 8 total, 8 up, 7 in 2024-09-13T14:45:48.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[107038]: pgmap v401: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 2274/198096 objects degraded (1.148%); 1554/198096 objects misplaced (0.784%); 2.9 MiB/s, 76 objects/s recovering 2024-09-13T14:45:48.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:48.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:48.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:48 smithi154 ceph-mon[107038]: osdmap e141: 8 total, 8 up, 7 in 2024-09-13T14:45:48.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:48 smithi172 ceph-mon[100696]: pgmap v401: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 2274/198096 objects degraded (1.148%); 1554/198096 objects misplaced (0.784%); 2.9 MiB/s, 76 objects/s recovering 2024-09-13T14:45:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:48 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:45:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:48 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:45:48.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:48 smithi172 ceph-mon[100696]: osdmap e141: 8 total, 8 up, 7 in 2024-09-13T14:45:48.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:48.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:48.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:48.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:49.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:49.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:49.130 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:49.210 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:45:49.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:49.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:49.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:49.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:49.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:49.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:49.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:49.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:49.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:49.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:49.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:49.898 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:45:49.899 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:45:49.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:45:49.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:45:49.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:49.977 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:45:49.977 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:45:50.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:50.018 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:45:50.018 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:45:50.048 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:45:50.048 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:45:50.078 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:45:50.078 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:45:50.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:50.114 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:45:50.114 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:45:50.135 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:50.148 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:45:50.148 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:45:50.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:45:50.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:50.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:50.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:50.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:50.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:50.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:50.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:50.665 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:50 smithi154 ceph-mon[102314]: pgmap v403: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 744 op/s; 2274/192483 objects degraded (1.181%); 1357/192483 objects misplaced (0.705%); 3.6 MiB/s, 87 objects/s recovering 2024-09-13T14:45:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:50 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2274/192483 objects degraded (1.181%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:50.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3413752393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:50.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:50 smithi154 ceph-mon[107038]: pgmap v403: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 744 op/s; 2274/192483 objects degraded (1.181%); 1357/192483 objects misplaced (0.705%); 3.6 MiB/s, 87 objects/s recovering 2024-09-13T14:45:50.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:50 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2274/192483 objects degraded (1.181%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:50.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3413752393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:50.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:50 smithi172 ceph-mon[100696]: pgmap v403: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 744 op/s; 2274/192483 objects degraded (1.181%); 1357/192483 objects misplaced (0.705%); 3.6 MiB/s, 87 objects/s recovering 2024-09-13T14:45:50.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:50 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2274/192483 objects degraded (1.181%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:50.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3413752393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:45:50.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:50.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:50.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:50.921 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:50.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:51.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:51.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:51.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:51.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:51.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:51.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:51.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:51.461 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:51.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:51.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:51.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:51.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:51.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:51.883 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:51.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:51.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:52.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:52.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:52.149 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:45:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:45:51] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:45:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.164 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:52.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:52.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:52.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:52.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:52.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:52.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:52.548 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:52.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:52.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-13T14:45:52.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:52.706 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:52 smithi154 ceph-mon[102314]: pgmap v404: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 2274/186969 objects degraded (1.216%); 1357/186969 objects misplaced (0.726%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:45:52.706 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:52 smithi154 ceph-mon[107038]: pgmap v404: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 2274/186969 objects degraded (1.216%); 1357/186969 objects misplaced (0.726%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:45:52.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:52 smithi172 ceph-mon[100696]: pgmap v404: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 2274/186969 objects degraded (1.216%); 1357/186969 objects misplaced (0.726%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:45:52.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:52.816 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:52.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:52.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:52.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:53.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:53.072 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:53.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:53.202 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:53.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:53.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:53.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:53.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:53.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:53.587 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:53.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:53.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:53.661 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:53.680 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:45:53.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.773 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:53.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:53.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:53.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:53.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:54.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:54.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:54.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:54.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:54.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:54.345 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:54.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:54.474 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:54.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:54.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:54.650 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:54 smithi154 ceph-mon[102314]: pgmap v405: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 2274/184800 objects degraded (1.231%); 1159/184800 objects misplaced (0.627%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:54.650 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:54 smithi154 ceph-mon[107038]: pgmap v405: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 2274/184800 objects degraded (1.231%); 1159/184800 objects misplaced (0.627%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:54.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:54.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:54 smithi172 ceph-mon[100696]: pgmap v405: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 2274/184800 objects degraded (1.231%); 1159/184800 objects misplaced (0.627%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:54.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:54.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:54.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:54.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:54.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:55.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:55.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:55.173 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:55.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:55.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:55.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:55.455 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:55.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:45:55.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:55.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:55.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:55.725 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:55.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2274/184800 objects degraded (1.231%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:55.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:55.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:55.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:55.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:55.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2274/184800 objects degraded (1.231%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:55.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2274/184800 objects degraded (1.231%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:45:56.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.000 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:56.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:56.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:56.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:56.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:56.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:56.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.431 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:56.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:56.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:45:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:56.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:45:56.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:56.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:56.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:56 smithi154 ceph-mon[102314]: pgmap v406: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 2274/176808 objects degraded (1.286%); 1159/176808 objects misplaced (0.656%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:56.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:56 smithi154 ceph-mon[107038]: pgmap v406: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 2274/176808 objects degraded (1.286%); 1159/176808 objects misplaced (0.656%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:56.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:56 smithi172 ceph-mon[100696]: pgmap v406: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 2274/176808 objects degraded (1.286%); 1159/176808 objects misplaced (0.656%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:56.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:56.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:56.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:56.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:56.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:57.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:57.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:57.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:57.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:57.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:57.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:57.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:57.516 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:57.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:57.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:57.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:57.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:57.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:57.900 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:57.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:58.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:58.029 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:58.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:58.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:58.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:45:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:58.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:45:58.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:58.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:58.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:58.592 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:58.615 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:58 smithi154 ceph-mon[102314]: pgmap v407: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 2274/175380 objects degraded (1.297%); 1159/175380 objects misplaced (0.661%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:58.615 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:58.615 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:45:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:58.615 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:58 smithi154 ceph-mon[107038]: pgmap v407: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 2274/175380 objects degraded (1.297%); 1159/175380 objects misplaced (0.661%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:58.615 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:58.616 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:45:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:58.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:58.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:58.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:58 smithi172 ceph-mon[100696]: pgmap v407: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 756 op/s; 2274/175380 objects degraded (1.297%); 1159/175380 objects misplaced (0.661%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:45:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:45:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:45:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:45:58.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:45:58.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:45:58.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:45:58.982 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:58.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:45:59.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:59.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:59.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:45:59.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:45:59.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:59.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:59.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:45:59.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:45:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:45:59.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:45:59.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:45:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:59.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:45:59.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:45:59.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:45:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:45:59.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:00.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:00.057 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:00.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:00.107 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:00.107 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:00.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:00.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:00.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:00.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:00.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:00.482 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:00.513 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:46:00.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:00.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[102314]: pgmap v408: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 839 op/s; 2274/169344 objects degraded (1.343%); 918/169344 objects misplaced (0.542%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:46:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2274/169344 objects degraded (1.343%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:00.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/513030211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[107038]: pgmap v408: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 839 op/s; 2274/169344 objects degraded (1.343%); 918/169344 objects misplaced (0.542%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:46:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2274/169344 objects degraded (1.343%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/513030211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:00 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:00.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:00.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:00.892 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:00.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:00.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:00 smithi172 ceph-mon[100696]: pgmap v408: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 839 op/s; 2274/169344 objects degraded (1.343%); 918/169344 objects misplaced (0.542%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:46:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2274/169344 objects degraded (1.343%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/513030211' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:00 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:00.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.020 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:01.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:01.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:01.151 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:01.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:01.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:01.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:01.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:01.410 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:01.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:01.554 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:01.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:01.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:01.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:01 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:01.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:01 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:01.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:01 smithi154 ceph-mon[102314]: osdmap e142: 8 total, 8 up, 7 in 2024-09-13T14:46:01.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:01 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:01.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:01 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:01.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:01 smithi154 ceph-mon[107038]: osdmap e142: 8 total, 8 up, 7 in 2024-09-13T14:46:01.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:01.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:01.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:01.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:01.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:01.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:01 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:01.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:01 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:01.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:01 smithi172 ceph-mon[100696]: osdmap e142: 8 total, 8 up, 7 in 2024-09-13T14:46:02.064 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:02.088 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:02.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:02.204 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:46:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:46:01] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:46:02.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:02.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:46:02.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:02.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:02.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:02.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:02.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:02.632 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:02 smithi154 ceph-mon[102314]: pgmap v410: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 2274/163362 objects degraded (1.392%); 918/163362 objects misplaced (0.562%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:02.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:02 smithi154 ceph-mon[107038]: pgmap v410: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 2274/163362 objects degraded (1.392%); 918/163362 objects misplaced (0.562%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:02.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:02.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:02.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:02.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:02.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:02.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:02 smithi172 ceph-mon[100696]: pgmap v410: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 2274/163362 objects degraded (1.392%); 918/163362 objects misplaced (0.562%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:02.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:03.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:03.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:03.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:03.164 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:03.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.305 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:03.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:03.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:46:03.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:03.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:03.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:03.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:03.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:03.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:03.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:03.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:03.916 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:46:03.916 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:03.953 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:46:03.954 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:03.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:03.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:03.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:04.000 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:46:04.000 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:04.035 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:46:04.036 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:04.079 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:46:04.079 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:04.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:04.117 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:46:04.117 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:04.120 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:04.150 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:46:04.150 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:04.191 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:46:04.192 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:04.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:04.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:04.278 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7, 6] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:46:04.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:46:04.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:04.373 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:04.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:04.527 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:04.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:04.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:04 smithi154 ceph-mon[102314]: pgmap v411: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 2274/160959 objects degraded (1.413%); 706/160959 objects misplaced (0.439%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:04 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3496946826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:04 smithi154 ceph-mon[107038]: pgmap v411: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 2274/160959 objects degraded (1.413%); 706/160959 objects misplaced (0.439%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:04 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3496946826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:04.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.785 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:04.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:04.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:04.913 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:04.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:04.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:04 smithi172 ceph-mon[100696]: pgmap v411: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 2274/160959 objects degraded (1.413%); 706/160959 objects misplaced (0.439%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:04 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3496946826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:05.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:05.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:05.177 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:05.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:05.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:05.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:05.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:05.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:05.590 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:05.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:05.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:05.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:05.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:05.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:05.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2274/160959 objects degraded (1.413%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:05.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2274/160959 objects degraded (1.413%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:05.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:05.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2274/160959 objects degraded (1.413%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:06.011 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:06.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:06.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:06.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.296 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:06.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:06.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:06.451 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:06.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:06.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:06.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:06.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:06 smithi154 ceph-mon[102314]: pgmap v412: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2274/152856 objects degraded (1.488%); 706/152856 objects misplaced (0.462%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:06.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:06 smithi154 ceph-mon[107038]: pgmap v412: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2274/152856 objects degraded (1.488%); 706/152856 objects misplaced (0.462%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:06.734 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:06.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:06.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:06.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:06.959 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:06.963 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:06.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:06.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:06 smithi172 ceph-mon[100696]: pgmap v412: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2274/152856 objects degraded (1.488%); 706/152856 objects misplaced (0.462%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:07.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:07.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:07.185 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:07.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:07.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:07.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:07.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:07.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:07.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:07.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:07.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:07.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:07.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:07.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:07.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:07.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:08.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:08.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:08.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:08.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:08.245 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:08.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:08.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:46:08.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:08.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:08.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:08.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:08.742 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:08 smithi154 ceph-mon[102314]: pgmap v413: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2274/151440 objects degraded (1.502%); 706/151440 objects misplaced (0.466%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:08.742 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:08 smithi154 ceph-mon[107038]: pgmap v413: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2274/151440 objects degraded (1.502%); 706/151440 objects misplaced (0.466%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:08.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:08.790 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:08.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:08 smithi172 ceph-mon[100696]: pgmap v413: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2274/151440 objects degraded (1.502%); 706/151440 objects misplaced (0.466%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:08.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:08.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:08.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:09.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:09.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:09.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:09.207 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:09.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:09.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:09.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:09.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:09.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:09.610 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:09.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:09.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:09.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:09.870 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:09.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:09.872 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:46:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:46:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:46:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:46:09.873 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:46:09.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:10.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:10.177 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:10.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:10.307 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:10.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:10.439 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:10.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:10.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:10.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:10.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:10 smithi154 ceph-mon[102314]: pgmap v414: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 2274/145767 objects degraded (1.560%); 386/145767 objects misplaced (0.265%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:46:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2274/145767 objects degraded (1.560%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:10.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:10 smithi154 ceph-mon[107038]: pgmap v414: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 2274/145767 objects degraded (1.560%); 386/145767 objects misplaced (0.265%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:46:10.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2274/145767 objects degraded (1.560%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:10.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:10 smithi172 ceph-mon[100696]: pgmap v414: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 2274/145767 objects degraded (1.560%); 386/145767 objects misplaced (0.265%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:46:10.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2274/145767 objects degraded (1.560%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:10.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:10.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:10.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:10.964 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:10.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:11.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:11.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:11.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:11.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:11.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:46:11.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:11.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:11.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:11.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:11.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:11.554 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:46:11.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:11.647 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:11.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:11.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:11.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:11 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3531226809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:11.876 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:11 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3531226809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:11.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:11.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:11.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:11.937 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:11.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:11 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3531226809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:12.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:12.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:12.161 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:12.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:46:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:46:11] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:46:12.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:12.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:12.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:12.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:12.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:12.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:12.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[102314]: pgmap v415: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 744 op/s; 2274/140214 objects degraded (1.622%); 386/140214 objects misplaced (0.275%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:46:12.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:12.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3531226809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:12.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[102314]: osdmap e143: 8 total, 8 up, 7 in 2024-09-13T14:46:12.663 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[107038]: pgmap v415: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 744 op/s; 2274/140214 objects degraded (1.622%); 386/140214 objects misplaced (0.275%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:46:12.663 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:12.663 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3531226809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:12.663 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:12 smithi154 ceph-mon[107038]: osdmap e143: 8 total, 8 up, 7 in 2024-09-13T14:46:12.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:12.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:12.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:12.888 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:12.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:12.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:12 smithi172 ceph-mon[100696]: pgmap v415: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 744 op/s; 2274/140214 objects degraded (1.622%); 386/140214 objects misplaced (0.275%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:46:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:12 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3531226809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:12.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:12 smithi172 ceph-mon[100696]: osdmap e143: 8 total, 8 up, 7 in 2024-09-13T14:46:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:13.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:13.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:13.143 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:13.246 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:13.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:13.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.525 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:13.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:13.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:13.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:13.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:13.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:13.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:13.969 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:46:14.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:14.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:14.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:14.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[102314]: pgmap v417: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 2274/137964 objects degraded (1.648%); 80/137964 objects misplaced (0.058%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3343669272' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:46:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:46:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[107038]: pgmap v417: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 2274/137964 objects degraded (1.648%); 80/137964 objects misplaced (0.058%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3343669272' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:46:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:13 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:46:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:13 smithi172 ceph-mon[100696]: pgmap v417: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.8 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 2274/137964 objects degraded (1.648%); 80/137964 objects misplaced (0.058%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:14.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3343669272' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:46:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:13 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:46:14.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:14.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:14.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:46:14.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:14.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:14.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:14.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:14.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:14.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:14.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:14.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:14.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:14.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:14.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:15.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:15.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:15.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:14 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:46:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:14 smithi154 ceph-mon[102314]: osdmap e144: 8 total, 8 up, 7 in 2024-09-13T14:46:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:14 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2274/137964 objects degraded (1.648%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:14 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:46:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:14 smithi154 ceph-mon[107038]: osdmap e144: 8 total, 8 up, 7 in 2024-09-13T14:46:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:14 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2274/137964 objects degraded (1.648%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:14 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:46:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:14 smithi172 ceph-mon[100696]: osdmap e144: 8 total, 8 up, 7 in 2024-09-13T14:46:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:14 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2274/137964 objects degraded (1.648%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:15.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:15.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:15.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:15.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:15.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:15.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:15.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:15.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:15.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:15.846 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:15.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:15.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:15.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:16.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:16.102 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:16.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:16.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:15 smithi172 ceph-mon[100696]: pgmap v419: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 2274/129552 objects degraded (1.755%); 80/129552 objects misplaced (0.062%); 4.9 MiB/s, 78 objects/s recovering 2024-09-13T14:46:16.247 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:16.270 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:16.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:16.390 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:15 smithi154 ceph-mon[102314]: pgmap v419: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 2274/129552 objects degraded (1.755%); 80/129552 objects misplaced (0.062%); 4.9 MiB/s, 78 objects/s recovering 2024-09-13T14:46:16.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:15 smithi154 ceph-mon[107038]: pgmap v419: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 2274/129552 objects degraded (1.755%); 80/129552 objects misplaced (0.062%); 4.9 MiB/s, 78 objects/s recovering 2024-09-13T14:46:16.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:16.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:16.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:16.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:16.676 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:16.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:46:16.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:16.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:16.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:16.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:16.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:17.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:17.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:17.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:17.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.274 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:17.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:17.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:17.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:17.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:17.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:17.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:17.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:17.648 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:17.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:17.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:17.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:17.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:17.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:17.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:18.003 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:18.020 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:18.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:18.130 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:18 smithi154 ceph-mon[102314]: pgmap v420: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 2274/128055 objects degraded (1.776%); 80/128055 objects misplaced (0.062%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:46:18.130 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:18 smithi154 ceph-mon[102314]: osdmap e145: 8 total, 8 up, 7 in 2024-09-13T14:46:18.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:18 smithi154 ceph-mon[107038]: pgmap v420: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 2274/128055 objects degraded (1.776%); 80/128055 objects misplaced (0.062%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:46:18.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:18 smithi154 ceph-mon[107038]: osdmap e145: 8 total, 8 up, 7 in 2024-09-13T14:46:18.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:18.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:18.278 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:18.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:18.404 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:18.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:18 smithi172 ceph-mon[100696]: pgmap v420: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 2274/128055 objects degraded (1.776%); 80/128055 objects misplaced (0.062%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:46:18.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:18 smithi172 ceph-mon[100696]: osdmap e145: 8 total, 8 up, 7 in 2024-09-13T14:46:18.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:18.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:18.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:18.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:18.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:18.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:18.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:18.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:18.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:19.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:19.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:19.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:19.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:19.259 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:19 smithi154 ceph-mon[102314]: osdmap e146: 8 total, 8 up, 7 in 2024-09-13T14:46:19.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:19 smithi154 ceph-mon[107038]: osdmap e146: 8 total, 8 up, 7 in 2024-09-13T14:46:19.334 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:19.364 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:19.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:19.470 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:19.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:19 smithi172 ceph-mon[100696]: osdmap e146: 8 total, 8 up, 7 in 2024-09-13T14:46:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:19.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:19.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:19.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:19.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:19.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:19.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:19.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:19.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:20.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:20.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:20.160 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:20 smithi154 ceph-mon[102314]: pgmap v423: 89 pgs: 1 peering, 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.5 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 859 op/s; 16/122478 objects degraded (0.013%); 1912/122478 objects misplaced (1.561%); 1.3 MiB/s, 68 objects/s recovering 2024-09-13T14:46:20.160 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 16/122478 objects degraded (0.013%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:20.160 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:20 smithi154 ceph-mon[107038]: pgmap v423: 89 pgs: 1 peering, 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.5 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 859 op/s; 16/122478 objects degraded (0.013%); 1912/122478 objects misplaced (1.561%); 1.3 MiB/s, 68 objects/s recovering 2024-09-13T14:46:20.160 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 16/122478 objects degraded (0.013%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:20.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:20.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:20.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:20 smithi172 ceph-mon[100696]: pgmap v423: 89 pgs: 1 peering, 1 active+recovery_wait+degraded, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.5 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 859 op/s; 16/122478 objects degraded (0.013%); 1912/122478 objects misplaced (1.561%); 1.3 MiB/s, 68 objects/s recovering 2024-09-13T14:46:20.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 16/122478 objects degraded (0.013%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:20.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:20.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:20.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:20.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:20.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:20.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:20.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:20.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:20.713 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:20.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:20.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:20.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:20.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:20.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:21.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:21.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:21.118 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:21.119 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:21.119 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:21.119 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:21.119 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:21.119 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:21.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.230 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:21.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:21.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:21.344 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1673042665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:21.344 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1673042665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:21.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:21.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1673042665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:21.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:21.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:21.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:21.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:21.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:21.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:21.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:21.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:22.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:22.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:22.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:22.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:22 smithi154 ceph-mon[102314]: pgmap v424: 89 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.4 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 8/117039 objects degraded (0.007%); 1912/117039 objects misplaced (1.634%); 1.1 MiB/s, 60 objects/s recovering 2024-09-13T14:46:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:46:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:46:21] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:46:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:22 smithi154 ceph-mon[107038]: pgmap v424: 89 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.4 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 8/117039 objects degraded (0.007%); 1912/117039 objects misplaced (1.634%); 1.1 MiB/s, 60 objects/s recovering 2024-09-13T14:46:22.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:22.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:22.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:22.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:22.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:22.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:22 smithi172 ceph-mon[100696]: pgmap v424: 89 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.4 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 831 op/s; 8/117039 objects degraded (0.007%); 1912/117039 objects misplaced (1.634%); 1.1 MiB/s, 60 objects/s recovering 2024-09-13T14:46:22.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:22.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:22.705 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:22.705 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:22.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:22.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:22.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:22.870 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:22.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:22.996 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:22.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:23.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:23.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:23.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.251 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:23.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:23.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:23.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:23.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:23.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:23.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:23.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:46:23.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:23.678 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:23.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:23.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:23.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:23.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:23.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:24.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:24.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:24.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:24.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:24.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:24.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:24.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:24 smithi154 ceph-mon[102314]: pgmap v425: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.3 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 8/115119 objects degraded (0.007%); 1659/115119 objects misplaced (1.441%); 2.9 MiB/s, 84 objects/s recovering 2024-09-13T14:46:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:24 smithi154 ceph-mon[107038]: pgmap v425: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.3 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 8/115119 objects degraded (0.007%); 1659/115119 objects misplaced (1.441%); 2.9 MiB/s, 84 objects/s recovering 2024-09-13T14:46:24.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:24 smithi172 ceph-mon[100696]: pgmap v425: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.3 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 8/115119 objects degraded (0.007%); 1659/115119 objects misplaced (1.441%); 2.9 MiB/s, 84 objects/s recovering 2024-09-13T14:46:24.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:24.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:24.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:24.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:24.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:24.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:24.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:24.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:24.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:25.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:25.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:25.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/115119 objects degraded (0.007%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:25.128 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/115119 objects degraded (0.007%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:25.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:25.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:25.340 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:25.363 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:25.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:25.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:25.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/115119 objects degraded (0.007%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:25.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:25.565 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:25.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:25.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:25.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:25.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:25.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:25.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:25.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:26.085 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:46:26.094 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:46:26.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:26.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:26.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:26.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:26.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:26.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[102314]: pgmap v426: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 8/107469 objects degraded (0.007%); 1659/107469 objects misplaced (1.544%); 2.9 MiB/s, 84 objects/s recovering 2024-09-13T14:46:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2098792856' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4230134454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[107038]: pgmap v426: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 8/107469 objects degraded (0.007%); 1659/107469 objects misplaced (1.544%); 2.9 MiB/s, 84 objects/s recovering 2024-09-13T14:46:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2098792856' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4230134454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:26 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:26 smithi172 ceph-mon[100696]: pgmap v426: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 8/107469 objects degraded (0.007%); 1659/107469 objects misplaced (1.544%); 2.9 MiB/s, 84 objects/s recovering 2024-09-13T14:46:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2098792856' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:46:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4230134454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:26 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:46:26.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:26.539 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:26.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:26.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:26.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:26.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:26.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:26.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:26.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:27.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:27.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:27.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.194 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:27.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:27.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:27.321 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:27.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:27.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:27.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:27.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2098792856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:27.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:46:27.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[102314]: osdmap e147: 8 total, 8 up, 7 in 2024-09-13T14:46:27.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:27.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2098792856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:27.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:46:27.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:27 smithi154 ceph-mon[107038]: osdmap e147: 8 total, 8 up, 7 in 2024-09-13T14:46:27.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:27 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:27.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2098792856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:46:27.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:27 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:46:27.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:27 smithi172 ceph-mon[100696]: osdmap e147: 8 total, 8 up, 7 in 2024-09-13T14:46:27.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:27.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:27.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:27.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:27.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:27.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:27.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:27.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:27.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:28.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:28.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:28.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:28 smithi154 ceph-mon[102314]: pgmap v428: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 8/106128 objects degraded (0.008%); 1659/106128 objects misplaced (1.563%); 1.8 MiB/s, 31 objects/s recovering 2024-09-13T14:46:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:28 smithi154 ceph-mon[107038]: pgmap v428: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 8/106128 objects degraded (0.008%); 1659/106128 objects misplaced (1.563%); 1.8 MiB/s, 31 objects/s recovering 2024-09-13T14:46:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:28.235 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:28.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:28.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:28.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:28 smithi172 ceph-mon[100696]: pgmap v428: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.2 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 8/106128 objects degraded (0.008%); 1659/106128 objects misplaced (1.563%); 1.8 MiB/s, 31 objects/s recovering 2024-09-13T14:46:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:28.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:28.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:28.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:28.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:28.783 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:28.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:28.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:28.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:29.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:29.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:29.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:29.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:29.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:29.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:29.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:29.461 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:29.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:29.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:29.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:29.752 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:29.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:29.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:29.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:29.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:30.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:30.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:30.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:30.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:30 smithi154 ceph-mon[102314]: pgmap v429: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 732 op/s; 8/100485 objects degraded (0.008%); 1456/100485 objects misplaced (1.449%); 2.8 MiB/s, 46 objects/s recovering 2024-09-13T14:46:30.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/100485 objects degraded (0.008%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:30.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:30 smithi154 ceph-mon[107038]: pgmap v429: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 732 op/s; 8/100485 objects degraded (0.008%); 1456/100485 objects misplaced (1.449%); 2.8 MiB/s, 46 objects/s recovering 2024-09-13T14:46:30.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/100485 objects degraded (0.008%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:30.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.296 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:30.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:30.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:30.423 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:30.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:30 smithi172 ceph-mon[100696]: pgmap v429: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.0 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 732 op/s; 8/100485 objects degraded (0.008%); 1456/100485 objects misplaced (1.449%); 2.8 MiB/s, 46 objects/s recovering 2024-09-13T14:46:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/100485 objects degraded (0.008%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:30.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:30.551 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:30.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:30.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:30.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:30.823 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:30.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:30.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:30.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:31.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:31.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:31.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:31.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:31.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:31.369 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:31.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:31.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:31.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:31.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:31.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:31.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:31.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:31.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:31.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:32.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:32.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:32.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:46:32.136 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:32 smithi154 ceph-mon[102314]: pgmap v430: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 8/94380 objects degraded (0.008%); 1456/94380 objects misplaced (1.543%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:32.136 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:46:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:46:31] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-13T14:46:32.136 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:32 smithi154 ceph-mon[107038]: pgmap v430: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 8/94380 objects degraded (0.008%); 1456/94380 objects misplaced (1.543%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:32.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:32.198 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:32.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.350 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:32.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:32.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:32.480 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:32.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:32 smithi172 ceph-mon[100696]: pgmap v430: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 8/94380 objects degraded (0.008%); 1456/94380 objects misplaced (1.543%); 2.8 MiB/s, 45 objects/s recovering 2024-09-13T14:46:32.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:32.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:32.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:32.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:32.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:32.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:46:32.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:32.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:32.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:33.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:33.045 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:33.061 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:46:33.062 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:33.095 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:46:33.096 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:33.143 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:46:33.143 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:33.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:33.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:33.191 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:46:33.191 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:33.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:33.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3520906945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:33.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3520906945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:33.223 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:46:33.224 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:33.274 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:46:33.274 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:33.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:33.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:33.324 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:46:33.324 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:192686 5.0 e147/110 5:0f40704a:::benchmark_data_smithi154_144140_object82333:head v 147'12017, mlcod=147'12015)", 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:33.331814+0000", 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.006578731, 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066101069999999996, 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:46:33.345 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331814+0000", 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331814+0000", 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331816+0000", 2024-09-13T14:46:33.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.689e-06 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331827+0000", 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1054e-05 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331846+0000", 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9009999999999999e-05 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331857+0000", 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1039000000000001e-05 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331963+0000", 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010668299999999999 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:33.331986+0000", 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3139e-05 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:46:33.348 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:46:33.349 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:46:33.349 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:46:33.349 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:46:33.349 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 1 2024-09-13T14:46:33.349 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:46:33.358 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:46:33.359 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:33.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:33.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:33.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3520906945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:33.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:33.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:33.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:33.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:33.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:33.865 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:33.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:33.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:34.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:34.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:34.138 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:34.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:34.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:34.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:34.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:34.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:34.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:34 smithi154 ceph-mon[102314]: pgmap v431: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 8/92130 objects degraded (0.009%); 1228/92130 objects misplaced (1.333%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:34 smithi154 ceph-mon[107038]: pgmap v431: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 8/92130 objects degraded (0.009%); 1228/92130 objects misplaced (1.333%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:34.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:34 smithi172 ceph-mon[100696]: pgmap v431: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.9 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 766 op/s; 8/92130 objects degraded (0.009%); 1228/92130 objects misplaced (1.333%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:34.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:34.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:34.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:34.668 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:34.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:34.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:34.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:34.942 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:34.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:34.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:35.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:35.046 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:35.053 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:35.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:35.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:35.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:35.284 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/92130 objects degraded (0.009%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:35.285 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/92130 objects degraded (0.009%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:35.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:35.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:35.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/92130 objects degraded (0.009%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:35.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:46:35.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:35.533 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:35.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.672 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:35.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:35.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:35.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:35.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:35.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:35.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:35.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:36.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:36.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:36.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:36 smithi154 ceph-mon[102314]: pgmap v432: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 8/84249 objects degraded (0.009%); 1228/84249 objects misplaced (1.458%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:36.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:36 smithi154 ceph-mon[107038]: pgmap v432: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 8/84249 objects degraded (0.009%); 1228/84249 objects misplaced (1.458%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.251 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:36.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:36.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:36.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:36.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:36.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:36 smithi172 ceph-mon[100696]: pgmap v432: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 8/84249 objects degraded (0.009%); 1228/84249 objects misplaced (1.458%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:46:36.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:36.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:36.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:36.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:36.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:36.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:36.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:36.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:36.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:37.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:46:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.045 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:37.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:37.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:37.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:37.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:37.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:37.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:37.467 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:37.550 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:37.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:37.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:37.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:37.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:37.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:37.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:37.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:37.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:38.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:38.064 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:38.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:38.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:38.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:38.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:38.330 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:38 smithi154 ceph-mon[102314]: pgmap v433: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 8/82827 objects degraded (0.010%); 1228/82827 objects misplaced (1.483%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:46:38.330 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:38 smithi154 ceph-mon[107038]: pgmap v433: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 8/82827 objects degraded (0.010%); 1228/82827 objects misplaced (1.483%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:46:38.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:38.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:38.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:38 smithi172 ceph-mon[100696]: pgmap v433: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.7 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 8/82827 objects degraded (0.010%); 1228/82827 objects misplaced (1.483%); 2.5 MiB/s, 39 objects/s recovering 2024-09-13T14:46:38.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:38.576 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:38.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:38.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:38.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:38.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:38.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:38.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:39.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:39.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:39.146 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:39.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:39.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:39.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:39.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:39.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:39.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:39.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:39.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:39.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:39.657 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:39.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:39.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:39.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:39.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:39.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:40.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:40.056 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:40.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:40.151 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:46:40.151 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:46:40.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:40.181 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:40.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:40 smithi172 ceph-mon[100696]: pgmap v434: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.6 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 707 op/s; 8/80640 objects degraded (0.010%); 991/80640 objects misplaced (1.229%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:46:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:40 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3032635095' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:46:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:40 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:46:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:40 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3817333076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:40 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/80640 objects degraded (0.010%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:40.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:40.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:40.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:40.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[102314]: pgmap v434: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.6 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 707 op/s; 8/80640 objects degraded (0.010%); 991/80640 objects misplaced (1.229%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:46:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3032635095' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:46:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:46:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3817333076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/80640 objects degraded (0.010%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:40.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[107038]: pgmap v434: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.6 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 707 op/s; 8/80640 objects degraded (0.010%); 991/80640 objects misplaced (1.229%); 3.5 MiB/s, 55 objects/s recovering 2024-09-13T14:46:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3032635095' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:46:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:46:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3817333076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:46:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/80640 objects degraded (0.010%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:40.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:40.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:40.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.706 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:40.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:40.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:40.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:40.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:40.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:40.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:41.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:41.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:41.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:41.237 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:41.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:41.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:46:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[102314]: osdmap e148: 8 total, 8 up, 7 in 2024-09-13T14:46:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:46:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-mon[107038]: osdmap e148: 8 total, 8 up, 7 in 2024-09-13T14:46:41.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:41 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:46:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:41 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:46:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:41 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:46:41.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:41 smithi172 ceph-mon[100696]: osdmap e148: 8 total, 8 up, 7 in 2024-09-13T14:46:41.496 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:41.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:41.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:41.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:41.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:41.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:41.900 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:41.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:42.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:46:42.026 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:42.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:42.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:42.166 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:46:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:46:41] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-13T14:46:42.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:42.305 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:42.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:42.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:42 smithi154 ceph-mon[102314]: pgmap v436: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 8/71082 objects degraded (0.011%); 991/71082 objects misplaced (1.394%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:46:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:42 smithi154 ceph-mon[107038]: pgmap v436: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 8/71082 objects degraded (0.011%); 991/71082 objects misplaced (1.394%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:46:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:42 smithi172 ceph-mon[100696]: pgmap v436: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 776 op/s; 8/71082 objects degraded (0.011%); 991/71082 objects misplaced (1.394%); 2.9 MiB/s, 46 objects/s recovering 2024-09-13T14:46:42.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:42.567 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:42.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:42.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:42.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:42.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:42.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:42.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:42.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:43.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:43.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:43.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:43.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:43.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:43.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:43.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:43.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:43.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:43.649 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:43.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:43.776 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:43.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:43.903 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:43.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:44.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:44.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:44.061 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:43 smithi154 ceph-mon[102314]: pgmap v437: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 8/68808 objects degraded (0.012%); 749/68808 objects misplaced (1.089%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:44.062 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:44.062 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:44.062 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:43 smithi154 ceph-mon[107038]: pgmap v437: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 8/68808 objects degraded (0.012%); 749/68808 objects misplaced (1.089%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:44.062 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:44.062 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:44.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:44.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:43 smithi172 ceph-mon[100696]: pgmap v437: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.4 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 8/68808 objects degraded (0.012%); 749/68808 objects misplaced (1.089%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:44.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:44.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:44.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:44.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:44.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:44.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:44.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:44.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:44.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:44.861 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:44.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:44.988 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:44.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:45.034 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:46:45.034 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:46:45.046 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:46:45.046 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:46:45.046 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:46:45.046 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:46:45.047 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:46:45.081 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:46:45.081 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:46:45.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:45.117 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:46:45.118 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:46:45.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:45.156 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:46:45.156 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:46:45.157 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/68808 objects degraded (0.012%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:45.157 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/68808 objects degraded (0.012%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:45.196 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:46:45.197 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:46:45.214 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:46:45.214 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:46:45.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:45.230 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:46:45.231 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:46:45.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/68808 objects degraded (0.012%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:45.249 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:46:45.250 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:46:45.251 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:45.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:45.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:46:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:45.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:45.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:45.516 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:45.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:45.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:45.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:45.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:45.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:45.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:45.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:46.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:46.052 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:46.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:46.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:45 smithi172 ceph-mon[100696]: pgmap v438: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 8/60804 objects degraded (0.013%); 749/60804 objects misplaced (1.232%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:45 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1805593660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:46.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:45 smithi154 ceph-mon[102314]: pgmap v438: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 8/60804 objects degraded (0.013%); 749/60804 objects misplaced (1.232%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:46.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:45 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1805593660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:46.342 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:45 smithi154 ceph-mon[107038]: pgmap v438: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 8/60804 objects degraded (0.013%); 749/60804 objects misplaced (1.232%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:46.342 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:45 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1805593660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:46:46.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:46.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:46.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:46.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.730 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:46.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:46.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:46.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:46.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:46.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:46.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:47.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:47.134 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:47.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:47.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:47.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:47.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:46:47.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:47.332 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:47.367 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:47.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:47.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:46:47.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.569 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:47.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:47.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:47.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:47.841 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:47.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:47.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:47.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:48.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:48.115 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:47 smithi172 ceph-mon[100696]: pgmap v439: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 8/59322 objects degraded (0.013%); 749/59322 objects misplaced (1.263%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:48.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:48.173 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:47 smithi154 ceph-mon[102314]: pgmap v439: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 8/59322 objects degraded (0.013%); 749/59322 objects misplaced (1.263%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:48.173 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:47 smithi154 ceph-mon[107038]: pgmap v439: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 8/59322 objects degraded (0.013%); 749/59322 objects misplaced (1.263%); 3.0 MiB/s, 47 objects/s recovering 2024-09-13T14:46:48.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:48.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:48.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:48.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:48.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:48.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:48.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:48.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:48.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:46:48.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:48.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:48.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:48.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:48.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:49.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:49.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:49.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:49.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:49.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:49.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:49.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:49.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:49.649 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:49.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:49.813 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:49.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:49.912 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:49.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:50.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:50.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:50.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:50.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:50.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:50 smithi172 ceph-mon[100696]: pgmap v440: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 8/57060 objects degraded (0.014%); 471/57060 objects misplaced (0.825%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:46:50.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:50 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/57060 objects degraded (0.014%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:50.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:50.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:50.390 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:50 smithi154 ceph-mon[102314]: pgmap v440: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 8/57060 objects degraded (0.014%); 471/57060 objects misplaced (0.825%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:46:50.391 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:50 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/57060 objects degraded (0.014%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:50.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:50 smithi154 ceph-mon[107038]: pgmap v440: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 1.2 GiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 8/57060 objects degraded (0.014%); 471/57060 objects misplaced (0.825%); 3.2 MiB/s, 51 objects/s recovering 2024-09-13T14:46:50.391 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:50 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/57060 objects degraded (0.014%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:50.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:50.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:50.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:50.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:50.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:50.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:50.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:50.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:50.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:50.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:50.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:51.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:51.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:51.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:51.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:51.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:51.388 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:51.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:51.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:51.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:51.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:51.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:51.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:51.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:51.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:51.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:52.029 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:52.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:52.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:52.162 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:52 smithi154 ceph-mon[102314]: pgmap v441: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 994 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 8/47583 objects degraded (0.017%); 471/47583 objects misplaced (0.990%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:46:52.162 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:52.162 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:52.163 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:46:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:46:51] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-09-13T14:46:52.163 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:52 smithi154 ceph-mon[107038]: pgmap v441: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 994 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 8/47583 objects degraded (0.017%); 471/47583 objects misplaced (0.990%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:46:52.163 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:52.163 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:52.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:52.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:52.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:52.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:52.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:52 smithi172 ceph-mon[100696]: pgmap v441: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 994 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 8/47583 objects degraded (0.017%); 471/47583 objects misplaced (0.990%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:46:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:52.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:52.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:52.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:52.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:52.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:52.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:52.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:52.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:52.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:53.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:53.011 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:46:53.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:53.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:53.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:53.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:53.265 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:53.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:53.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:53.426 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:53 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2023054342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:53.426 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:53 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:53.427 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:53 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2023054342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:53.427 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:53 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:53.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:53 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2023054342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:53.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:53 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:46:53.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.561 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:53.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:53.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:53.687 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:53.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:53.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:53.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:53.989 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:53.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:54.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:54.113 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[102314]: pgmap v442: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 946 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 716 op/s; 8/45294 objects degraded (0.018%); 122/45294 objects misplaced (0.269%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:46:54.113 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:54.113 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:54.113 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[102314]: osdmap e149: 8 total, 8 up, 7 in 2024-09-13T14:46:54.114 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[107038]: pgmap v442: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 946 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 716 op/s; 8/45294 objects degraded (0.018%); 122/45294 objects misplaced (0.269%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:46:54.114 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:54.114 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:54.114 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:54 smithi154 ceph-mon[107038]: osdmap e149: 8 total, 8 up, 7 in 2024-09-13T14:46:54.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:54.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.252 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:54.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:54.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:54.377 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:54.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:54.482 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:46:54.483 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:46:54.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:54 smithi172 ceph-mon[100696]: pgmap v442: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 946 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 716 op/s; 8/45294 objects degraded (0.018%); 122/45294 objects misplaced (0.269%); 4.5 MiB/s, 72 objects/s recovering 2024-09-13T14:46:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:54 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:46:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:54 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:46:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:54 smithi172 ceph-mon[100696]: osdmap e149: 8 total, 8 up, 7 in 2024-09-13T14:46:54.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:54.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:54.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:54.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:54.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:54.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:54.928 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:54.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:55.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.068 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.133 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/45294 objects degraded (0.018%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:55.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/45294 objects degraded (0.018%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:55.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:55.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:55.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:55.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:55.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:55.480 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:55.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/45294 objects degraded (0.018%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:46:55.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:55.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:55.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:46:55.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:55.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:55.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:55.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:55.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:55.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:55.988 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:55.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:56.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:56.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:56.120 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:56.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:56.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:56 smithi154 ceph-mon[102314]: pgmap v444: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 778 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 8/37188 objects degraded (0.022%); 122/37188 objects misplaced (0.328%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:56.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:56 smithi154 ceph-mon[107038]: pgmap v444: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 778 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 8/37188 objects degraded (0.022%); 122/37188 objects misplaced (0.328%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:56.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:56.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:56.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:56.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:56.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:56 smithi172 ceph-mon[100696]: pgmap v444: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 778 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 8/37188 objects degraded (0.022%); 122/37188 objects misplaced (0.328%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:56.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:56.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:56.841 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:46:56.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:56.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:56.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:57.047 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:46:57.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:57.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:57.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:46:57.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:57.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:57.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:57.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:57.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2388085547' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:46:57.352 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2388085547' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:46:57.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:57.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2388085547' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:46:57.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:46:57.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:46:57.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:57.721 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:57.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:57.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:57.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:57.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:57.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:58.094 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:58.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[102314]: pgmap v445: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 747 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 8/35727 objects degraded (0.022%); 122/35727 objects misplaced (0.341%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2388085547' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:46:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[102314]: osdmap e150: 8 total, 8 up, 7 in 2024-09-13T14:46:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[107038]: pgmap v445: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 747 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 8/35727 objects degraded (0.022%); 122/35727 objects misplaced (0.341%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2388085547' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:46:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[107038]: osdmap e150: 8 total, 8 up, 7 in 2024-09-13T14:46:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:58.222 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:58.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:46:58.364 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:46:58.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:58 smithi172 ceph-mon[100696]: pgmap v445: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 747 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 8/35727 objects degraded (0.022%); 122/35727 objects misplaced (0.341%); 3.9 MiB/s, 62 objects/s recovering 2024-09-13T14:46:58.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2388085547' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:46:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:58 smithi172 ceph-mon[100696]: osdmap e150: 8 total, 8 up, 7 in 2024-09-13T14:46:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:46:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:46:58.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:58.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:58.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:46:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.647 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:58.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:58.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:58.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:58.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:46:58.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:58.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:46:59.024 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:46:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:46:59.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:59.157 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:46:59 smithi154 ceph-mon[102314]: osdmap e151: 8 total, 8 up, 7 in 2024-09-13T14:46:59.158 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:46:59 smithi154 ceph-mon[107038]: osdmap e151: 8 total, 8 up, 7 in 2024-09-13T14:46:59.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:59.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:59.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:59.304 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:46:59.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.432 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:59.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:59.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:46:59 smithi172 ceph-mon[100696]: osdmap e151: 8 total, 8 up, 7 in 2024-09-13T14:46:59.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:46:59.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:46:59.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:46:59.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:46:59.710 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:46:59.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:46:59.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:46:59.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:46:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:46:59.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:46:59.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:00.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:00.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:00.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:00.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:00 smithi154 ceph-mon[102314]: pgmap v448: 89 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 707 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 8/33822 objects degraded (0.024%); 1.3 MiB/s, 67 objects/s recovering 2024-09-13T14:47:00.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8/33822 objects degraded (0.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:47:00.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:00 smithi154 ceph-mon[107038]: pgmap v448: 89 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 707 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 8/33822 objects degraded (0.024%); 1.3 MiB/s, 67 objects/s recovering 2024-09-13T14:47:00.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8/33822 objects degraded (0.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:47:00.222 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:00.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.359 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:00.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:00.422 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:47:00.423 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:00.459 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:47:00.459 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:00.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:00.484 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:00 smithi172 ceph-mon[100696]: pgmap v448: 89 pgs: 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 707 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 8/33822 objects degraded (0.024%); 1.3 MiB/s, 67 objects/s recovering 2024-09-13T14:47:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8/33822 objects degraded (0.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:47:00.499 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:47:00.499 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:00.536 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:47:00.536 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:00.574 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:47:00.574 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:00.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:00.609 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:47:00.609 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:00.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:00.642 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:47:00.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:00.677 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:47:00.677 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:00.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:00.752 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:00.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:00.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:00.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:01.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:01.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:01.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:01.181 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:01.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:01.317 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3123185255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:01.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3123185255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:01.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:01.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:01.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:01.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3123185255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:01.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:01.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:01.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:01.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:01.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:01.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:01.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:47:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:47:01] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:47:01.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:02.024 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:02.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:02.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:02.173 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:02.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:02.296 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:02.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:02.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:02 smithi154 ceph-mon[102314]: pgmap v449: 89 pgs: 89 active+clean; 538 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 1.0 MiB/s, 52 objects/s recovering 2024-09-13T14:47:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:02 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/33822 objects degraded (0.024%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:47:02.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:02 smithi154 ceph-mon[107038]: pgmap v449: 89 pgs: 89 active+clean; 538 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 1.0 MiB/s, 52 objects/s recovering 2024-09-13T14:47:02.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:02 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/33822 objects degraded (0.024%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:47:02.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:02 smithi172 ceph-mon[100696]: pgmap v449: 89 pgs: 89 active+clean; 538 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 828 op/s; 1.0 MiB/s, 52 objects/s recovering 2024-09-13T14:47:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:02 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/33822 objects degraded (0.024%), 2 pgs degraded, 2 pgs undersized) 2024-09-13T14:47:02.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:02.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:02.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:02.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:02.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:02.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:02.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:02.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:02.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:03.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:03.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:03.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:03.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:03.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:03.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:03.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:03.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:03.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:03.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:03.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:03.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:03.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:03.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:03.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:03.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.044 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:04.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:04.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:04.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:04.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:04.186 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:04 smithi154 ceph-mon[102314]: pgmap v450: 89 pgs: 89 active+clean; 506 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 541 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-09-13T14:47:04.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:04 smithi154 ceph-mon[107038]: pgmap v450: 89 pgs: 89 active+clean; 506 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 541 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-09-13T14:47:04.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:04.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:04.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:04.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:04.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:04 smithi172 ceph-mon[100696]: pgmap v450: 89 pgs: 89 active+clean; 506 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 541 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-09-13T14:47:04.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:04.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:04.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:04.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:04.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:04.838 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:04.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:04.983 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:04.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:05.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:05.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:05.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:05.242 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:05.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:05.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:05.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:05.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:05.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:05.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:05.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:05.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:05.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:05.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:05.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:05.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:06.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:06.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:06.149 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:47:06.149 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:47:06.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.149 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216130 5.b 5:dc61108e:::benchmark_data_smithi154_144140_object105777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.149 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.114791+0000", 2024-09-13T14:47:06.149 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.048873478999999997, 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.048914605, 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216130 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114791+0000", 2024-09-13T14:47:06.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114791+0000", 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114792+0000", 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8899999999999999e-07 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114795+0000", 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.642e-06 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114805+0000", 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3540000000000006e-06 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114811+0000", 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6540000000000002e-06 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114866+0000", 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4975000000000001e-05 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114962+0000", 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6031000000000003e-05 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:47:06.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.115060+0000", 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7801999999999996e-05 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.126722+0000", 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011661698999999999 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.126819+0000", 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7069000000000002e-05 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.154 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216139 5.b 5:d16f6513:::benchmark_data_smithi154_144140_object105786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.130584+0000", 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.033080875000000003, 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033212474999999998, 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216139 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130584+0000", 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130584+0000", 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130584+0000", 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1600000000000001e-07 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130587+0000", 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.264e-06 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130596+0000", 2024-09-13T14:47:06.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1859999999999992e-06 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130601+0000", 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6690000000000003e-06 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.136651+0000", 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060497040000000004 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.136754+0000", 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102775 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.136856+0000", 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102606 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:216137 5.2 e151/150 5:475571b5:::benchmark_data_smithi154_144140_object105784:head v 151'13453, mlcod=151'13451)", 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.132654+0000", 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.031010737999999999, 2024-09-13T14:47:06.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031218306000000001, 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.132654+0000", 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.132654+0000", 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.132655+0000", 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1230000000000001e-06 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.132663+0000", 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5979999999999997e-06 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.132677+0000", 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3213e-05 2024-09-13T14:47:06.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.132696+0000", 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9581000000000001e-05 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.133515+0000", 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081880899999999999 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.133532+0000", 2024-09-13T14:47:06.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7227999999999998e-05 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:216138 5.2 e151/150 5:4af3599c:::benchmark_data_smithi154_144140_object105785:head v 151'13454, mlcod=151'13451)", 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137748+0000", 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025916725000000002, 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026179211000000001, 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:47:06.163 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137748+0000", 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137748+0000", 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137749+0000", 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2299999999999999e-07 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137757+0000", 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9659999999999996e-06 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137767+0000", 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.084e-05 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137775+0000", 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5680000000000002e-06 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137885+0000", 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011032 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137948+0000", 2024-09-13T14:47:06.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2390000000000004e-05 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:216126 5.a e151/110)", 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137798+0000", 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025866285999999999, 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026177718999999999, 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137798+0000", 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137798+0000", 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137799+0000", 2024-09-13T14:47:06.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5700000000000001e-07 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137802+0000", 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1549999999999999e-06 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137810+0000", 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5660000000000001e-06 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137814+0000", 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4209999999999997e-06 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:216128 5.a e151/110)", 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137823+0000", 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.02584121, 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026191364000000002, 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:06.170 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137823+0000", 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137823+0000", 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137823+0000", 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5100000000000001e-07 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137825+0000", 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9309999999999998e-06 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137831+0000", 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4439999999999999e-06 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137835+0000", 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0080000000000004e-06 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216140 5.8 5:115f50ee:::benchmark_data_smithi154_144140_object105787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137828+0000", 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025836438999999999, 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026232558, 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T14:47:06.173 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216140 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137828+0000", 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137828+0000", 2024-09-13T14:47:06.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137830+0000", 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6440000000000001e-06 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137834+0000", 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8280000000000002e-06 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137844+0000", 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5459999999999997e-06 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137850+0000", 2024-09-13T14:47:06.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1149999999999999e-06 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.138338+0000", 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048809299999999999 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216141 5.d 5.107d9efd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137873+0000", 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025791199000000001, 2024-09-13T14:47:06.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026241595, 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216141 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137873+0000", 2024-09-13T14:47:06.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137873+0000", 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137874+0000", 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2399999999999998e-07 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137876+0000", 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.649e-06 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137884+0000", 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3100000000000003e-06 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137889+0000", 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6879999999999996e-06 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.180 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:216125 5.a e151/110)", 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137919+0000", 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025745338999999999, 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026245807999999999, 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137919+0000", 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137919+0000", 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137920+0000", 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4200000000000001e-07 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137922+0000", 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.886e-06 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137929+0000", 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8820000000000003e-06 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137934+0000", 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1399999999999999e-06 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.183 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:216126 5.a e151/110)", 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137942+0000", 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025721938, 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026260070999999999, 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137942+0000", 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137942+0000", 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137943+0000", 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4900000000000002e-07 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137945+0000", 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.866e-06 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137950+0000", 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.541e-06 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137954+0000", 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1409999999999998e-06 2024-09-13T14:47:06.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:216130 5.b e151/111)", 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.137982+0000", 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025682228000000001, 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026261240000000002, 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137982+0000", 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137982+0000", 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137983+0000", 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5699999999999998e-07 2024-09-13T14:47:06.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137985+0000", 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6460000000000002e-06 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137991+0000", 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9309999999999996e-06 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.137997+0000", 2024-09-13T14:47:06.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.891e-06 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216124 5.d 5:b49f24b5:::benchmark_data_smithi154_144140_object105771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.113649+0000", 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.050015621000000003, 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.050640685999999997, 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.190 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216124 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.113649+0000", 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.113649+0000", 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.113650+0000", 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6360000000000001e-06 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.113654+0000", 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.788e-06 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.113664+0000", 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T14:47:06.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.113670+0000", 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7060000000000001e-06 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114436+0000", 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00076674000000000002 2024-09-13T14:47:06.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114555+0000", 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011838200000000001 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114668+0000", 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011359800000000001 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130153+0000", 2024-09-13T14:47:06.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015485268 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130181+0000", 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7628999999999999e-05 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.195 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216142 5.a 5.db7d3bda (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.138244+0000", 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.025420460999999998, 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026120338999999999, 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216142 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.138244+0000", 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.138244+0000", 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.138245+0000", 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3200000000000005e-07 2024-09-13T14:47:06.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.138248+0000", 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.999e-06 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.138257+0000", 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3919999999999992e-06 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.138264+0000", 2024-09-13T14:47:06.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5390000000000004e-06 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216125 5.a 5:5b8a7376:::benchmark_data_smithi154_144140_object105772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.114295+0000", 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.049369414, 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.050122861999999997, 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.199 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216125 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114295+0000", 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114295+0000", 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114296+0000", 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2999999999999999e-07 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114300+0000", 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1509999999999997e-06 2024-09-13T14:47:06.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114310+0000", 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8020000000000003e-06 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114316+0000", 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6660000000000002e-06 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.116935+0000", 2024-09-13T14:47:06.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002618202 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.117055+0000", 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120506 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.117171+0000", 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115362 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.127834+0000", 2024-09-13T14:47:06.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010663600000000001 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.127876+0000", 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1375999999999998e-05 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216126 5.a 5:54573b0b:::benchmark_data_smithi154_144140_object105773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.114339+0000", 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.049325193000000003, 2024-09-13T14:47:06.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.050154542000000003, 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216126 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114339+0000", 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114339+0000", 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114340+0000", 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.89e-07 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114342+0000", 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4770000000000002e-06 2024-09-13T14:47:06.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114349+0000", 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9090000000000003e-06 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114354+0000", 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0629999999999996e-06 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.122322+0000", 2024-09-13T14:47:06.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079681969999999998 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.122438+0000", 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116118 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.122550+0000", 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111731 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216127 5.d 5:b4cfcd84:::benchmark_data_smithi154_144140_object105774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.114362+0000", 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.049302713999999997, 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.050202732999999999, 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216127 2024-09-13T14:47:06.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114362+0000", 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114362+0000", 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114362+0000", 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0499999999999999e-07 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114364+0000", 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.697e-06 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114369+0000", 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.429e-06 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114374+0000", 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2640000000000002e-06 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.125009+0000", 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010635466999999999 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.125107+0000", 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7683000000000006e-05 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.125207+0000", 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9846000000000006e-05 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.130854+0000", 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056471580000000002 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:06.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.134883+0000", 2024-09-13T14:47:06.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040296120000000001 2024-09-13T14:47:06.214 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.214 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:216128 5.a 5:548c1a8a:::benchmark_data_smithi154_144140_object105775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:47:04.114384+0000", 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.049280319000000003, 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.050255774000000003, 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:06.215 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 216128 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114384+0000", 2024-09-13T14:47:06.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114384+0000", 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:06.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114384+0000", 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6600000000000002e-07 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114386+0000", 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.81e-06 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114392+0000", 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.993e-06 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.114397+0000", 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3250000000000001e-06 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.127902+0000", 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013505827999999999 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.128015+0000", 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112372 2024-09-13T14:47:06.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:47:04.128127+0000", 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111866 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 17 2024-09-13T14:47:06.220 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:47:06.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:06.255 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:06.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:06.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:06 smithi154 ceph-mon[102314]: pgmap v451: 89 pgs: 89 active+clean; 324 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 846 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-09-13T14:47:06.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:06 smithi154 ceph-mon[107038]: pgmap v451: 89 pgs: 89 active+clean; 324 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 846 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-09-13T14:47:06.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:06 smithi172 ceph-mon[100696]: pgmap v451: 89 pgs: 89 active+clean; 324 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 846 op/s; 1.0 MiB/s, 51 objects/s recovering 2024-09-13T14:47:06.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:06.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:06.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:06.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:06.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:06.810 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:06.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:06.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:06.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:07.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:07.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:07.164 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:07.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:07.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:47:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:07.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:07.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:07.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:07.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:07.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:07.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:07.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:07.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:07.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:07.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:07.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:07.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:08.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:08.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:08.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:08.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:08.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:08.356 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:08 smithi172 ceph-mon[100696]: pgmap v452: 89 pgs: 89 active+clean; 293 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 852 KiB/s, 42 objects/s recovering 2024-09-13T14:47:08.364 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:08 smithi154 ceph-mon[102314]: pgmap v452: 89 pgs: 89 active+clean; 293 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 852 KiB/s, 42 objects/s recovering 2024-09-13T14:47:08.365 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:08 smithi154 ceph-mon[107038]: pgmap v452: 89 pgs: 89 active+clean; 293 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 852 KiB/s, 42 objects/s recovering 2024-09-13T14:47:08.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:08.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:08.530 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:08.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:08.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:08.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:08.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:08.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:08.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:08.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:09.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:09.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:09.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:09.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:09.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.359 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:09.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:09.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:09.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:09.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:09.626 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:09.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:09.767 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Removed 110350 objects 2024-09-13T14:47:09.767 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Clean up completed and total clean up time :142.223 2024-09-13T14:47:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.768 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:09.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:09.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:09.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:09.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:09.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:10.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:10.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:10.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:10.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:10 smithi154 ceph-mon[107038]: pgmap v453: 89 pgs: 89 active+clean; 246 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 703 op/s; 774 KiB/s, 38 objects/s recovering 2024-09-13T14:47:10.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:10 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:10.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:10 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:10 smithi154 ceph-mon[102314]: pgmap v453: 89 pgs: 89 active+clean; 246 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 703 op/s; 774 KiB/s, 38 objects/s recovering 2024-09-13T14:47:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:10 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:10 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:10.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:10 smithi172 ceph-mon[100696]: pgmap v453: 89 pgs: 89 active+clean; 246 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 703 op/s; 774 KiB/s, 38 objects/s recovering 2024-09-13T14:47:10.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:10 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:10.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:10 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:10.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:10.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:10.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.390 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:10.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:10.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:10.512 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:10.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:10.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:10.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:10.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:10.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:10.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:10.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:10.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:10.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:11.009 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:11.101 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:47:11.101 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:47:11.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3403149039' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:47:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:47:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2891159157' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:11.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3403149039' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:47:11.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:47:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2891159157' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:11.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:11.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:11.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:11.389 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:11.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:11 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3403149039' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:47:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:11 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:47:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:11 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2891159157' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:11 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:11.515 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:11.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:11.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:11.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.771 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:11.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T14:47:11.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:11.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:11.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:11.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:12.022 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:12.098 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:47:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:47:11] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:47:12.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:12.146 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:12.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:12.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:12.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:12.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:12.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: pgmap v454: 89 pgs: 89 active+clean; 44 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:47:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:47:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: osdmap e152: 8 total, 8 up, 7 in 2024-09-13T14:47:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: pgmap v454: 89 pgs: 89 active+clean; 44 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: osdmap e152: 8 total, 8 up, 7 in 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:12 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:12.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: pgmap v454: 89 pgs: 89 active+clean; 44 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 48 KiB/s, 0 objects/s recovering 2024-09-13T14:47:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:47:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: osdmap e152: 8 total, 8 up, 7 in 2024-09-13T14:47:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:47:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:47:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:12 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:12.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:12.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:12.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:12.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:12.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:12.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:12.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:12.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:12.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:12.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:13.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:13.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:13.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:13.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:13.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:13.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:13.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]} : dispatch 2024-09-13T14:47:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-13T14:47:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]} : dispatch 2024-09-13T14:47:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-13T14:47:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:13.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]} : dispatch 2024-09-13T14:47:13.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]} : dispatch 2024-09-13T14:47:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:13.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:13.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:13.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:13.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:13.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:13.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:13.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:13.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:13.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:14.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:14.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:14.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:14.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:14.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:14.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:14.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:14.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:14.400 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:14.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:14.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[102314]: pgmap v456: 89 pgs: 89 active+clean; 31 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:47:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-09-13T14:47:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:47:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[102314]: osdmap e153: 8 total, 8 up, 7 in 2024-09-13T14:47:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[107038]: pgmap v456: 89 pgs: 89 active+clean; 31 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:47:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-09-13T14:47:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:47:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:14 smithi154 ceph-mon[107038]: osdmap e153: 8 total, 8 up, 7 in 2024-09-13T14:47:14.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:14 smithi172 ceph-mon[100696]: pgmap v456: 89 pgs: 89 active+clean; 31 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 760 op/s 2024-09-13T14:47:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-09-13T14:47:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-09-13T14:47:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:14 smithi172 ceph-mon[100696]: osdmap e153: 8 total, 8 up, 7 in 2024-09-13T14:47:14.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:14.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:14.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:14.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:14.805 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:14.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:14.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:14.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:15.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:47:15.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.091 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:15.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:15.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:15.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:15.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:15.408 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T14:47:15.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T14:47:15.409 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:15 smithi154 ceph-mon[102314]: osdmap e154: 8 total, 8 up, 7 in 2024-09-13T14:47:15.409 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:15 smithi154 ceph-mon[107038]: osdmap e154: 8 total, 8 up, 7 in 2024-09-13T14:47:15.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:15.474 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:15.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:15 smithi172 ceph-mon[100696]: osdmap e154: 8 total, 8 up, 7 in 2024-09-13T14:47:15.517 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:47:15.517 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:15.525 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:15.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:15.578 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:47:15.578 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:15.578 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:47:15.579 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:15.585 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:47:15.585 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:15.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:15.602 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:47:15.602 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:15.618 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:47:15.618 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:15.635 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:47:15.635 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:15.651 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:47:15.651 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:15.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:15.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:15.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:15.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:15.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:15.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:15.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:16.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:16.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:16.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:16 smithi154 ceph-mon[102314]: pgmap v459: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 644 op/s 2024-09-13T14:47:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4130094327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:16.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:16 smithi154 ceph-mon[107038]: pgmap v459: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 644 op/s 2024-09-13T14:47:16.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4130094327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:16.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:16.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:16.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:16.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:16.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:16.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:16 smithi172 ceph-mon[100696]: pgmap v459: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 644 op/s 2024-09-13T14:47:16.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4130094327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:16.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:16.601 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:16.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:16.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:16.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:16.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:16.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:16.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:16.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:17.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:17.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:17.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:17.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:17.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:17.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:17.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:17.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:17.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:17.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:17.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:17.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:17.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:17.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:17.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:17.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:18.016 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:18.050 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:18.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:18.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:18.189 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:18.215 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:18.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:18.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:18.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:18 smithi154 ceph-mon[102314]: pgmap v460: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 104 op/s 2024-09-13T14:47:18.353 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:18 smithi154 ceph-mon[107038]: pgmap v460: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 104 op/s 2024-09-13T14:47:18.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:18.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:18.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:18 smithi172 ceph-mon[100696]: pgmap v460: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 104 op/s 2024-09-13T14:47:18.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:18.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:18.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:18.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:18.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:18.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:18.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:18.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:18.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:19.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:19.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:19.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:19.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:19.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:19.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:19.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:19.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:19.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:19.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:19.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:19.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:19.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:19.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:19.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:19.837 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:19.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:19.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:19.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:20.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:20.088 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:20.146 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T14:47:20.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.212 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:20 smithi154 ceph-mon[107038]: pgmap v461: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 80 op/s 2024-09-13T14:47:20.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2680585458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:47:20.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:47:20.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:20 smithi154 ceph-mon[102314]: pgmap v461: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 80 op/s 2024-09-13T14:47:20.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2680585458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:47:20.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:47:20.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:20 smithi172 ceph-mon[100696]: pgmap v461: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 80 op/s 2024-09-13T14:47:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2680585458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:47:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T14:47:20.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:20.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:20.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:20.710 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:20.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:20.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:20.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:20.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:21.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:21.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:21.158 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:47:21.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:21.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:21.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:21.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:21.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:21.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:21.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:47:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[102314]: osdmap e155: 8 total, 8 up, 7 in 2024-09-13T14:47:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1283344171' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:21.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:47:21.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[107038]: osdmap e155: 8 total, 8 up, 7 in 2024-09-13T14:47:21.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1283344171' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:21.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:21.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:21 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T14:47:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:21 smithi172 ceph-mon[100696]: osdmap e155: 8 total, 8 up, 7 in 2024-09-13T14:47:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1283344171' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:21 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:47:21.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:21.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:21.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:21.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:21.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.837 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:21.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:21.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:21.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:21.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:22.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:22.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:22.123 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:47:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:47:21] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-13T14:47:22.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:22.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:22.231 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:22.231 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:22.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:22.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:22.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:22.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[102314]: pgmap v463: 89 pgs: 89 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 54 op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[102314]: osdmap e156: 8 total, 8 up, 7 in 2024-09-13T14:47:22.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[107038]: pgmap v463: 89 pgs: 89 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 54 op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:22 smithi154 ceph-mon[107038]: osdmap e156: 8 total, 8 up, 7 in 2024-09-13T14:47:22.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:22 smithi172 ceph-mon[100696]: pgmap v463: 89 pgs: 89 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 54 op/s; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:22.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:22 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:22 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:47:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:22 smithi172 ceph-mon[100696]: osdmap e156: 8 total, 8 up, 7 in 2024-09-13T14:47:22.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:22.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:22.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:22.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:22.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:22.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:22.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:47:22.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:22.962 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:22.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:23.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:23.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:23.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:23.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:23.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:23.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:23.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:23.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:23.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:23.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:23.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:23.702 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:23.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:23.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:23.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:23.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:23.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:23.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:24.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:24.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:24.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:24.222 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:24.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:24.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:24.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:24 smithi154 ceph-mon[102314]: pgmap v465: 89 pgs: 89 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:24 smithi154 ceph-mon[107038]: pgmap v465: 89 pgs: 89 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:24.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.474 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:24.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:24.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:24 smithi172 ceph-mon[100696]: pgmap v465: 89 pgs: 89 active+clean; 4.6 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:24.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:24.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:47:24.601 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:24.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:24.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:24.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:24.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:24.862 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:24.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:24.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:24.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:25.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:25.168 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_0' removed 2024-09-13T14:47:25.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:25.238 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:25.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:25.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:25.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:47:25.390 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:25 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2115042674' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:47:25.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:25 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2115042674' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:47:25.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:25.485 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:25.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:25 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2115042674' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:47:25.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:25.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:25.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:25.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:25.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:25.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:25.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:25.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:25.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:26.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:26.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:26.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:26 smithi154 ceph-mon[102314]: pgmap v466: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2115042674' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:47:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:26 smithi154 ceph-mon[102314]: osdmap e157: 8 total, 8 up, 7 in 2024-09-13T14:47:26.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:26 smithi154 ceph-mon[107038]: pgmap v466: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:26.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2115042674' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:47:26.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:26 smithi154 ceph-mon[107038]: osdmap e157: 8 total, 8 up, 7 in 2024-09-13T14:47:26.221 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:26.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:26.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:26.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:26.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:26 smithi172 ceph-mon[100696]: pgmap v466: 89 pgs: 89 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2 B/s, 0 objects/s recovering 2024-09-13T14:47:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2115042674' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:47:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:26 smithi172 ceph-mon[100696]: osdmap e157: 8 total, 8 up, 7 in 2024-09-13T14:47:26.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:26.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:26.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:26.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:26.826 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:47:26.827 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:47:26.827 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-13T14:47:26.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-13T14:47:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:26.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:26.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:26.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:26.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:27.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:27.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:27.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:27.213 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:27.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:27.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:27.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:27.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:27.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:27.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:27.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:27.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:27.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:27.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:27.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:27.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:27.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:27.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:28.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:28.078 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:28.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:28.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:28 smithi154 ceph-mon[107038]: pgmap v468: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:28 smithi154 ceph-mon[102314]: pgmap v468: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:28.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:28.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:28.379 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:47:28.380 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:28.396 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:47:28.397 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:28.412 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:47:28.412 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:28.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:28.429 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:47:28.429 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:28.444 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:47:28.444 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:28.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:28.460 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:47:28.460 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:28.476 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:47:28.476 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:28 smithi172 ceph-mon[100696]: pgmap v468: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:28.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:28.492 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:47:28.492 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:28.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:28.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:28.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:28.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:28.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:28.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:28.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:28.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:28.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:29.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:29.065 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:29.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:29.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:29.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:29.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:29.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:29.427 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:29 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1547952666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:29.428 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:29 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1547952666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:29.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:29.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:29 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1547952666' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:29.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:29.559 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:29.566 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:29.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:29.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:29.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:29.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:29.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:29.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:29.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:30.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:30.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:30.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:30.179 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:30.272 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:30 smithi154 ceph-mon[102314]: pgmap v469: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:30.273 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:30 smithi154 ceph-mon[107038]: pgmap v469: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:30.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:30.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:30.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:30.430 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:30.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:30 smithi172 ceph-mon[100696]: pgmap v469: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:30.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:30.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:30.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:30.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:30.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:30.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:30.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:30.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:30.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:30.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:31.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:31.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:31.094 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:31.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:31.186 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:31.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:31.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:31.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:31.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:31.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:31.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:31.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:31.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:31.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:31.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:31.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:31.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:31.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:32.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:32.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:32.155 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:47:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:47:31] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-13T14:47:32.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:32.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:32.185 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:47:32.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:32.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:32.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:32.439 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:32 smithi154 ceph-mon[102314]: pgmap v470: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3396959693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:47:32.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:32 smithi154 ceph-mon[107038]: pgmap v470: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:32.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3396959693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:47:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:32 smithi172 ceph-mon[100696]: pgmap v470: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3396959693' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:47:32.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:32.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:32.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:32.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:32.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:32.810 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:47:32.810 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:47:32.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27212 5.a 5:58ee1d02:::benchmark_data_smithi154_144140_object27211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.91513835399999, 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17096689200000001, 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.811 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27212 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664198+0000", 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9800000000000003e-07 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664220+0000", 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1591999999999999e-05 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664227+0000", 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.712e-06 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664233+0000", 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8289999999999996e-06 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669218+0000", 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049845829999999999 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669320+0000", 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102304 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669434+0000", 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113581 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674379+0000", 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00494525 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674700+0000", 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032070699999999999 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.834993+0000", 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160292992 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835121+0000", 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012801800000000001 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835164+0000", 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3726000000000001e-05 2024-09-13T14:47:32.818 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27213 5.a 5:571e06b1:::benchmark_data_smithi154_144140_object27212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.91509432199999, 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17104000799999999, 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.819 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27213 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664242+0000", 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2e-07 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664265+0000", 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3476e-05 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664273+0000", 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2230000000000001e-06 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664278+0000", 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1599999999999997e-06 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.670614+0000", 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063359059999999997 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671024+0000", 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041025499999999999 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671138+0000", 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114418 2024-09-13T14:47:32.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674735+0000", 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035970080000000001 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674771+0000", 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5506999999999997e-05 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835177+0000", 2024-09-13T14:47:32.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16040633400000001 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835217+0000", 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9641000000000002e-05 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835281+0000", 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4659999999999994e-05 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.825 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27221 5.a 5:51807044:::benchmark_data_smithi154_144140_object27220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.91094076100001, 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16984976800000001, 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27221 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.826 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668396+0000", 2024-09-13T14:47:32.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9999999999999997e-07 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668416+0000", 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0516000000000001e-05 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668424+0000", 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6389999999999992e-06 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668430+0000", 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9939999999999997e-06 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672751+0000", 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043213569999999996 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673077+0000", 2024-09-13T14:47:32.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032614299999999998 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673198+0000", 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120492 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833603+0000", 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160404986 2024-09-13T14:47:32.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835418+0000", 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018152349999999999 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838152+0000", 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002733989 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838206+0000", 2024-09-13T14:47:32.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4165000000000001e-05 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838245+0000", 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8652000000000002e-05 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27226 5.a 5:58f709a9:::benchmark_data_smithi154_144140_object27225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:47:32.832 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.910180051, 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.169199034, 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27226 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1900000000000001e-07 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669187+0000", 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0571999999999998e-05 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669198+0000", 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0920999999999999e-05 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669205+0000", 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6479999999999997e-06 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674834+0000", 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056293519999999998 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674951+0000", 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011713 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675069+0000", 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117726 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835380+0000", 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160310762 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835455+0000", 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5813000000000001e-05 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838286+0000", 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028307559999999998 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838325+0000", 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8574e-05 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838355+0000", 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9961e-05 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.838 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27229 5.2 e117/111 5:47a3bf38:::benchmark_data_smithi154_144140_object27228:head v 117'1668, mlcod=117'1665)", 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.908104073, 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16704787199999999, 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:47:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5599999999999995e-07 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671265+0000", 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3231000000000001e-05 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671274+0000", 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9479999999999997e-06 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671280+0000", 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T14:47:32.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672510+0000", 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012303030000000001 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672591+0000", 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1227e-05 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838248+0000", 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16565628499999999 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838280+0000", 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1881999999999997e-05 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.843 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27212 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.90800410400001, 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16395237200000001, 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8700000000000001e-07 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671335+0000", 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.858e-06 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671342+0000", 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4189999999999996e-06 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671347+0000", 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9089999999999999e-06 2024-09-13T14:47:32.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674368+0000", 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003020371 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674374+0000", 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2829999999999998e-06 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835284+0000", 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160910145 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27231 5.d 5:bd1c1b1b:::benchmark_data_smithi154_144140_object27230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.90768510500001, 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16226318100000001, 2024-09-13T14:47:32.848 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27231 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671652+0000", 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0100000000000003e-07 2024-09-13T14:47:32.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671671+0000", 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9686000000000001e-05 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671680+0000", 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6989999999999992e-06 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671685+0000", 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2950000000000001e-06 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671954+0000", 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026847900000000002 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672066+0000", 2024-09-13T14:47:32.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011244 2024-09-13T14:47:32.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:47:32.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672249+0000", 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000182573 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674493+0000", 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022442439999999998 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674564+0000", 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1538000000000005e-05 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675871+0000", 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013069449999999999 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833838+0000", 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15796623000000001 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833914+0000", 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6150999999999996e-05 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.90426041699999, 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158535863, 2024-09-13T14:47:32.856 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675076+0000", 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5400000000000002e-07 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675079+0000", 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.365e-06 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675087+0000", 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6249999999999998e-06 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675093+0000", 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8300000000000001e-06 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833585+0000", 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158492884 2024-09-13T14:47:32.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833596+0000", 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0645999999999999e-05 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833611+0000", 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5058999999999999e-05 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27237 5.b 5:d9e8c993:::benchmark_data_smithi154_144140_object27236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.904251372, 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16640950199999999, 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27237 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675086+0000", 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.274e-06 2024-09-13T14:47:32.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675108+0000", 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2412000000000002e-05 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675120+0000", 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1729000000000001e-05 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675127+0000", 2024-09-13T14:47:32.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7680000000000003e-06 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675594+0000", 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000466992 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675717+0000", 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012363700000000001 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:47:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833850+0000", 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158133311 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836021+0000", 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021703880000000001 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836035+0000", 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3657e-05 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841412+0000", 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053777620000000003 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841458+0000", 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5235000000000003e-05 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841494+0000", 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6337000000000001e-05 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27239 5.2 e117/111 5:4246bfbf:::benchmark_data_smithi154_144140_object27238:head v 117'1669, mlcod=117'1665)", 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.903785494, 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16283244499999999, 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675551+0000", 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4000000000000001e-07 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675624+0000", 2024-09-13T14:47:32.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3522000000000005e-05 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675633+0000", 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0499999999999997e-06 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675639+0000", 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8839999999999999e-06 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833644+0000", 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158004476 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833663+0000", 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9238000000000001e-05 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838359+0000", 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046954900000000001 2024-09-13T14:47:32.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838383+0000", 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4145e-05 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27231 5.d e117/111 ondisk, result = 0)", 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.903645213, 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15823179200000001, 2024-09-13T14:47:32.871 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8500000000000001e-07 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675695+0000", 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9099999999999998e-06 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675705+0000", 2024-09-13T14:47:32.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6670000000000008e-06 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675710+0000", 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7810000000000002e-06 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675848+0000", 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137677 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675855+0000", 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1790000000000002e-06 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833922+0000", 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15806699299999999 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.875 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27241 5.6 5:6b00c174:::benchmark_data_smithi154_144140_object27240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.90359767999999, 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16225515099999999, 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27241 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.876 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675739+0000", 2024-09-13T14:47:32.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6499999999999999e-07 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675758+0000", 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8799000000000001e-05 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675766+0000", 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4649999999999994e-06 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675772+0000", 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6470000000000003e-06 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675823+0000", 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1010000000000001e-05 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675943+0000", 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120389 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833416+0000", 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15747296699999999 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835725+0000", 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023092400000000002 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835762+0000", 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6693000000000003e-05 2024-09-13T14:47:32.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837162+0000", 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001400152 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837949+0000", 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000787417 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837993+0000", 2024-09-13T14:47:32.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3707e-05 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27240 5.e e117/111 5:77f483c1:::benchmark_data_smithi154_144140_object27239:head v 117'1698, mlcod=117'1696)", 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.903586152, 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16227988300000001, 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675755+0000", 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5079999999999996e-06 2024-09-13T14:47:32.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675810+0000", 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.452e-05 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675818+0000", 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8659999999999995e-06 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675824+0000", 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7930000000000003e-06 2024-09-13T14:47:32.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675915+0000", 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0556000000000005e-05 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675932+0000", 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6979000000000001e-05 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837997+0000", 2024-09-13T14:47:32.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16206567699999999 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838029+0000", 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1983999999999997e-05 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27226 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:47:32.886 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.90244651899999, 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15849907399999999, 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676890+0000", 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7599999999999997e-07 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676894+0000", 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1590000000000003e-06 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833403+0000", 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.156509065 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833418+0000", 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5142e-05 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835366+0000", 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019481570000000001 2024-09-13T14:47:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835374+0000", 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5639999999999999e-06 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835388+0000", 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4411e-05 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 243.90237531599999, 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15846376300000001, 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:47:32.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676961+0000", 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4199999999999995e-07 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676965+0000", 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1130000000000003e-06 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833821+0000", 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15685534100000001 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833830+0000", 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1130000000000007e-06 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835407+0000", 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015777499999999999 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835413+0000", 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835424+0000", 2024-09-13T14:47:32.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0829000000000001e-05 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:117172 5.a 5:5df8026a:::benchmark_data_smithi154_144140_object6819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 154.41742219400001, 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15741727699999999, 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.895 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 117172 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161915+0000", 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.314e-06 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161919+0000", 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6569999999999996e-06 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161933+0000", 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3055e-05 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161940+0000", 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.593e-06 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164296+0000", 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023560709999999999 2024-09-13T14:47:32.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164406+0000", 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011003899999999999 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164510+0000", 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010334400000000001 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.175396+0000", 2024-09-13T14:47:32.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010886682 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179579+0000", 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041824269999999998 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319249+0000", 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13966985800000001 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319294+0000", 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5141e-05 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319331+0000", 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7095999999999999e-05 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.901 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:117176 5.a 5:574be98d:::benchmark_data_smithi154_144140_object6823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 154.41722568899999, 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15802838399999999, 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 117176 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162111+0000", 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5499999999999997e-07 2024-09-13T14:47:32.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162115+0000", 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6679999999999999e-06 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162124+0000", 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3700000000000001e-06 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162130+0000", 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.789e-06 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171194+0000", 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090644290000000006 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171302+0000", 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108194 2024-09-13T14:47:32.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171403+0000", 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100468 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179555+0000", 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008152365 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179585+0000", 2024-09-13T14:47:32.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0063e-05 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320052+0000", 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14046650699999999 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320103+0000", 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0924000000000002e-05 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320138+0000", 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5751999999999999e-05 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:117185 5.f e132/130 5:f308a48e:::benchmark_data_smithi154_144140_object6832:head v 132'7432, mlcod=125'5196)", 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 154.40452332300001, 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.151216923, 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.908 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174813+0000", 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7100000000000001e-07 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174820+0000", 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8739999999999997e-06 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174828+0000", 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4130000000000006e-06 2024-09-13T14:47:32.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174834+0000", 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8370000000000002e-06 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319258+0000", 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14442416499999999 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319282+0000", 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3923000000000001e-05 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.325925+0000", 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066427910000000003 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.326029+0000", 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104249 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.912 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:117851 5.e e133/111 5:700ea90e:::benchmark_data_smithi154_144140_object7498:head v 133'7344, mlcod=133'7342)", 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 153.48270529300001, 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.26475042900000001, 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096632+0000", 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.158e-06 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096640+0000", 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4800000000000001e-06 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219022+0000", 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.122381897 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219039+0000", 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6616000000000002e-05 2024-09-13T14:47:32.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219381+0000", 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000342443 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219401+0000", 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9698e-05 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.361334+0000", 2024-09-13T14:47:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.141933647 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.361381+0000", 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6489999999999997e-05 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:197886 5.d 5:ba09dd65:::benchmark_data_smithi154_144140_object87533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:47:32.917 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 51.566448254000001, 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15809542200000001, 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 197886 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:47:32.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012889+0000", 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2810000000000001e-06 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:47:32.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012893+0000", 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.771e-06 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012906+0000", 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.329e-05 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012913+0000", 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3089999999999998e-06 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012969+0000", 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.013085+0000", 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115708 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.013194+0000", 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010938300000000001 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.018482+0000", 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052877269999999999 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.125895+0000", 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.107413177 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170889+0000", 2024-09-13T14:47:32.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.044993742000000003 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170944+0000", 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4966999999999997e-05 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170983+0000", 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8781e-05 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:47:32.924 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:47:32.924 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:47:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:32.925 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:32.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:33.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:33.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:33.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:33.178 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:33.280 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:33.306 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:33.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:33.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:33.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3396959693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:47:33.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:33 smithi154 ceph-mon[102314]: osdmap e158: 8 total, 8 up, 7 in 2024-09-13T14:47:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3396959693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:47:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:33 smithi154 ceph-mon[107038]: osdmap e158: 8 total, 8 up, 7 in 2024-09-13T14:47:33.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3396959693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:47:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:33 smithi172 ceph-mon[100696]: osdmap e158: 8 total, 8 up, 7 in 2024-09-13T14:47:33.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:33.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:33.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:47:33.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:33.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:33.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:33.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:33.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:33.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:33.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:34.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:34.074 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:34.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:34.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:34.202 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:34.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:34 smithi154 ceph-mon[102314]: pgmap v472: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:34.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:34 smithi154 ceph-mon[107038]: pgmap v472: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:34.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:34.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:34.455 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:34.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:34 smithi172 ceph-mon[100696]: pgmap v472: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:34.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:34.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:34.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:34.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:34.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:34.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:34.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:34.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:34.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:35.012 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:47:35.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:47:35.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:35.090 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:35.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:35.213 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:35.235 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:35.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:35.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:35.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:35.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:35.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:35.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:35.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:35.712 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:35.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:35.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:35.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:35.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:35.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:36.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:36.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:36.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:36.199 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:47:36.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:36.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:36.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:36 smithi154 ceph-mon[102314]: pgmap v473: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:36.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/563458362' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:36.341 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:36 smithi154 ceph-mon[107038]: pgmap v473: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:36.341 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/563458362' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:36.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:36.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:36.475 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:36.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:36.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:36 smithi172 ceph-mon[100696]: pgmap v473: 73 pgs: 73 active+clean; 4.6 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:36.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/563458362' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:47:36.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:36.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:36.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:36.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:36.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:36.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:36.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:36.975 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:36.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:37.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:37.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:37.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:37.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:37.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:37.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:37.442 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:37.442 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:47:37.442 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/563458362' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:37.442 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[102314]: osdmap e159: 8 total, 8 up, 7 in 2024-09-13T14:47:37.443 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:37.443 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:47:37.443 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/563458362' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:37.443 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:37 smithi154 ceph-mon[107038]: osdmap e159: 8 total, 8 up, 7 in 2024-09-13T14:47:37.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:37.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:37 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:47:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:37 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:47:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:37 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/563458362' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:47:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:37 smithi172 ceph-mon[100696]: osdmap e159: 8 total, 8 up, 7 in 2024-09-13T14:47:37.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:37.596 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:37.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:37.721 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:37.738 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:37.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:37.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:37.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:37.975 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:37.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:38.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:38.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:47:38.102 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:38.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.210 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_1' created 2024-09-13T14:47:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.232 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:38.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:38.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:38.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:38.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:38.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:38 smithi154 ceph-mon[102314]: pgmap v475: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:38.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/209278016' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:47:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:38 smithi154 ceph-mon[107038]: pgmap v475: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:38.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/209278016' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:47:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.487 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:38.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:38.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:38 smithi172 ceph-mon[100696]: pgmap v475: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:38.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/209278016' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-13T14:47:38.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:38.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:38.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:38.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:38.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:38.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:38.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:38.985 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:38.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:39.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:39.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:39.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:47:39.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:39.239 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:39.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:39.366 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:39.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:39.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:39 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/209278016' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:47:39.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:39 smithi154 ceph-mon[102314]: osdmap e160: 8 total, 8 up, 7 in 2024-09-13T14:47:39.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:39 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/209278016' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:47:39.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:39 smithi154 ceph-mon[107038]: osdmap e160: 8 total, 8 up, 7 in 2024-09-13T14:47:39.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:39 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/209278016' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-13T14:47:39.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:39 smithi172 ceph-mon[100696]: osdmap e160: 8 total, 8 up, 7 in 2024-09-13T14:47:39.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:39.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:39.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:39.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-13T14:47:39.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:39.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:39.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:39.872 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:39.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:39.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:40.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:40.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:40.223 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:47:40.224 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:47:40.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:40.256 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:40.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:40.384 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:40.458 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:40 smithi154 ceph-mon[102314]: pgmap v477: 89 pgs: 1 creating+peering, 15 unknown, 73 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:40.459 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:40 smithi154 ceph-mon[102314]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:47:40.459 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:40 smithi154 ceph-mon[102314]: osdmap e161: 8 total, 8 up, 7 in 2024-09-13T14:47:40.459 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:40 smithi154 ceph-mon[107038]: pgmap v477: 89 pgs: 1 creating+peering, 15 unknown, 73 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:40.459 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:40 smithi154 ceph-mon[107038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:47:40.459 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:40 smithi154 ceph-mon[107038]: osdmap e161: 8 total, 8 up, 7 in 2024-09-13T14:47:40.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:40 smithi172 ceph-mon[100696]: pgmap v477: 89 pgs: 1 creating+peering, 15 unknown, 73 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:40.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:40 smithi172 ceph-mon[100696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:47:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:40 smithi172 ceph-mon[100696]: osdmap e161: 8 total, 8 up, 7 in 2024-09-13T14:47:40.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:40.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:40.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:40.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:40.759 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:40.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:40.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:40.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:40.947 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:47:40.947 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:47:40.947 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T14:47:40.947 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:47:40.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:41.010 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:41.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:41.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:41.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:41.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:41.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:41.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:41.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:41.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:41 smithi172 ceph-mon[100696]: from='client.34881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:41 smithi172 ceph-mon[100696]: osdmap e162: 8 total, 8 up, 7 in 2024-09-13T14:47:41.513 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:41.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:41.617 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:41 smithi154 ceph-mon[102314]: from='client.34881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:41.617 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:41 smithi154 ceph-mon[102314]: osdmap e162: 8 total, 8 up, 7 in 2024-09-13T14:47:41.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:41 smithi154 ceph-mon[107038]: from='client.34881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:41.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:41 smithi154 ceph-mon[107038]: osdmap e162: 8 total, 8 up, 7 in 2024-09-13T14:47:41.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:47:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:41.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:41.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:41.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:41.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:41.886 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:41.887 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:41.887 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:41.888 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:41.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:41.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:42.016 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:42.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:42.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:47:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:47:41] "GET /metrics HTTP/1.1" 200 34157 "" "Prometheus/2.51.0" 2024-09-13T14:47:42.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:42.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:42.366 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-13T14:47:39.217027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6333,"num_read_kb":13788,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":2963387,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":240,"acting":240,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":224,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1402596,"kb_used_data":399688,"kb_used_omap":32,"kb_used_meta":1002783,"kb_avail":654813468,"statfs":{"total":671965249536,"available":670528991232,"internally_reserved":0,"allocated":409280512,"data_stored":403730641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33228,"internal_metadata":1026850356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.323754"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":174,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187523+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:47:32.187523+0000","last_peered":"2024-09-13T14:47:32.187523+0000","last_clean":"2024-09-13T14:47:32.187523+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:47:32.187523+0000","last_undegraded":"2024-09-13T14:47:32.187523+0000","last_fullsized":"2024-09-13T14:47:32.187523+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361321+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:47:37.361321+0000","last_peered":"2024-09-13T14:47:37.361321+0000","last_clean":"2024-09-13T14:47:37.361321+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:47:37.361321+0000","last_undegraded":"2024-09-13T14:47:37.361321+0000","last_fullsized":"2024-09-13T14:47:37.361321+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":139,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361552+0000","last_change":"2024-09-13T14:44:54.719939+0000","last_active":"2024-09-13T14:47:37.361552+0000","last_peered":"2024-09-13T14:47:37.361552+0000","last_clean":"2024-09-13T14:47:37.361552+0000","last_became_active":"2024-09-13T14:44:54.719577+0000","last_became_peered":"2024-09-13T14:44:54.719577+0000","last_unstale":"2024-09-13T14:47:37.361552+0000","last_undegraded":"2024-09-13T14:47:37.361552+0000","last_fullsized":"2024-09-13T14:47:37.361552+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:03.642987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":166,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266287+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:47:37.266287+0000","last_peered":"2024-09-13T14:47:37.266287+0000","last_clean":"2024-09-13T14:47:37.266287+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:47:37.266287+0000","last_undegraded":"2024-09-13T14:47:37.266287+0000","last_fullsized":"2024-09-13T14:47:37.266287+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":166,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266506+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:47:37.266506+0000","last_peered":"2024-09-13T14:47:37.266506+0000","last_clean":"2024-09-13T14:47:37.266506+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:47:37.266506+0000","last_undegraded":"2024-09-13T14:47:37.266506+0000","last_fullsized":"2024-09-13T14:47:37.266506+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":127,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361300+0000","last_change":"2024-09-13T14:44:54.725347+0000","last_active":"2024-09-13T14:47:37.361300+0000","last_peered":"2024-09-13T14:47:37.361300+0000","last_clean":"2024-09-13T14:47:37.361300+0000","last_became_active":"2024-09-13T14:44:54.724452+0000","last_became_peered":"2024-09-13T14:44:54.724452+0000","last_unstale":"2024-09-13T14:47:37.361300+0000","last_undegraded":"2024-09-13T14:47:37.361300+0000","last_fullsized":"2024-09-13T14:47:37.361300+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:13.635099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":174,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187940+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:47:32.187940+0000","last_peered":"2024-09-13T14:47:32.187940+0000","last_clean":"2024-09-13T14:47:32.187940+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:47:32.187940+0000","last_undegraded":"2024-09-13T14:47:32.187940+0000","last_fullsized":"2024-09-13T14:47:32.187940+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":51,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.227976+0000","last_change":"2024-09-13T14:44:54.836457+0000","last_active":"2024-09-13T14:47:37.227976+0000","last_peered":"2024-09-13T14:47:37.227976+0000","last_clean":"2024-09-13T14:47:37.227976+0000","last_became_active":"2024-09-13T14:44:54.721186+0000","last_became_peered":"2024-09-13T14:44:54.721186+0000","last_unstale":"2024-09-13T14:47:37.227976+0000","last_undegraded":"2024-09-13T14:47:37.227976+0000","last_fullsized":"2024-09-13T14:47:37.227976+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:49.281864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":160,"state":"creating+peering","last_fresh":"2024-09-13T14:47:38.218851+0000","last_change":"2024-09-13T14:47:38.218851+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.218851+0000","last_undegraded":"2024-09-13T14:47:38.218851+0000","last_fullsized":"2024-09-13T14:47:38.218851+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:15.899344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":138,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.362039+0000","last_change":"2024-09-13T14:47:14.132191+0000","last_active":"2024-09-13T14:47:37.362039+0000","last_peered":"2024-09-13T14:47:37.362039+0000","last_clean":"2024-09-13T14:47:37.362039+0000","last_became_active":"2024-09-13T14:47:14.129611+0000","last_became_peered":"2024-09-13T14:47:14.129611+0000","last_unstale":"2024-09-13T14:47:37.362039+0000","last_undegraded":"2024-09-13T14:47:37.362039+0000","last_fullsized":"2024-09-13T14:47:37.362039+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:54.945230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":80,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361366+0000","last_change":"2024-09-13T14:44:54.720112+0000","last_active":"2024-09-13T14:47:37.361366+0000","last_peered":"2024-09-13T14:47:37.361366+0000","last_clean":"2024-09-13T14:47:37.361366+0000","last_became_active":"2024-09-13T14:44:54.719784+0000","last_became_peered":"2024-09-13T14:44:54.719784+0000","last_unstale":"2024-09-13T14:47:37.361366+0000","last_undegraded":"2024-09-13T14:47:37.361366+0000","last_fullsized":"2024-09-13T14:47:37.361366+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:33.432164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":126,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361965+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:47:37.361965+0000","last_peered":"2024-09-13T14:47:37.361965+0000","last_clean":"2024-09-13T14:47:37.361965+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:47:37.361965+0000","last_undegraded":"2024-09-13T14:47:37.361965+0000","last_fullsized":"2024-09-13T14:47:37.361965+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":146,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361251+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:47:37.361251+0000","last_peered":"2024-09-13T14:47:37.361251+0000","last_clean":"2024-09-13T14:47:37.361251+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:47:37.361251+0000","last_undegraded":"2024-09-13T14:47:37.361251+0000","last_fullsized":"2024-09-13T14:47:37.361251+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":160,"state":"creating+peering","last_fresh":"2024-09-13T14:47:38.217971+0000","last_change":"2024-09-13T14:47:38.217971+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.217971+0000","last_undegraded":"2024-09-13T14:47:38.217971+0000","last_fullsized":"2024-09-13T14:47:38.217971+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:27.140676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":149,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361712+0000","last_change":"2024-09-13T14:44:54.828261+0000","last_active":"2024-09-13T14:47:37.361712+0000","last_peered":"2024-09-13T14:47:37.361712+0000","last_clean":"2024-09-13T14:47:37.361712+0000","last_became_active":"2024-09-13T14:44:54.719757+0000","last_became_peered":"2024-09-13T14:44:54.719757+0000","last_unstale":"2024-09-13T14:47:37.361712+0000","last_undegraded":"2024-09-13T14:47:37.361712+0000","last_fullsized":"2024-09-13T14:47:37.361712+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:00.058836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":41,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.228030+0000","last_change":"2024-09-13T14:44:54.728690+0000","last_active":"2024-09-13T14:47:37.228030+0000","last_peered":"2024-09-13T14:47:37.228030+0000","last_clean":"2024-09-13T14:47:37.228030+0000","last_became_active":"2024-09-13T14:44:54.728349+0000","last_became_peered":"2024-09-13T14:44:54.728349+0000","last_unstale":"2024-09-13T14:47:37.228030+0000","last_undegraded":"2024-09-13T14:47:37.228030+0000","last_fullsized":"2024-09-13T14:47:37.228030+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:40.554955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":181,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187659+0000","last_change":"2024-09-13T14:44:54.716717+0000","last_active":"2024-09-13T14:47:32.187659+0000","last_peered":"2024-09-13T14:47:32.187659+0000","last_clean":"2024-09-13T14:47:32.187659+0000","last_became_active":"2024-09-13T14:44:54.716382+0000","last_became_peered":"2024-09-13T14:44:54.716382+0000","last_unstale":"2024-09-13T14:47:32.187659+0000","last_undegraded":"2024-09-13T14:47:32.187659+0000","last_fullsized":"2024-09-13T14:47:32.187659+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:16.968408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":166,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266456+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:47:37.266456+0000","last_peered":"2024-09-13T14:47:37.266456+0000","last_clean":"2024-09-13T14:47:37.266456+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:47:37.266456+0000","last_undegraded":"2024-09-13T14:47:37.266456+0000","last_fullsized":"2024-09-13T14:47:37.266456+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":160,"state":"creating+peering","last_fresh":"2024-09-13T14:47:38.215043+0000","last_change":"2024-09-13T14:47:38.215043+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.215043+0000","last_undegraded":"2024-09-13T14:47:38.215043+0000","last_fullsized":"2024-09-13T14:47:38.215043+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:51.711980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":172,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187485+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:47:32.187485+0000","last_peered":"2024-09-13T14:47:32.187485+0000","last_clean":"2024-09-13T14:47:32.187485+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:47:32.187485+0000","last_undegraded":"2024-09-13T14:47:32.187485+0000","last_fullsized":"2024-09-13T14:47:32.187485+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":140,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361388+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:47:37.361388+0000","last_peered":"2024-09-13T14:47:37.361388+0000","last_clean":"2024-09-13T14:47:37.361388+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:47:37.361388+0000","last_undegraded":"2024-09-13T14:47:37.361388+0000","last_fullsized":"2024-09-13T14:47:37.361388+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":160,"state":"creating+peering","last_fresh":"2024-09-13T14:47:38.216251+0000","last_change":"2024-09-13T14:47:38.216251+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.216251+0000","last_undegraded":"2024-09-13T14:47:38.216251+0000","last_fullsized":"2024-09-13T14:47:38.216251+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":41,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.228128+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:47:37.228128+0000","last_peered":"2024-09-13T14:47:37.228128+0000","last_clean":"2024-09-13T14:47:37.228128+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:47:37.228128+0000","last_undegraded":"2024-09-13T14:47:37.228128+0000","last_fullsized":"2024-09-13T14:47:37.228128+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":137,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361782+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:47:37.361782+0000","last_peered":"2024-09-13T14:47:37.361782+0000","last_clean":"2024-09-13T14:47:37.361782+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:47:37.361782+0000","last_undegraded":"2024-09-13T14:47:37.361782+0000","last_fullsized":"2024-09-13T14:47:37.361782+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":123,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361279+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:47:37.361279+0000","last_peered":"2024-09-13T14:47:37.361279+0000","last_clean":"2024-09-13T14:47:37.361279+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:47:37.361279+0000","last_undegraded":"2024-09-13T14:47:37.361279+0000","last_fullsized":"2024-09-13T14:47:37.361279+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":140,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361836+0000","last_change":"2024-09-13T14:44:54.718721+0000","last_active":"2024-09-13T14:47:37.361836+0000","last_peered":"2024-09-13T14:47:37.361836+0000","last_clean":"2024-09-13T14:47:37.361836+0000","last_became_active":"2024-09-13T14:44:54.717559+0000","last_became_peered":"2024-09-13T14:44:54.717559+0000","last_unstale":"2024-09-13T14:47:37.361836+0000","last_undegraded":"2024-09-13T14:47:37.361836+0000","last_fullsized":"2024-09-13T14:47:37.361836+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:01.660819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":175,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.188079+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:47:32.188079+0000","last_peered":"2024-09-13T14:47:32.188079+0000","last_clean":"2024-09-13T14:47:32.188079+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:47:32.188079+0000","last_undegraded":"2024-09-13T14:47:32.188079+0000","last_fullsized":"2024-09-13T14:47:32.188079+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T14:47:38.217658+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:47:38.217658+0000","last_peered":"2024-09-13T14:47:38.217658+0000","last_clean":"2024-09-13T14:47:38.217658+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:47:38.217658+0000","last_undegraded":"2024-09-13T14:47:38.217658+0000","last_fullsized":"2024-09-13T14:47:38.217658+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":501,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361459+0000","last_change":"2024-09-13T14:46:58.058606+0000","last_active":"2024-09-13T14:47:37.361459+0000","last_peered":"2024-09-13T14:47:37.361459+0000","last_clean":"2024-09-13T14:47:37.361459+0000","last_became_active":"2024-09-13T14:46:58.058149+0000","last_became_peered":"2024-09-13T14:46:58.058149+0000","last_unstale":"2024-09-13T14:47:37.361459+0000","last_undegraded":"2024-09-13T14:47:37.361459+0000","last_fullsized":"2024-09-13T14:47:37.361459+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:16.382630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":682,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T14:47:38.217228+0000","last_change":"2024-09-13T14:43:12.684107+0000","last_active":"2024-09-13T14:47:38.217228+0000","last_peered":"2024-09-13T14:47:38.217228+0000","last_clean":"2024-09-13T14:47:38.217228+0000","last_became_active":"2024-09-13T14:43:12.683522+0000","last_became_peered":"2024-09-13T14:43:12.683522+0000","last_unstale":"2024-09-13T14:47:38.217228+0000","last_undegraded":"2024-09-13T14:47:38.217228+0000","last_fullsized":"2024-09-13T14:47:38.217228+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:52.396314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":144,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187324+0000","last_change":"2024-09-13T14:44:54.717238+0000","last_active":"2024-09-13T14:47:32.187324+0000","last_peered":"2024-09-13T14:47:32.187324+0000","last_clean":"2024-09-13T14:47:32.187324+0000","last_became_active":"2024-09-13T14:44:54.716732+0000","last_became_peered":"2024-09-13T14:44:54.716732+0000","last_unstale":"2024-09-13T14:47:32.187324+0000","last_undegraded":"2024-09-13T14:47:32.187324+0000","last_fullsized":"2024-09-13T14:47:32.187324+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:17.542085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":41,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.228249+0000","last_change":"2024-09-13T14:44:54.728995+0000","last_active":"2024-09-13T14:47:37.228249+0000","last_peered":"2024-09-13T14:47:37.228249+0000","last_clean":"2024-09-13T14:47:37.228249+0000","last_became_active":"2024-09-13T14:44:54.728737+0000","last_became_peered":"2024-09-13T14:44:54.728737+0000","last_unstale":"2024-09-13T14:47:37.228249+0000","last_undegraded":"2024-09-13T14:47:37.228249+0000","last_fullsized":"2024-09-13T14:47:37.228249+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:10.038052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":314,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361411+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:47:37.361411+0000","last_peered":"2024-09-13T14:47:37.361411+0000","last_clean":"2024-09-13T14:47:37.361411+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:47:37.361411+0000","last_undegraded":"2024-09-13T14:47:37.361411+0000","last_fullsized":"2024-09-13T14:47:37.361411+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":166,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266158+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:47:37.266158+0000","last_peered":"2024-09-13T14:47:37.266158+0000","last_clean":"2024-09-13T14:47:37.266158+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:47:37.266158+0000","last_undegraded":"2024-09-13T14:47:37.266158+0000","last_fullsized":"2024-09-13T14:47:37.266158+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":177,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266592+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:47:37.266592+0000","last_peered":"2024-09-13T14:47:37.266592+0000","last_clean":"2024-09-13T14:47:37.266592+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:47:37.266592+0000","last_undegraded":"2024-09-13T14:47:37.266592+0000","last_fullsized":"2024-09-13T14:47:37.266592+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":160,"state":"creating+peering","last_fresh":"2024-09-13T14:47:38.218085+0000","last_change":"2024-09-13T14:47:38.218085+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.218085+0000","last_undegraded":"2024-09-13T14:47:38.218085+0000","last_fullsized":"2024-09-13T14:47:38.218085+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:17.904674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":732,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T14:47:38.217128+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:47:38.217128+0000","last_peered":"2024-09-13T14:47:38.217128+0000","last_clean":"2024-09-13T14:47:38.217128+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:47:38.217128+0000","last_undegraded":"2024-09-13T14:47:38.217128+0000","last_fullsized":"2024-09-13T14:47:38.217128+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":169,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.380776+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:47:33.380776+0000","last_peered":"2024-09-13T14:47:33.380776+0000","last_clean":"2024-09-13T14:47:33.380776+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:47:33.380776+0000","last_undegraded":"2024-09-13T14:47:33.380776+0000","last_fullsized":"2024-09-13T14:47:33.380776+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":188,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T14:47:38.217956+0000","last_change":"2024-09-13T14:43:09.679993+0000","last_active":"2024-09-13T14:47:38.217956+0000","last_peered":"2024-09-13T14:47:38.217956+0000","last_clean":"2024-09-13T14:47:38.217956+0000","last_became_active":"2024-09-13T14:43:09.679699+0000","last_became_peered":"2024-09-13T14:43:09.679699+0000","last_unstale":"2024-09-13T14:47:38.217956+0000","last_undegraded":"2024-09-13T14:47:38.217956+0000","last_fullsized":"2024-09-13T14:47:38.217956+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:20.969934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":797,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.380806+0000","last_change":"2024-09-13T14:46:16.880474+0000","last_active":"2024-09-13T14:47:33.380806+0000","last_peered":"2024-09-13T14:47:33.380806+0000","last_clean":"2024-09-13T14:47:33.380806+0000","last_became_active":"2024-09-13T14:44:54.729425+0000","last_became_peered":"2024-09-13T14:44:54.729425+0000","last_unstale":"2024-09-13T14:47:33.380806+0000","last_undegraded":"2024-09-13T14:47:33.380806+0000","last_fullsized":"2024-09-13T14:47:33.380806+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:20.841594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":142,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361580+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:47:37.361580+0000","last_peered":"2024-09-13T14:47:37.361580+0000","last_clean":"2024-09-13T14:47:37.361580+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:47:37.361580+0000","last_undegraded":"2024-09-13T14:47:37.361580+0000","last_fullsized":"2024-09-13T14:47:37.361580+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":166,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.380965+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:47:33.380965+0000","last_peered":"2024-09-13T14:47:33.380965+0000","last_clean":"2024-09-13T14:47:33.380965+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:47:33.380965+0000","last_undegraded":"2024-09-13T14:47:33.380965+0000","last_fullsized":"2024-09-13T14:47:33.380965+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":198,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187852+0000","last_change":"2024-09-13T14:44:54.716747+0000","last_active":"2024-09-13T14:47:32.187852+0000","last_peered":"2024-09-13T14:47:32.187852+0000","last_clean":"2024-09-13T14:47:32.187852+0000","last_became_active":"2024-09-13T14:44:54.716438+0000","last_became_peered":"2024-09-13T14:44:54.716438+0000","last_unstale":"2024-09-13T14:47:32.187852+0000","last_undegraded":"2024-09-13T14:47:32.187852+0000","last_fullsized":"2024-09-13T14:47:32.187852+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:21.007581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":694,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.380544+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:47:33.380544+0000","last_peered":"2024-09-13T14:47:33.380544+0000","last_clean":"2024-09-13T14:47:33.380544+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:47:33.380544+0000","last_undegraded":"2024-09-13T14:47:33.380544+0000","last_fullsized":"2024-09-13T14:47:33.380544+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":151,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.380941+0000","last_change":"2024-09-13T14:43:09.681228+0000","last_active":"2024-09-13T14:47:33.380941+0000","last_peered":"2024-09-13T14:47:33.380941+0000","last_clean":"2024-09-13T14:47:33.380941+0000","last_became_active":"2024-09-13T14:43:09.680908+0000","last_became_peered":"2024-09-13T14:43:09.680908+0000","last_unstale":"2024-09-13T14:47:33.380941+0000","last_undegraded":"2024-09-13T14:47:33.380941+0000","last_fullsized":"2024-09-13T14:47:33.380941+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:33.151302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T14:47:38.199052+0000","last_change":"2024-09-13T14:47:38.199052+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.199052+0000","last_undegraded":"2024-09-13T14:47:38.199052+0000","last_fullsized":"2024-09-13T14:47:38.199052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":162,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.381018+0000","last_change":"2024-09-13T14:43:09.680063+0000","last_active":"2024-09-13T14:47:33.381018+0000","last_peered":"2024-09-13T14:47:33.381018+0000","last_clean":"2024-09-13T14:47:33.381018+0000","last_became_active":"2024-09-13T14:43:09.679740+0000","last_became_peered":"2024-09-13T14:43:09.679740+0000","last_unstale":"2024-09-13T14:47:33.381018+0000","last_undegraded":"2024-09-13T14:47:33.381018+0000","last_fullsized":"2024-09-13T14:47:33.381018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:55.701172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":474,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361795+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:47:37.361795+0000","last_peered":"2024-09-13T14:47:37.361795+0000","last_clean":"2024-09-13T14:47:37.361795+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:47:37.361795+0000","last_undegraded":"2024-09-13T14:47:37.361795+0000","last_fullsized":"2024-09-13T14:47:37.361795+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":162,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266391+0000","last_change":"2024-09-13T14:43:09.682650+0000","last_active":"2024-09-13T14:47:37.266391+0000","last_peered":"2024-09-13T14:47:37.266391+0000","last_clean":"2024-09-13T14:47:37.266391+0000","last_became_active":"2024-09-13T14:43:09.682335+0000","last_became_peered":"2024-09-13T14:43:09.682335+0000","last_unstale":"2024-09-13T14:47:37.266391+0000","last_undegraded":"2024-09-13T14:47:37.266391+0000","last_fullsized":"2024-09-13T14:47:37.266391+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:26.113631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":160,"state":"creating+peering","last_fresh":"2024-09-13T14:47:38.215046+0000","last_change":"2024-09-13T14:47:38.215046+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.215046+0000","last_undegraded":"2024-09-13T14:47:38.215046+0000","last_fullsized":"2024-09-13T14:47:38.215046+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:46.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":172,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.188138+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:47:32.188138+0000","last_peered":"2024-09-13T14:47:32.188138+0000","last_clean":"2024-09-13T14:47:32.188138+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:47:32.188138+0000","last_undegraded":"2024-09-13T14:47:32.188138+0000","last_fullsized":"2024-09-13T14:47:32.188138+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":626,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.381214+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:47:33.381214+0000","last_peered":"2024-09-13T14:47:33.381214+0000","last_clean":"2024-09-13T14:47:33.381214+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:47:33.381214+0000","last_undegraded":"2024-09-13T14:47:33.381214+0000","last_fullsized":"2024-09-13T14:47:33.381214+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":169,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187994+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:47:32.187994+0000","last_peered":"2024-09-13T14:47:32.187994+0000","last_clean":"2024-09-13T14:47:32.187994+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:47:32.187994+0000","last_undegraded":"2024-09-13T14:47:32.187994+0000","last_fullsized":"2024-09-13T14:47:32.187994+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":160,"state":"creating+peering","last_fresh":"2024-09-13T14:47:38.218003+0000","last_change":"2024-09-13T14:47:38.218003+0000","last_active":"2024-09-13T14:47:38.199052+0000","last_peered":"2024-09-13T14:47:38.199052+0000","last_clean":"2024-09-13T14:47:38.199052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T14:47:38.218003+0000","last_undegraded":"2024-09-13T14:47:38.218003+0000","last_fullsized":"2024-09-13T14:47:38.218003+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:41.520707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":171,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266455+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:47:37.266455+0000","last_peered":"2024-09-13T14:47:37.266455+0000","last_clean":"2024-09-13T14:47:37.266455+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:47:37.266455+0000","last_undegraded":"2024-09-13T14:47:37.266455+0000","last_fullsized":"2024-09-13T14:47:37.266455+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":656,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.228321+0000","last_change":"2024-09-13T14:46:18.008986+0000","last_active":"2024-09-13T14:47:37.228321+0000","last_peered":"2024-09-13T14:47:37.228321+0000","last_clean":"2024-09-13T14:47:37.228321+0000","last_became_active":"2024-09-13T14:46:18.008512+0000","last_became_peered":"2024-09-13T14:46:18.008512+0000","last_unstale":"2024-09-13T14:47:37.228321+0000","last_undegraded":"2024-09-13T14:47:37.228321+0000","last_fullsized":"2024-09-13T14:47:37.228321+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:41.601601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":135,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361178+0000","last_change":"2024-09-13T14:44:54.718379+0000","last_active":"2024-09-13T14:47:37.361178+0000","last_peered":"2024-09-13T14:47:37.361178+0000","last_clean":"2024-09-13T14:47:37.361178+0000","last_became_active":"2024-09-13T14:44:54.717339+0000","last_became_peered":"2024-09-13T14:44:54.717339+0000","last_unstale":"2024-09-13T14:47:37.361178+0000","last_undegraded":"2024-09-13T14:47:37.361178+0000","last_fullsized":"2024-09-13T14:47:37.361178+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:34.326913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":140,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361483+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:47:37.361483+0000","last_peered":"2024-09-13T14:47:37.361483+0000","last_clean":"2024-09-13T14:47:37.361483+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:47:37.361483+0000","last_undegraded":"2024-09-13T14:47:37.361483+0000","last_fullsized":"2024-09-13T14:47:37.361483+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":169,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361703+0000","last_change":"2024-09-13T14:44:54.826556+0000","last_active":"2024-09-13T14:47:37.361703+0000","last_peered":"2024-09-13T14:47:37.361703+0000","last_clean":"2024-09-13T14:47:37.361703+0000","last_became_active":"2024-09-13T14:44:54.718081+0000","last_became_peered":"2024-09-13T14:44:54.718081+0000","last_unstale":"2024-09-13T14:47:37.361703+0000","last_undegraded":"2024-09-13T14:47:37.361703+0000","last_fullsized":"2024-09-13T14:47:37.361703+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:58.858037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":169,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.380889+0000","last_change":"2024-09-13T14:44:54.715785+0000","last_active":"2024-09-13T14:47:33.380889+0000","last_peered":"2024-09-13T14:47:33.380889+0000","last_clean":"2024-09-13T14:47:33.380889+0000","last_became_active":"2024-09-13T14:44:54.715454+0000","last_became_peered":"2024-09-13T14:44:54.715454+0000","last_unstale":"2024-09-13T14:47:33.380889+0000","last_undegraded":"2024-09-13T14:47:33.380889+0000","last_fullsized":"2024-09-13T14:47:33.380889+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:18.225794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":127,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361769+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:47:37.361769+0000","last_peered":"2024-09-13T14:47:37.361769+0000","last_clean":"2024-09-13T14:47:37.361769+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:47:37.361769+0000","last_undegraded":"2024-09-13T14:47:37.361769+0000","last_fullsized":"2024-09-13T14:47:37.361769+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":112,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361689+0000","last_change":"2024-09-13T14:43:09.682269+0000","last_active":"2024-09-13T14:47:37.361689+0000","last_peered":"2024-09-13T14:47:37.361689+0000","last_clean":"2024-09-13T14:47:37.361689+0000","last_became_active":"2024-09-13T14:43:09.681735+0000","last_became_peered":"2024-09-13T14:43:09.681735+0000","last_unstale":"2024-09-13T14:47:37.361689+0000","last_undegraded":"2024-09-13T14:47:37.361689+0000","last_fullsized":"2024-09-13T14:47:37.361689+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:48.513102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":137,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187342+0000","last_change":"2024-09-13T14:44:54.725385+0000","last_active":"2024-09-13T14:47:32.187342+0000","last_peered":"2024-09-13T14:47:32.187342+0000","last_clean":"2024-09-13T14:47:32.187342+0000","last_became_active":"2024-09-13T14:44:54.725140+0000","last_became_peered":"2024-09-13T14:44:54.725140+0000","last_unstale":"2024-09-13T14:47:32.187342+0000","last_undegraded":"2024-09-13T14:47:32.187342+0000","last_fullsized":"2024-09-13T14:47:32.187342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:58.246065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":187,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266517+0000","last_change":"2024-09-13T14:44:54.718815+0000","last_active":"2024-09-13T14:47:37.266517+0000","last_peered":"2024-09-13T14:47:37.266517+0000","last_clean":"2024-09-13T14:47:37.266517+0000","last_became_active":"2024-09-13T14:44:54.718516+0000","last_became_peered":"2024-09-13T14:44:54.718516+0000","last_unstale":"2024-09-13T14:47:37.266517+0000","last_undegraded":"2024-09-13T14:47:37.266517+0000","last_fullsized":"2024-09-13T14:47:37.266517+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:31.898033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":188,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187269+0000","last_change":"2024-09-13T14:44:54.718935+0000","last_active":"2024-09-13T14:47:32.187269+0000","last_peered":"2024-09-13T14:47:32.187269+0000","last_clean":"2024-09-13T14:47:32.187269+0000","last_became_active":"2024-09-13T14:44:54.718584+0000","last_became_peered":"2024-09-13T14:44:54.718584+0000","last_unstale":"2024-09-13T14:47:32.187269+0000","last_undegraded":"2024-09-13T14:47:32.187269+0000","last_fullsized":"2024-09-13T14:47:32.187269+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:58.902926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":41,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.228308+0000","last_change":"2024-09-13T14:44:54.721244+0000","last_active":"2024-09-13T14:47:37.228308+0000","last_peered":"2024-09-13T14:47:37.228308+0000","last_clean":"2024-09-13T14:47:37.228308+0000","last_became_active":"2024-09-13T14:44:54.720944+0000","last_became_peered":"2024-09-13T14:44:54.720944+0000","last_unstale":"2024-09-13T14:47:37.228308+0000","last_undegraded":"2024-09-13T14:47:37.228308+0000","last_fullsized":"2024-09-13T14:47:37.228308+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:18.635223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":190,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.188019+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:47:32.188019+0000","last_peered":"2024-09-13T14:47:32.188019+0000","last_clean":"2024-09-13T14:47:32.188019+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:47:32.188019+0000","last_undegraded":"2024-09-13T14:47:32.188019+0000","last_fullsized":"2024-09-13T14:47:32.188019+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":166,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266569+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:47:37.266569+0000","last_peered":"2024-09-13T14:47:37.266569+0000","last_clean":"2024-09-13T14:47:37.266569+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:47:37.266569+0000","last_undegraded":"2024-09-13T14:47:37.266569+0000","last_fullsized":"2024-09-13T14:47:37.266569+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":154,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:33.380764+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:47:33.380764+0000","last_peered":"2024-09-13T14:47:33.380764+0000","last_clean":"2024-09-13T14:47:33.380764+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:47:33.380764+0000","last_undegraded":"2024-09-13T14:47:33.380764+0000","last_fullsized":"2024-09-13T14:47:33.380764+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":173,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187761+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:47:32.187761+0000","last_peered":"2024-09-13T14:47:32.187761+0000","last_clean":"2024-09-13T14:47:32.187761+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:47:32.187761+0000","last_undegraded":"2024-09-13T14:47:32.187761+0000","last_fullsized":"2024-09-13T14:47:32.187761+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":138,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.362083+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:47:37.362083+0000","last_peered":"2024-09-13T14:47:37.362083+0000","last_clean":"2024-09-13T14:47:37.362083+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:47:37.362083+0000","last_undegraded":"2024-09-13T14:47:37.362083+0000","last_fullsized":"2024-09-13T14:47:37.362083+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T14:47:38.216829+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:47:38.216829+0000","last_peered":"2024-09-13T14:47:38.216829+0000","last_clean":"2024-09-13T14:47:38.216829+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:47:38.216829+0000","last_undegraded":"2024-09-13T14:47:38.216829+0000","last_fullsized":"2024-09-13T14:47:38.216829+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":160,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266346+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:47:37.266346+0000","last_peered":"2024-09-13T14:47:37.266346+0000","last_clean":"2024-09-13T14:47:37.266346+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:47:37.266346+0000","last_undegraded":"2024-09-13T14:47:37.266346+0000","last_fullsized":"2024-09-13T14:47:37.266346+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":139,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361482+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:47:37.361482+0000","last_peered":"2024-09-13T14:47:37.361482+0000","last_clean":"2024-09-13T14:47:37.361482+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:47:37.361482+0000","last_undegraded":"2024-09-13T14:47:37.361482+0000","last_fullsized":"2024-09-13T14:47:37.361482+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":102,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361792+0000","last_change":"2024-09-13T14:44:54.830007+0000","last_active":"2024-09-13T14:47:37.361792+0000","last_peered":"2024-09-13T14:47:37.361792+0000","last_clean":"2024-09-13T14:47:37.361792+0000","last_became_active":"2024-09-13T14:44:54.720757+0000","last_became_peered":"2024-09-13T14:44:54.720757+0000","last_unstale":"2024-09-13T14:47:37.361792+0000","last_undegraded":"2024-09-13T14:47:37.361792+0000","last_fullsized":"2024-09-13T14:47:37.361792+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:31.423552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":136,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.362011+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:47:37.362011+0000","last_peered":"2024-09-13T14:47:37.362011+0000","last_clean":"2024-09-13T14:47:37.362011+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:47:37.362011+0000","last_undegraded":"2024-09-13T14:47:37.362011+0000","last_fullsized":"2024-09-13T14:47:37.362011+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":102,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.361843+0000","last_change":"2024-09-13T14:44:54.834271+0000","last_active":"2024-09-13T14:47:37.361843+0000","last_peered":"2024-09-13T14:47:37.361843+0000","last_clean":"2024-09-13T14:47:37.361843+0000","last_became_active":"2024-09-13T14:44:54.721681+0000","last_became_peered":"2024-09-13T14:44:54.721681+0000","last_unstale":"2024-09-13T14:47:37.361843+0000","last_undegraded":"2024-09-13T14:47:37.361843+0000","last_fullsized":"2024-09-13T14:47:37.361843+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:59.844315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":193,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187596+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:47:32.187596+0000","last_peered":"2024-09-13T14:47:32.187596+0000","last_clean":"2024-09-13T14:47:32.187596+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:47:32.187596+0000","last_undegraded":"2024-09-13T14:47:32.187596+0000","last_fullsized":"2024-09-13T14:47:32.187596+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":195,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T14:47:32.187938+0000","last_change":"2024-09-13T14:44:54.716788+0000","last_active":"2024-09-13T14:47:32.187938+0000","last_peered":"2024-09-13T14:47:32.187938+0000","last_clean":"2024-09-13T14:47:32.187938+0000","last_became_active":"2024-09-13T14:44:54.716462+0000","last_became_peered":"2024-09-13T14:44:54.716462+0000","last_unstale":"2024-09-13T14:47:32.187938+0000","last_undegraded":"2024-09-13T14:47:32.187938+0000","last_fullsized":"2024-09-13T14:47:32.187938+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:11.645196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":140,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-13T14:47:38.213180+0000","last_change":"2024-09-13T14:44:54.713470+0000","last_active":"2024-09-13T14:47:38.213180+0000","last_peered":"2024-09-13T14:47:38.213180+0000","last_clean":"2024-09-13T14:47:38.213180+0000","last_became_active":"2024-09-13T14:44:54.713122+0000","last_became_peered":"2024-09-13T14:44:54.713122+0000","last_unstale":"2024-09-13T14:47:38.213180+0000","last_undegraded":"2024-09-13T14:47:38.213180+0000","last_fullsized":"2024-09-13T14:47:38.213180+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:57.220599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":210,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T14:47:37.266518+0000","last_change":"2024-09-13T14:44:54.821762+0000","last_active":"2024-09-13T14:47:37.266518+0000","last_peered":"2024-09-13T14:47:37.266518+0000","last_clean":"2024-09-13T14:47:37.266518+0000","last_became_active":"2024-09-13T14:44:54.721169+0000","last_became_peered":"2024-09-13T14:44:54.721169+0000","last_unstale":"2024-09-13T14:47:37.266518+0000","last_undegraded":"2024-09-13T14:47:37.266518+0000","last_fullsized":"2024-09-13T14:47:37.266518+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:26.845873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":21,"acting":21,"num_store_stats":4},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":11490,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12582988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234243,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245352,"kb_used_data":115800,"kb_used_omap":1,"kb_used_meta":129534,"kb_avail":93499800,"statfs":{"total":95995035648,"available":95743795200,"internally_reserved":0,"allocated":118579200,"data_stored":117772824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":132643275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495879,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119632,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93625520,"statfs":{"total":95995035648,"available":95872532480,"internally_reserved":0,"allocated":2428928,"data_stored":1634598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757516,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019154,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181408,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":93563744,"statfs":{"total":95995035648,"available":95809273856,"internally_reserved":0,"allocated":1851392,"data_stored":1052903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183892422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468041,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99960,"kb_used_data":5868,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":93645192,"statfs":{"total":95995035648,"available":95892676608,"internally_reserved":0,"allocated":6008832,"data_stored":5225991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":96332964},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631568,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":168084,"kb_used_data":5896,"kb_used_omap":8,"kb_used_meta":162167,"kb_avail":93577068,"statfs":{"total":95995035648,"available":95822917632,"internally_reserved":0,"allocated":6037504,"data_stored":5242977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":166059999},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795094,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397300,"kb_used_data":261480,"kb_used_omap":7,"kb_used_meta":135800,"kb_avail":93347852,"statfs":{"total":95995035648,"available":95588200448,"internally_reserved":0,"allocated":267755520,"data_stored":266976634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":139059826},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958621,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190860,"kb_used_data":6464,"kb_used_omap":7,"kb_used_meta":184376,"kb_avail":93554292,"statfs":{"total":95995035648,"available":95799595008,"internally_reserved":0,"allocated":6619136,"data_stored":5824714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":188801511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:47:42.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:47:42.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:42.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:42.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:42 smithi172 ceph-mon[100696]: pgmap v480: 89 pgs: 7 creating+peering, 2 unknown, 80 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:42.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:42.519 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:42.570 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:42 smithi154 ceph-mon[102314]: pgmap v480: 89 pgs: 7 creating+peering, 2 unknown, 80 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:42.570 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:42 smithi154 ceph-mon[107038]: pgmap v480: 89 pgs: 7 creating+peering, 2 unknown, 80 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:42.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:42.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:42.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:42.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:42.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:42.893 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:42.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:42.949 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:42.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:43.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:43.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:43.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:43.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:43.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:43.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:43.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:43.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:43.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:43.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:43.638 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:43.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:43.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:43.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:43.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:43.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:43.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:43.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:44.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:44.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:44.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:44.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:44.145 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:43 smithi154 ceph-mon[102314]: pgmap v481: 89 pgs: 7 creating+peering, 82 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:44.145 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:44.145 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:44.146 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:43 smithi154 ceph-mon[107038]: pgmap v481: 89 pgs: 7 creating+peering, 82 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:44.146 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:44.146 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:43 smithi172 ceph-mon[100696]: pgmap v481: 89 pgs: 7 creating+peering, 82 active+clean; 4.6 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:47:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:44.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:44.268 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:44.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:44.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:44.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:44.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:44.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:44.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:44.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:44.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:44.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:44.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:44.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:44.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:45.019 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:45.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:45.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:45.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:45.266 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:45.366 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:45.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:45.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:45.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:45.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:45.619 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:45.645 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:45.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:45.768 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:45.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:45.896 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:45.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:45.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:46.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:46.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:46.134 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:45 smithi154 ceph-mon[102314]: pgmap v482: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:46.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:45 smithi154 ceph-mon[107038]: pgmap v482: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:46.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:45 smithi172 ceph-mon[100696]: pgmap v482: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:46.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:46.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:46.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:46.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:46.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:46.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:46.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:46.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:46.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:46.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:46.808 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:46.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:46.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:46.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:46.995 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:47:46.995 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:47:46.995 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:47:46.995 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:47:46.995 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:47:47.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:47.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:47.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:47.184 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:47.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:47.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:47.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:47.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:47.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:47.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:47.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:47.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:47:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:47.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:47.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:47.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:47.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:47.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:47.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:48.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:48.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:48.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:48.192 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:47 smithi154 ceph-mon[102314]: pgmap v483: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:48.192 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:47 smithi154 ceph-mon[107038]: pgmap v483: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:48.195 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:47 smithi172 ceph-mon[100696]: pgmap v483: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:48.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:48.322 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:48.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:48.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:48.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:48.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:48.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:48.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:48.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:48.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:48.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:48.948 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:48.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:48.976 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:47:49.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.077 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:49.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:49.107 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3651675562' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:49.107 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:48 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:49.107 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3651675562' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:49.107 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:48 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:49.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:49.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:49.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3651675562' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:49.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:48 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:47:49.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:49.329 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:49.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:49.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:49.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:49.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:49.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:49.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:49.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:49.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:49.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:49.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:49.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:49.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:50.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:50.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:50.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:50.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[102314]: pgmap v484: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[102314]: osdmap e163: 8 total, 8 up, 7 in 2024-09-13T14:47:50.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[107038]: pgmap v484: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:50.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:49 smithi154 ceph-mon[107038]: osdmap e163: 8 total, 8 up, 7 in 2024-09-13T14:47:50.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:49 smithi172 ceph-mon[100696]: pgmap v484: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:50.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:49 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:47:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:49 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:47:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:49 smithi172 ceph-mon[100696]: osdmap e163: 8 total, 8 up, 7 in 2024-09-13T14:47:50.272 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:47:50.273 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:47:50.289 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:47:50.290 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:47:50.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:50.305 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:47:50.306 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:47:50.321 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:47:50.322 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:47:50.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:50.338 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:47:50.338 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:47:50.355 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:47:50.355 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:47:50.371 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:47:50.371 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:47:50.388 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:47:50.388 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:47:50.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:50.455 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:50.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:50.578 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:50.648 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:47:50.649 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:47:50.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:50.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:50.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:50.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:50.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:50.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:50.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:51.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:51.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:51.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:51.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:51.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/548462109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:51.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:50 smithi172 ceph-mon[100696]: from='client.34899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:51.296 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/548462109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:51.296 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:50 smithi154 ceph-mon[102314]: from='client.34899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:51.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/548462109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:47:51.296 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:50 smithi154 ceph-mon[107038]: from='client.34899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:47:51.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:51.331 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:51.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:51.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:51.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:51.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:51.677 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-13T14:47:48.972140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6333,"num_read_kb":13788,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":2963387,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":271,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1339800,"kb_used_data":332728,"kb_used_omap":32,"kb_used_meta":1006943,"kb_avail":654876264,"statfs":{"total":671965249536,"available":670593294336,"internally_reserved":0,"allocated":340713472,"data_stored":335086186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33228,"internal_metadata":1031110196},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.753070"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":182,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291660+0000","last_change":"2024-09-13T14:41:41.488347+0000","last_active":"2024-09-13T14:47:41.291660+0000","last_peered":"2024-09-13T14:47:41.291660+0000","last_clean":"2024-09-13T14:47:41.291660+0000","last_became_active":"2024-09-13T14:41:41.488054+0000","last_became_peered":"2024-09-13T14:41:41.488054+0000","last_unstale":"2024-09-13T14:47:41.291660+0000","last_undegraded":"2024-09-13T14:47:41.291660+0000","last_fullsized":"2024-09-13T14:47:41.291660+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:24.139665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.219735+0000","last_change":"2024-09-13T14:41:09.650027+0000","last_active":"2024-09-13T14:47:39.219735+0000","last_peered":"2024-09-13T14:47:39.219735+0000","last_clean":"2024-09-13T14:47:39.219735+0000","last_became_active":"2024-09-13T14:41:09.649713+0000","last_became_peered":"2024-09-13T14:41:09.649713+0000","last_unstale":"2024-09-13T14:47:39.219735+0000","last_undegraded":"2024-09-13T14:47:39.219735+0000","last_fullsized":"2024-09-13T14:47:39.219735+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:07.857736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":141,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.219745+0000","last_change":"2024-09-13T14:44:54.719939+0000","last_active":"2024-09-13T14:47:39.219745+0000","last_peered":"2024-09-13T14:47:39.219745+0000","last_clean":"2024-09-13T14:47:39.219745+0000","last_became_active":"2024-09-13T14:44:54.719577+0000","last_became_peered":"2024-09-13T14:44:54.719577+0000","last_unstale":"2024-09-13T14:47:39.219745+0000","last_undegraded":"2024-09-13T14:47:39.219745+0000","last_fullsized":"2024-09-13T14:47:39.219745+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:38:03.642987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":172,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766574+0000","last_change":"2024-09-13T14:41:41.491470+0000","last_active":"2024-09-13T14:47:40.766574+0000","last_peered":"2024-09-13T14:47:40.766574+0000","last_clean":"2024-09-13T14:47:40.766574+0000","last_became_active":"2024-09-13T14:41:41.491147+0000","last_became_peered":"2024-09-13T14:41:41.491147+0000","last_unstale":"2024-09-13T14:47:40.766574+0000","last_undegraded":"2024-09-13T14:47:40.766574+0000","last_fullsized":"2024-09-13T14:47:40.766574+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:50.931757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":172,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766805+0000","last_change":"2024-09-13T14:41:41.489614+0000","last_active":"2024-09-13T14:47:40.766805+0000","last_peered":"2024-09-13T14:47:40.766805+0000","last_clean":"2024-09-13T14:47:40.766805+0000","last_became_active":"2024-09-13T14:41:41.489345+0000","last_became_peered":"2024-09-13T14:41:41.489345+0000","last_unstale":"2024-09-13T14:47:40.766805+0000","last_undegraded":"2024-09-13T14:47:40.766805+0000","last_fullsized":"2024-09-13T14:47:40.766805+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:32.086719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":129,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.219329+0000","last_change":"2024-09-13T14:44:54.725347+0000","last_active":"2024-09-13T14:47:39.219329+0000","last_peered":"2024-09-13T14:47:39.219329+0000","last_clean":"2024-09-13T14:47:39.219329+0000","last_became_active":"2024-09-13T14:44:54.724452+0000","last_became_peered":"2024-09-13T14:44:54.724452+0000","last_unstale":"2024-09-13T14:47:39.219329+0000","last_undegraded":"2024-09-13T14:47:39.219329+0000","last_fullsized":"2024-09-13T14:47:39.219329+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:13.635099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":182,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291919+0000","last_change":"2024-09-13T14:41:41.485968+0000","last_active":"2024-09-13T14:47:41.291919+0000","last_peered":"2024-09-13T14:47:41.291919+0000","last_clean":"2024-09-13T14:47:41.291919+0000","last_became_active":"2024-09-13T14:41:41.485539+0000","last_became_peered":"2024-09-13T14:41:41.485539+0000","last_unstale":"2024-09-13T14:47:41.291919+0000","last_undegraded":"2024-09-13T14:47:41.291919+0000","last_fullsized":"2024-09-13T14:47:41.291919+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:53.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":53,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.223514+0000","last_change":"2024-09-13T14:44:54.836457+0000","last_active":"2024-09-13T14:47:39.223514+0000","last_peered":"2024-09-13T14:47:39.223514+0000","last_clean":"2024-09-13T14:47:39.223514+0000","last_became_active":"2024-09-13T14:44:54.721186+0000","last_became_peered":"2024-09-13T14:44:54.721186+0000","last_unstale":"2024-09-13T14:47:39.223514+0000","last_undegraded":"2024-09-13T14:47:39.223514+0000","last_fullsized":"2024-09-13T14:47:39.223514+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:49.281864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.227138+0000","last_change":"2024-09-13T14:47:39.226779+0000","last_active":"2024-09-13T14:47:39.227138+0000","last_peered":"2024-09-13T14:47:39.227138+0000","last_clean":"2024-09-13T14:47:39.227138+0000","last_became_active":"2024-09-13T14:47:39.226486+0000","last_became_peered":"2024-09-13T14:47:39.226486+0000","last_unstale":"2024-09-13T14:47:39.227138+0000","last_undegraded":"2024-09-13T14:47:39.227138+0000","last_fullsized":"2024-09-13T14:47:39.227138+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:15.899344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":140,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.220216+0000","last_change":"2024-09-13T14:47:14.132191+0000","last_active":"2024-09-13T14:47:39.220216+0000","last_peered":"2024-09-13T14:47:39.220216+0000","last_clean":"2024-09-13T14:47:39.220216+0000","last_became_active":"2024-09-13T14:47:14.129611+0000","last_became_peered":"2024-09-13T14:47:14.129611+0000","last_unstale":"2024-09-13T14:47:39.220216+0000","last_undegraded":"2024-09-13T14:47:39.220216+0000","last_fullsized":"2024-09-13T14:47:39.220216+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:54.945230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":82,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.218866+0000","last_change":"2024-09-13T14:44:54.720112+0000","last_active":"2024-09-13T14:47:39.218866+0000","last_peered":"2024-09-13T14:47:39.218866+0000","last_clean":"2024-09-13T14:47:39.218866+0000","last_became_active":"2024-09-13T14:44:54.719784+0000","last_became_peered":"2024-09-13T14:44:54.719784+0000","last_unstale":"2024-09-13T14:47:39.218866+0000","last_undegraded":"2024-09-13T14:47:39.218866+0000","last_fullsized":"2024-09-13T14:47:39.218866+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:33.432164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291370+0000","last_change":"2024-09-13T14:47:39.231248+0000","last_active":"2024-09-13T14:47:41.291370+0000","last_peered":"2024-09-13T14:47:41.291370+0000","last_clean":"2024-09-13T14:47:41.291370+0000","last_became_active":"2024-09-13T14:47:39.230931+0000","last_became_peered":"2024-09-13T14:47:39.230931+0000","last_unstale":"2024-09-13T14:47:41.291370+0000","last_undegraded":"2024-09-13T14:47:41.291370+0000","last_fullsized":"2024-09-13T14:47:41.291370+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:09.875585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.219889+0000","last_change":"2024-09-13T14:40:36.876005+0000","last_active":"2024-09-13T14:47:39.219889+0000","last_peered":"2024-09-13T14:47:39.219889+0000","last_clean":"2024-09-13T14:47:39.219889+0000","last_became_active":"2024-09-13T14:40:36.875658+0000","last_became_peered":"2024-09-13T14:40:36.875658+0000","last_unstale":"2024-09-13T14:47:39.219889+0000","last_undegraded":"2024-09-13T14:47:39.219889+0000","last_fullsized":"2024-09-13T14:47:39.219889+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:19.243640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":148,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222137+0000","last_change":"2024-09-13T14:41:41.493612+0000","last_active":"2024-09-13T14:47:39.222137+0000","last_peered":"2024-09-13T14:47:39.222137+0000","last_clean":"2024-09-13T14:47:39.222137+0000","last_became_active":"2024-09-13T14:41:41.493270+0000","last_became_peered":"2024-09-13T14:41:41.493270+0000","last_unstale":"2024-09-13T14:47:39.222137+0000","last_undegraded":"2024-09-13T14:47:39.222137+0000","last_fullsized":"2024-09-13T14:47:39.222137+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:29.961130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.230021+0000","last_change":"2024-09-13T14:47:39.229605+0000","last_active":"2024-09-13T14:47:39.230021+0000","last_peered":"2024-09-13T14:47:39.230021+0000","last_clean":"2024-09-13T14:47:39.230021+0000","last_became_active":"2024-09-13T14:47:39.229294+0000","last_became_peered":"2024-09-13T14:47:39.229294+0000","last_unstale":"2024-09-13T14:47:39.230021+0000","last_undegraded":"2024-09-13T14:47:39.230021+0000","last_fullsized":"2024-09-13T14:47:39.230021+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:38:27.140676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":151,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.220092+0000","last_change":"2024-09-13T14:44:54.828261+0000","last_active":"2024-09-13T14:47:39.220092+0000","last_peered":"2024-09-13T14:47:39.220092+0000","last_clean":"2024-09-13T14:47:39.220092+0000","last_became_active":"2024-09-13T14:44:54.719757+0000","last_became_peered":"2024-09-13T14:44:54.719757+0000","last_unstale":"2024-09-13T14:47:39.220092+0000","last_undegraded":"2024-09-13T14:47:39.220092+0000","last_fullsized":"2024-09-13T14:47:39.220092+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:00.058836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":43,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.224624+0000","last_change":"2024-09-13T14:44:54.728690+0000","last_active":"2024-09-13T14:47:39.224624+0000","last_peered":"2024-09-13T14:47:39.224624+0000","last_clean":"2024-09-13T14:47:39.224624+0000","last_became_active":"2024-09-13T14:44:54.728349+0000","last_became_peered":"2024-09-13T14:44:54.728349+0000","last_unstale":"2024-09-13T14:47:39.224624+0000","last_undegraded":"2024-09-13T14:47:39.224624+0000","last_fullsized":"2024-09-13T14:47:39.224624+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:40.554955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766602+0000","last_change":"2024-09-13T14:47:39.225263+0000","last_active":"2024-09-13T14:47:40.766602+0000","last_peered":"2024-09-13T14:47:40.766602+0000","last_clean":"2024-09-13T14:47:40.766602+0000","last_became_active":"2024-09-13T14:47:39.224964+0000","last_became_peered":"2024-09-13T14:47:39.224964+0000","last_unstale":"2024-09-13T14:47:40.766602+0000","last_undegraded":"2024-09-13T14:47:40.766602+0000","last_fullsized":"2024-09-13T14:47:40.766602+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:20.537415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":190,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291789+0000","last_change":"2024-09-13T14:44:54.716717+0000","last_active":"2024-09-13T14:47:41.291789+0000","last_peered":"2024-09-13T14:47:41.291789+0000","last_clean":"2024-09-13T14:47:41.291789+0000","last_became_active":"2024-09-13T14:44:54.716382+0000","last_became_peered":"2024-09-13T14:44:54.716382+0000","last_unstale":"2024-09-13T14:47:41.291789+0000","last_undegraded":"2024-09-13T14:47:41.291789+0000","last_fullsized":"2024-09-13T14:47:41.291789+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:16.968408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":172,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766851+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:47:40.766851+0000","last_peered":"2024-09-13T14:47:40.766851+0000","last_clean":"2024-09-13T14:47:40.766851+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:47:40.766851+0000","last_undegraded":"2024-09-13T14:47:40.766851+0000","last_fullsized":"2024-09-13T14:47:40.766851+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.231650+0000","last_change":"2024-09-13T14:47:39.231300+0000","last_active":"2024-09-13T14:47:39.231650+0000","last_peered":"2024-09-13T14:47:39.231650+0000","last_clean":"2024-09-13T14:47:39.231650+0000","last_became_active":"2024-09-13T14:47:39.230762+0000","last_became_peered":"2024-09-13T14:47:39.230762+0000","last_unstale":"2024-09-13T14:47:39.231650+0000","last_undegraded":"2024-09-13T14:47:39.231650+0000","last_fullsized":"2024-09-13T14:47:39.231650+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:51.711980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":180,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291657+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:47:41.291657+0000","last_peered":"2024-09-13T14:47:41.291657+0000","last_clean":"2024-09-13T14:47:41.291657+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:47:41.291657+0000","last_undegraded":"2024-09-13T14:47:41.291657+0000","last_fullsized":"2024-09-13T14:47:41.291657+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222037+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:47:39.222037+0000","last_peered":"2024-09-13T14:47:39.222037+0000","last_clean":"2024-09-13T14:47:39.222037+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:47:39.222037+0000","last_undegraded":"2024-09-13T14:47:39.222037+0000","last_fullsized":"2024-09-13T14:47:39.222037+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.220064+0000","last_change":"2024-09-13T14:47:39.230078+0000","last_active":"2024-09-13T14:47:40.220064+0000","last_peered":"2024-09-13T14:47:40.220064+0000","last_clean":"2024-09-13T14:47:40.220064+0000","last_became_active":"2024-09-13T14:47:39.229639+0000","last_became_peered":"2024-09-13T14:47:39.229639+0000","last_unstale":"2024-09-13T14:47:40.220064+0000","last_undegraded":"2024-09-13T14:47:40.220064+0000","last_fullsized":"2024-09-13T14:47:40.220064+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:47.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.224780+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:47:39.224780+0000","last_peered":"2024-09-13T14:47:39.224780+0000","last_clean":"2024-09-13T14:47:39.224780+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:47:39.224780+0000","last_undegraded":"2024-09-13T14:47:39.224780+0000","last_fullsized":"2024-09-13T14:47:39.224780+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":139,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222798+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:47:39.222798+0000","last_peered":"2024-09-13T14:47:39.222798+0000","last_clean":"2024-09-13T14:47:39.222798+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:47:39.222798+0000","last_undegraded":"2024-09-13T14:47:39.222798+0000","last_fullsized":"2024-09-13T14:47:39.222798+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291455+0000","last_change":"2024-09-13T14:47:39.229430+0000","last_active":"2024-09-13T14:47:41.291455+0000","last_peered":"2024-09-13T14:47:41.291455+0000","last_clean":"2024-09-13T14:47:41.291455+0000","last_became_active":"2024-09-13T14:47:39.229154+0000","last_became_peered":"2024-09-13T14:47:39.229154+0000","last_unstale":"2024-09-13T14:47:41.291455+0000","last_undegraded":"2024-09-13T14:47:41.291455+0000","last_fullsized":"2024-09-13T14:47:41.291455+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:03.773471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":125,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222268+0000","last_change":"2024-09-13T14:41:09.649685+0000","last_active":"2024-09-13T14:47:39.222268+0000","last_peered":"2024-09-13T14:47:39.222268+0000","last_clean":"2024-09-13T14:47:39.222268+0000","last_became_active":"2024-09-13T14:41:09.649383+0000","last_became_peered":"2024-09-13T14:41:09.649383+0000","last_unstale":"2024-09-13T14:47:39.222268+0000","last_undegraded":"2024-09-13T14:47:39.222268+0000","last_fullsized":"2024-09-13T14:47:39.222268+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:16:17.175981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222731+0000","last_change":"2024-09-13T14:44:54.718721+0000","last_active":"2024-09-13T14:47:39.222731+0000","last_peered":"2024-09-13T14:47:39.222731+0000","last_clean":"2024-09-13T14:47:39.222731+0000","last_became_active":"2024-09-13T14:44:54.717559+0000","last_became_peered":"2024-09-13T14:44:54.717559+0000","last_unstale":"2024-09-13T14:47:39.222731+0000","last_undegraded":"2024-09-13T14:47:39.222731+0000","last_fullsized":"2024-09-13T14:47:39.222731+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:01.660819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292234+0000","last_change":"2024-09-13T14:47:39.232495+0000","last_active":"2024-09-13T14:47:41.292234+0000","last_peered":"2024-09-13T14:47:41.292234+0000","last_clean":"2024-09-13T14:47:41.292234+0000","last_became_active":"2024-09-13T14:47:39.231965+0000","last_became_peered":"2024-09-13T14:47:39.231965+0000","last_unstale":"2024-09-13T14:47:41.292234+0000","last_undegraded":"2024-09-13T14:47:41.292234+0000","last_fullsized":"2024-09-13T14:47:41.292234+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:21.799558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":183,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291396+0000","last_change":"2024-09-13T14:41:41.488561+0000","last_active":"2024-09-13T14:47:41.291396+0000","last_peered":"2024-09-13T14:47:41.291396+0000","last_clean":"2024-09-13T14:47:41.291396+0000","last_became_active":"2024-09-13T14:41:41.487817+0000","last_became_peered":"2024-09-13T14:41:41.487817+0000","last_unstale":"2024-09-13T14:47:41.291396+0000","last_undegraded":"2024-09-13T14:47:41.291396+0000","last_fullsized":"2024-09-13T14:47:41.291396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:36.725234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.220083+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:47:40.220083+0000","last_peered":"2024-09-13T14:47:40.220083+0000","last_clean":"2024-09-13T14:47:40.220083+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:47:40.220083+0000","last_undegraded":"2024-09-13T14:47:40.220083+0000","last_fullsized":"2024-09-13T14:47:40.220083+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291845+0000","last_change":"2024-09-13T14:47:39.231717+0000","last_active":"2024-09-13T14:47:41.291845+0000","last_peered":"2024-09-13T14:47:41.291845+0000","last_clean":"2024-09-13T14:47:41.291845+0000","last_became_active":"2024-09-13T14:47:39.230669+0000","last_became_peered":"2024-09-13T14:47:39.230669+0000","last_unstale":"2024-09-13T14:47:41.291845+0000","last_undegraded":"2024-09-13T14:47:41.291845+0000","last_fullsized":"2024-09-13T14:47:41.291845+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:10.225604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":503,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.218970+0000","last_change":"2024-09-13T14:46:58.058606+0000","last_active":"2024-09-13T14:47:39.218970+0000","last_peered":"2024-09-13T14:47:39.218970+0000","last_clean":"2024-09-13T14:47:39.218970+0000","last_became_active":"2024-09-13T14:46:58.058149+0000","last_became_peered":"2024-09-13T14:46:58.058149+0000","last_unstale":"2024-09-13T14:47:39.218970+0000","last_undegraded":"2024-09-13T14:47:39.218970+0000","last_fullsized":"2024-09-13T14:47:39.218970+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:16.382630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":686,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.219797+0000","last_change":"2024-09-13T14:43:12.684107+0000","last_active":"2024-09-13T14:47:40.219797+0000","last_peered":"2024-09-13T14:47:40.219797+0000","last_clean":"2024-09-13T14:47:40.219797+0000","last_became_active":"2024-09-13T14:43:12.683522+0000","last_became_peered":"2024-09-13T14:43:12.683522+0000","last_unstale":"2024-09-13T14:47:40.219797+0000","last_undegraded":"2024-09-13T14:47:40.219797+0000","last_fullsized":"2024-09-13T14:47:40.219797+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:52.396314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":152,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291920+0000","last_change":"2024-09-13T14:44:54.717238+0000","last_active":"2024-09-13T14:47:41.291920+0000","last_peered":"2024-09-13T14:47:41.291920+0000","last_clean":"2024-09-13T14:47:41.291920+0000","last_became_active":"2024-09-13T14:44:54.716732+0000","last_became_peered":"2024-09-13T14:44:54.716732+0000","last_unstale":"2024-09-13T14:47:41.291920+0000","last_undegraded":"2024-09-13T14:47:41.291920+0000","last_fullsized":"2024-09-13T14:47:41.291920+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:17.542085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":43,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.224538+0000","last_change":"2024-09-13T14:44:54.728995+0000","last_active":"2024-09-13T14:47:39.224538+0000","last_peered":"2024-09-13T14:47:39.224538+0000","last_clean":"2024-09-13T14:47:39.224538+0000","last_became_active":"2024-09-13T14:44:54.728737+0000","last_became_peered":"2024-09-13T14:44:54.728737+0000","last_unstale":"2024-09-13T14:47:39.224538+0000","last_undegraded":"2024-09-13T14:47:39.224538+0000","last_fullsized":"2024-09-13T14:47:39.224538+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:10.038052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292037+0000","last_change":"2024-09-13T14:47:39.230238+0000","last_active":"2024-09-13T14:47:41.292037+0000","last_peered":"2024-09-13T14:47:41.292037+0000","last_clean":"2024-09-13T14:47:41.292037+0000","last_became_active":"2024-09-13T14:47:39.229822+0000","last_became_peered":"2024-09-13T14:47:39.229822+0000","last_unstale":"2024-09-13T14:47:41.292037+0000","last_undegraded":"2024-09-13T14:47:41.292037+0000","last_fullsized":"2024-09-13T14:47:41.292037+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:11.003106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":316,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.218880+0000","last_change":"2024-09-13T14:40:36.875534+0000","last_active":"2024-09-13T14:47:39.218880+0000","last_peered":"2024-09-13T14:47:39.218880+0000","last_clean":"2024-09-13T14:47:39.218880+0000","last_became_active":"2024-09-13T14:40:36.875059+0000","last_became_peered":"2024-09-13T14:40:36.875059+0000","last_unstale":"2024-09-13T14:47:39.218880+0000","last_undegraded":"2024-09-13T14:47:39.218880+0000","last_fullsized":"2024-09-13T14:47:39.218880+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:04.816530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":172,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766472+0000","last_change":"2024-09-13T14:41:41.492339+0000","last_active":"2024-09-13T14:47:40.766472+0000","last_peered":"2024-09-13T14:47:40.766472+0000","last_clean":"2024-09-13T14:47:40.766472+0000","last_became_active":"2024-09-13T14:41:41.492016+0000","last_became_peered":"2024-09-13T14:41:41.492016+0000","last_unstale":"2024-09-13T14:47:40.766472+0000","last_undegraded":"2024-09-13T14:47:40.766472+0000","last_fullsized":"2024-09-13T14:47:40.766472+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.868639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":183,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766928+0000","last_change":"2024-09-13T14:41:09.652903+0000","last_active":"2024-09-13T14:47:40.766928+0000","last_peered":"2024-09-13T14:47:40.766928+0000","last_clean":"2024-09-13T14:47:40.766928+0000","last_became_active":"2024-09-13T14:41:09.652600+0000","last_became_peered":"2024-09-13T14:41:09.652600+0000","last_unstale":"2024-09-13T14:47:40.766928+0000","last_undegraded":"2024-09-13T14:47:40.766928+0000","last_fullsized":"2024-09-13T14:47:40.766928+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:07.438633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.230170+0000","last_change":"2024-09-13T14:47:39.229736+0000","last_active":"2024-09-13T14:47:39.230170+0000","last_peered":"2024-09-13T14:47:39.230170+0000","last_clean":"2024-09-13T14:47:39.230170+0000","last_became_active":"2024-09-13T14:47:39.229377+0000","last_became_peered":"2024-09-13T14:47:39.229377+0000","last_unstale":"2024-09-13T14:47:39.230170+0000","last_undegraded":"2024-09-13T14:47:39.230170+0000","last_fullsized":"2024-09-13T14:47:39.230170+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:17.904674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":736,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.219665+0000","last_change":"2024-09-13T14:41:41.492611+0000","last_active":"2024-09-13T14:47:40.219665+0000","last_peered":"2024-09-13T14:47:40.219665+0000","last_clean":"2024-09-13T14:47:40.219665+0000","last_became_active":"2024-09-13T14:41:41.491974+0000","last_became_peered":"2024-09-13T14:41:41.491974+0000","last_unstale":"2024-09-13T14:47:40.219665+0000","last_undegraded":"2024-09-13T14:47:40.219665+0000","last_fullsized":"2024-09-13T14:47:40.219665+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:57.395359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":177,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291974+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:47:41.291974+0000","last_peered":"2024-09-13T14:47:41.291974+0000","last_clean":"2024-09-13T14:47:41.291974+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:47:41.291974+0000","last_undegraded":"2024-09-13T14:47:41.291974+0000","last_fullsized":"2024-09-13T14:47:41.291974+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":192,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.220200+0000","last_change":"2024-09-13T14:43:09.679993+0000","last_active":"2024-09-13T14:47:40.220200+0000","last_peered":"2024-09-13T14:47:40.220200+0000","last_clean":"2024-09-13T14:47:40.220200+0000","last_became_active":"2024-09-13T14:43:09.679699+0000","last_became_peered":"2024-09-13T14:43:09.679699+0000","last_unstale":"2024-09-13T14:47:40.220200+0000","last_undegraded":"2024-09-13T14:47:40.220200+0000","last_fullsized":"2024-09-13T14:47:40.220200+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:20.969934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292439+0000","last_change":"2024-09-13T14:47:39.229099+0000","last_active":"2024-09-13T14:47:41.292439+0000","last_peered":"2024-09-13T14:47:41.292439+0000","last_clean":"2024-09-13T14:47:41.292439+0000","last_became_active":"2024-09-13T14:47:39.228749+0000","last_became_peered":"2024-09-13T14:47:39.228749+0000","last_unstale":"2024-09-13T14:47:41.292439+0000","last_undegraded":"2024-09-13T14:47:41.292439+0000","last_fullsized":"2024-09-13T14:47:41.292439+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:53:59.897392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":805,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292031+0000","last_change":"2024-09-13T14:46:16.880474+0000","last_active":"2024-09-13T14:47:41.292031+0000","last_peered":"2024-09-13T14:47:41.292031+0000","last_clean":"2024-09-13T14:47:41.292031+0000","last_became_active":"2024-09-13T14:44:54.729425+0000","last_became_peered":"2024-09-13T14:44:54.729425+0000","last_unstale":"2024-09-13T14:47:41.292031+0000","last_undegraded":"2024-09-13T14:47:41.292031+0000","last_fullsized":"2024-09-13T14:47:41.292031+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:20.841594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":144,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222466+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:47:39.222466+0000","last_peered":"2024-09-13T14:47:39.222466+0000","last_clean":"2024-09-13T14:47:39.222466+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:47:39.222466+0000","last_undegraded":"2024-09-13T14:47:39.222466+0000","last_fullsized":"2024-09-13T14:47:39.222466+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":174,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292190+0000","last_change":"2024-09-13T14:41:41.483817+0000","last_active":"2024-09-13T14:47:41.292190+0000","last_peered":"2024-09-13T14:47:41.292190+0000","last_clean":"2024-09-13T14:47:41.292190+0000","last_became_active":"2024-09-13T14:41:41.483352+0000","last_became_peered":"2024-09-13T14:41:41.483352+0000","last_unstale":"2024-09-13T14:47:41.292190+0000","last_undegraded":"2024-09-13T14:47:41.292190+0000","last_fullsized":"2024-09-13T14:47:41.292190+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:16.566572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766530+0000","last_change":"2024-09-13T14:47:39.228950+0000","last_active":"2024-09-13T14:47:40.766530+0000","last_peered":"2024-09-13T14:47:40.766530+0000","last_clean":"2024-09-13T14:47:40.766530+0000","last_became_active":"2024-09-13T14:47:39.228610+0000","last_became_peered":"2024-09-13T14:47:39.228610+0000","last_unstale":"2024-09-13T14:47:40.766530+0000","last_undegraded":"2024-09-13T14:47:40.766530+0000","last_fullsized":"2024-09-13T14:47:40.766530+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:06.094445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":206,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292281+0000","last_change":"2024-09-13T14:44:54.716747+0000","last_active":"2024-09-13T14:47:41.292281+0000","last_peered":"2024-09-13T14:47:41.292281+0000","last_clean":"2024-09-13T14:47:41.292281+0000","last_became_active":"2024-09-13T14:44:54.716438+0000","last_became_peered":"2024-09-13T14:44:54.716438+0000","last_unstale":"2024-09-13T14:47:41.292281+0000","last_undegraded":"2024-09-13T14:47:41.292281+0000","last_fullsized":"2024-09-13T14:47:41.292281+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:21.007581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":702,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291091+0000","last_change":"2024-09-13T14:40:36.879813+0000","last_active":"2024-09-13T14:47:41.291091+0000","last_peered":"2024-09-13T14:47:41.291091+0000","last_clean":"2024-09-13T14:47:41.291091+0000","last_became_active":"2024-09-13T14:40:36.879193+0000","last_became_peered":"2024-09-13T14:40:36.879193+0000","last_unstale":"2024-09-13T14:47:41.291091+0000","last_undegraded":"2024-09-13T14:47:41.291091+0000","last_fullsized":"2024-09-13T14:47:41.291091+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:59.354479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":159,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292268+0000","last_change":"2024-09-13T14:43:09.681228+0000","last_active":"2024-09-13T14:47:41.292268+0000","last_peered":"2024-09-13T14:47:41.292268+0000","last_clean":"2024-09-13T14:47:41.292268+0000","last_became_active":"2024-09-13T14:43:09.680908+0000","last_became_peered":"2024-09-13T14:43:09.680908+0000","last_unstale":"2024-09-13T14:47:41.292268+0000","last_undegraded":"2024-09-13T14:47:41.292268+0000","last_fullsized":"2024-09-13T14:47:41.292268+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:33.151302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291589+0000","last_change":"2024-09-13T14:47:39.228107+0000","last_active":"2024-09-13T14:47:41.291589+0000","last_peered":"2024-09-13T14:47:41.291589+0000","last_clean":"2024-09-13T14:47:41.291589+0000","last_became_active":"2024-09-13T14:47:39.227776+0000","last_became_peered":"2024-09-13T14:47:39.227776+0000","last_unstale":"2024-09-13T14:47:41.291589+0000","last_undegraded":"2024-09-13T14:47:41.291589+0000","last_fullsized":"2024-09-13T14:47:41.291589+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:27.854843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":170,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292284+0000","last_change":"2024-09-13T14:43:09.680063+0000","last_active":"2024-09-13T14:47:41.292284+0000","last_peered":"2024-09-13T14:47:41.292284+0000","last_clean":"2024-09-13T14:47:41.292284+0000","last_became_active":"2024-09-13T14:43:09.679740+0000","last_became_peered":"2024-09-13T14:43:09.679740+0000","last_unstale":"2024-09-13T14:47:41.292284+0000","last_undegraded":"2024-09-13T14:47:41.292284+0000","last_fullsized":"2024-09-13T14:47:41.292284+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:55.701172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":476,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.220182+0000","last_change":"2024-09-13T14:40:04.164227+0000","last_active":"2024-09-13T14:47:39.220182+0000","last_peered":"2024-09-13T14:47:39.220182+0000","last_clean":"2024-09-13T14:47:39.220182+0000","last_became_active":"2024-09-13T14:40:04.163752+0000","last_became_peered":"2024-09-13T14:40:04.163752+0000","last_unstale":"2024-09-13T14:47:39.220182+0000","last_undegraded":"2024-09-13T14:47:39.220182+0000","last_fullsized":"2024-09-13T14:47:39.220182+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:08:33.913801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":168,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766757+0000","last_change":"2024-09-13T14:43:09.682650+0000","last_active":"2024-09-13T14:47:40.766757+0000","last_peered":"2024-09-13T14:47:40.766757+0000","last_clean":"2024-09-13T14:47:40.766757+0000","last_became_active":"2024-09-13T14:43:09.682335+0000","last_became_peered":"2024-09-13T14:43:09.682335+0000","last_unstale":"2024-09-13T14:47:40.766757+0000","last_undegraded":"2024-09-13T14:47:40.766757+0000","last_fullsized":"2024-09-13T14:47:40.766757+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:26.113631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.229785+0000","last_change":"2024-09-13T14:47:39.229311+0000","last_active":"2024-09-13T14:47:39.229785+0000","last_peered":"2024-09-13T14:47:39.229785+0000","last_clean":"2024-09-13T14:47:39.229785+0000","last_became_active":"2024-09-13T14:47:39.229006+0000","last_became_peered":"2024-09-13T14:47:39.229006+0000","last_unstale":"2024-09-13T14:47:39.229785+0000","last_undegraded":"2024-09-13T14:47:39.229785+0000","last_fullsized":"2024-09-13T14:47:39.229785+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:46.063192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":180,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292041+0000","last_change":"2024-09-13T14:41:41.493486+0000","last_active":"2024-09-13T14:47:41.292041+0000","last_peered":"2024-09-13T14:47:41.292041+0000","last_clean":"2024-09-13T14:47:41.292041+0000","last_became_active":"2024-09-13T14:41:41.493079+0000","last_became_peered":"2024-09-13T14:41:41.493079+0000","last_unstale":"2024-09-13T14:47:41.292041+0000","last_undegraded":"2024-09-13T14:47:41.292041+0000","last_fullsized":"2024-09-13T14:47:41.292041+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:22.830976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":634,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292579+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:47:41.292579+0000","last_peered":"2024-09-13T14:47:41.292579+0000","last_clean":"2024-09-13T14:47:41.292579+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:47:41.292579+0000","last_undegraded":"2024-09-13T14:47:41.292579+0000","last_fullsized":"2024-09-13T14:47:41.292579+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":178,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291287+0000","last_change":"2024-09-13T14:41:41.486713+0000","last_active":"2024-09-13T14:47:41.291287+0000","last_peered":"2024-09-13T14:47:41.291287+0000","last_clean":"2024-09-13T14:47:41.291287+0000","last_became_active":"2024-09-13T14:41:41.486330+0000","last_became_peered":"2024-09-13T14:41:41.486330+0000","last_unstale":"2024-09-13T14:47:41.291287+0000","last_undegraded":"2024-09-13T14:47:41.291287+0000","last_fullsized":"2024-09-13T14:47:41.291287+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:41.753554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.227029+0000","last_change":"2024-09-13T14:47:39.226420+0000","last_active":"2024-09-13T14:47:39.227029+0000","last_peered":"2024-09-13T14:47:39.227029+0000","last_clean":"2024-09-13T14:47:39.227029+0000","last_became_active":"2024-09-13T14:47:39.226124+0000","last_became_peered":"2024-09-13T14:47:39.226124+0000","last_unstale":"2024-09-13T14:47:39.227029+0000","last_undegraded":"2024-09-13T14:47:39.227029+0000","last_fullsized":"2024-09-13T14:47:39.227029+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:41.520707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":177,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766821+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:47:40.766821+0000","last_peered":"2024-09-13T14:47:40.766821+0000","last_clean":"2024-09-13T14:47:40.766821+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:47:40.766821+0000","last_undegraded":"2024-09-13T14:47:40.766821+0000","last_fullsized":"2024-09-13T14:47:40.766821+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":658,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.224795+0000","last_change":"2024-09-13T14:46:18.008986+0000","last_active":"2024-09-13T14:47:39.224795+0000","last_peered":"2024-09-13T14:47:39.224795+0000","last_clean":"2024-09-13T14:47:39.224795+0000","last_became_active":"2024-09-13T14:46:18.008512+0000","last_became_peered":"2024-09-13T14:46:18.008512+0000","last_unstale":"2024-09-13T14:47:39.224795+0000","last_undegraded":"2024-09-13T14:47:39.224795+0000","last_fullsized":"2024-09-13T14:47:39.224795+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:32:41.601601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":137,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.221974+0000","last_change":"2024-09-13T14:44:54.718379+0000","last_active":"2024-09-13T14:47:39.221974+0000","last_peered":"2024-09-13T14:47:39.221974+0000","last_clean":"2024-09-13T14:47:39.221974+0000","last_became_active":"2024-09-13T14:44:54.717339+0000","last_became_peered":"2024-09-13T14:44:54.717339+0000","last_unstale":"2024-09-13T14:47:39.221974+0000","last_undegraded":"2024-09-13T14:47:39.221974+0000","last_fullsized":"2024-09-13T14:47:39.221974+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:34.326913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222386+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:47:39.222386+0000","last_peered":"2024-09-13T14:47:39.222386+0000","last_clean":"2024-09-13T14:47:39.222386+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:47:39.222386+0000","last_undegraded":"2024-09-13T14:47:39.222386+0000","last_fullsized":"2024-09-13T14:47:39.222386+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":171,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222499+0000","last_change":"2024-09-13T14:44:54.826556+0000","last_active":"2024-09-13T14:47:39.222499+0000","last_peered":"2024-09-13T14:47:39.222499+0000","last_clean":"2024-09-13T14:47:39.222499+0000","last_became_active":"2024-09-13T14:44:54.718081+0000","last_became_peered":"2024-09-13T14:44:54.718081+0000","last_unstale":"2024-09-13T14:47:39.222499+0000","last_undegraded":"2024-09-13T14:47:39.222499+0000","last_fullsized":"2024-09-13T14:47:39.222499+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:58.858037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9512,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":177,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292128+0000","last_change":"2024-09-13T14:44:54.715785+0000","last_active":"2024-09-13T14:47:41.292128+0000","last_peered":"2024-09-13T14:47:41.292128+0000","last_clean":"2024-09-13T14:47:41.292128+0000","last_became_active":"2024-09-13T14:44:54.715454+0000","last_became_peered":"2024-09-13T14:44:54.715454+0000","last_unstale":"2024-09-13T14:47:41.292128+0000","last_undegraded":"2024-09-13T14:47:41.292128+0000","last_fullsized":"2024-09-13T14:47:41.292128+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:18.225794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":129,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.219890+0000","last_change":"2024-09-13T14:41:41.489532+0000","last_active":"2024-09-13T14:47:39.219890+0000","last_peered":"2024-09-13T14:47:39.219890+0000","last_clean":"2024-09-13T14:47:39.219890+0000","last_became_active":"2024-09-13T14:41:41.489123+0000","last_became_peered":"2024-09-13T14:41:41.489123+0000","last_unstale":"2024-09-13T14:47:39.219890+0000","last_undegraded":"2024-09-13T14:47:39.219890+0000","last_fullsized":"2024-09-13T14:47:39.219890+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:43:18.881113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":114,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.219728+0000","last_change":"2024-09-13T14:43:09.682269+0000","last_active":"2024-09-13T14:47:39.219728+0000","last_peered":"2024-09-13T14:47:39.219728+0000","last_clean":"2024-09-13T14:47:39.219728+0000","last_became_active":"2024-09-13T14:43:09.681735+0000","last_became_peered":"2024-09-13T14:43:09.681735+0000","last_unstale":"2024-09-13T14:47:39.219728+0000","last_undegraded":"2024-09-13T14:47:39.219728+0000","last_fullsized":"2024-09-13T14:47:39.219728+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:48.513102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":145,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291246+0000","last_change":"2024-09-13T14:44:54.725385+0000","last_active":"2024-09-13T14:47:41.291246+0000","last_peered":"2024-09-13T14:47:41.291246+0000","last_clean":"2024-09-13T14:47:41.291246+0000","last_became_active":"2024-09-13T14:44:54.725140+0000","last_became_peered":"2024-09-13T14:44:54.725140+0000","last_unstale":"2024-09-13T14:47:41.291246+0000","last_undegraded":"2024-09-13T14:47:41.291246+0000","last_fullsized":"2024-09-13T14:47:41.291246+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:58.246065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":193,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766892+0000","last_change":"2024-09-13T14:44:54.718815+0000","last_active":"2024-09-13T14:47:40.766892+0000","last_peered":"2024-09-13T14:47:40.766892+0000","last_clean":"2024-09-13T14:47:40.766892+0000","last_became_active":"2024-09-13T14:44:54.718516+0000","last_became_peered":"2024-09-13T14:44:54.718516+0000","last_unstale":"2024-09-13T14:47:40.766892+0000","last_undegraded":"2024-09-13T14:47:40.766892+0000","last_fullsized":"2024-09-13T14:47:40.766892+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:31.898033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":196,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291228+0000","last_change":"2024-09-13T14:44:54.718935+0000","last_active":"2024-09-13T14:47:41.291228+0000","last_peered":"2024-09-13T14:47:41.291228+0000","last_clean":"2024-09-13T14:47:41.291228+0000","last_became_active":"2024-09-13T14:44:54.718584+0000","last_became_peered":"2024-09-13T14:44:54.718584+0000","last_unstale":"2024-09-13T14:47:41.291228+0000","last_undegraded":"2024-09-13T14:47:41.291228+0000","last_fullsized":"2024-09-13T14:47:41.291228+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:09:58.902926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":43,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.224725+0000","last_change":"2024-09-13T14:44:54.721244+0000","last_active":"2024-09-13T14:47:39.224725+0000","last_peered":"2024-09-13T14:47:39.224725+0000","last_clean":"2024-09-13T14:47:39.224725+0000","last_became_active":"2024-09-13T14:44:54.720944+0000","last_became_peered":"2024-09-13T14:44:54.720944+0000","last_unstale":"2024-09-13T14:47:39.224725+0000","last_undegraded":"2024-09-13T14:47:39.224725+0000","last_fullsized":"2024-09-13T14:47:39.224725+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:45:18.635223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":198,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291375+0000","last_change":"2024-09-13T14:41:41.488102+0000","last_active":"2024-09-13T14:47:41.291375+0000","last_peered":"2024-09-13T14:47:41.291375+0000","last_clean":"2024-09-13T14:47:41.291375+0000","last_became_active":"2024-09-13T14:41:41.487556+0000","last_became_peered":"2024-09-13T14:41:41.487556+0000","last_unstale":"2024-09-13T14:47:41.291375+0000","last_undegraded":"2024-09-13T14:47:41.291375+0000","last_fullsized":"2024-09-13T14:47:41.291375+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:56:57.765827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":172,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766902+0000","last_change":"2024-09-13T14:41:41.490279+0000","last_active":"2024-09-13T14:47:40.766902+0000","last_peered":"2024-09-13T14:47:40.766902+0000","last_clean":"2024-09-13T14:47:40.766902+0000","last_became_active":"2024-09-13T14:41:41.490035+0000","last_became_peered":"2024-09-13T14:41:41.490035+0000","last_unstale":"2024-09-13T14:47:40.766902+0000","last_undegraded":"2024-09-13T14:47:40.766902+0000","last_fullsized":"2024-09-13T14:47:40.766902+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:26.828205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":162,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291798+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:47:41.291798+0000","last_peered":"2024-09-13T14:47:41.291798+0000","last_clean":"2024-09-13T14:47:41.291798+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:47:41.291798+0000","last_undegraded":"2024-09-13T14:47:41.291798+0000","last_fullsized":"2024-09-13T14:47:41.291798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":181,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292195+0000","last_change":"2024-09-13T14:41:41.493063+0000","last_active":"2024-09-13T14:47:41.292195+0000","last_peered":"2024-09-13T14:47:41.292195+0000","last_clean":"2024-09-13T14:47:41.292195+0000","last_became_active":"2024-09-13T14:41:41.492766+0000","last_became_peered":"2024-09-13T14:41:41.492766+0000","last_unstale":"2024-09-13T14:47:41.292195+0000","last_undegraded":"2024-09-13T14:47:41.292195+0000","last_fullsized":"2024-09-13T14:47:41.292195+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:20.054072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":140,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.223001+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:47:39.223001+0000","last_peered":"2024-09-13T14:47:39.223001+0000","last_clean":"2024-09-13T14:47:39.223001+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:47:39.223001+0000","last_undegraded":"2024-09-13T14:47:39.223001+0000","last_fullsized":"2024-09-13T14:47:39.223001+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":201,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.219708+0000","last_change":"2024-09-13T14:41:41.490887+0000","last_active":"2024-09-13T14:47:40.219708+0000","last_peered":"2024-09-13T14:47:40.219708+0000","last_clean":"2024-09-13T14:47:40.219708+0000","last_became_active":"2024-09-13T14:41:41.490445+0000","last_became_peered":"2024-09-13T14:41:41.490445+0000","last_unstale":"2024-09-13T14:47:40.219708+0000","last_undegraded":"2024-09-13T14:47:40.219708+0000","last_fullsized":"2024-09-13T14:47:40.219708+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:26.483504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":166,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766938+0000","last_change":"2024-09-13T14:40:04.164808+0000","last_active":"2024-09-13T14:47:40.766938+0000","last_peered":"2024-09-13T14:47:40.766938+0000","last_clean":"2024-09-13T14:47:40.766938+0000","last_became_active":"2024-09-13T14:40:04.164464+0000","last_became_peered":"2024-09-13T14:40:04.164464+0000","last_unstale":"2024-09-13T14:47:40.766938+0000","last_undegraded":"2024-09-13T14:47:40.766938+0000","last_fullsized":"2024-09-13T14:47:40.766938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:23.124553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":141,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222354+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:47:39.222354+0000","last_peered":"2024-09-13T14:47:39.222354+0000","last_clean":"2024-09-13T14:47:39.222354+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:47:39.222354+0000","last_undegraded":"2024-09-13T14:47:39.222354+0000","last_fullsized":"2024-09-13T14:47:39.222354+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222154+0000","last_change":"2024-09-13T14:44:54.830007+0000","last_active":"2024-09-13T14:47:39.222154+0000","last_peered":"2024-09-13T14:47:39.222154+0000","last_clean":"2024-09-13T14:47:39.222154+0000","last_became_active":"2024-09-13T14:44:54.720757+0000","last_became_peered":"2024-09-13T14:44:54.720757+0000","last_unstale":"2024-09-13T14:47:39.222154+0000","last_undegraded":"2024-09-13T14:47:39.222154+0000","last_fullsized":"2024-09-13T14:47:39.222154+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:31.423552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":138,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222962+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:47:39.222962+0000","last_peered":"2024-09-13T14:47:39.222962+0000","last_clean":"2024-09-13T14:47:39.222962+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:47:39.222962+0000","last_undegraded":"2024-09-13T14:47:39.222962+0000","last_fullsized":"2024-09-13T14:47:39.222962+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.222862+0000","last_change":"2024-09-13T14:44:54.834271+0000","last_active":"2024-09-13T14:47:39.222862+0000","last_peered":"2024-09-13T14:47:39.222862+0000","last_clean":"2024-09-13T14:47:39.222862+0000","last_became_active":"2024-09-13T14:44:54.721681+0000","last_became_peered":"2024-09-13T14:44:54.721681+0000","last_unstale":"2024-09-13T14:47:39.222862+0000","last_undegraded":"2024-09-13T14:47:39.222862+0000","last_fullsized":"2024-09-13T14:47:39.222862+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:59.844315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":201,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.291589+0000","last_change":"2024-09-13T14:41:41.488284+0000","last_active":"2024-09-13T14:47:41.291589+0000","last_peered":"2024-09-13T14:47:41.291589+0000","last_clean":"2024-09-13T14:47:41.291589+0000","last_became_active":"2024-09-13T14:41:41.487877+0000","last_became_peered":"2024-09-13T14:41:41.487877+0000","last_unstale":"2024-09-13T14:47:41.291589+0000","last_undegraded":"2024-09-13T14:47:41.291589+0000","last_fullsized":"2024-09-13T14:47:41.291589+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:10.995298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":204,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:41.292497+0000","last_change":"2024-09-13T14:44:54.716788+0000","last_active":"2024-09-13T14:47:41.292497+0000","last_peered":"2024-09-13T14:47:41.292497+0000","last_clean":"2024-09-13T14:47:41.292497+0000","last_became_active":"2024-09-13T14:44:54.716462+0000","last_became_peered":"2024-09-13T14:44:54.716462+0000","last_unstale":"2024-09-13T14:47:41.292497+0000","last_undegraded":"2024-09-13T14:47:41.292497+0000","last_fullsized":"2024-09-13T14:47:41.292497+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:11.645196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-13T14:47:39.221742+0000","last_change":"2024-09-13T14:44:54.713470+0000","last_active":"2024-09-13T14:47:39.221742+0000","last_peered":"2024-09-13T14:47:39.221742+0000","last_clean":"2024-09-13T14:47:39.221742+0000","last_became_active":"2024-09-13T14:44:54.713122+0000","last_became_peered":"2024-09-13T14:44:54.713122+0000","last_unstale":"2024-09-13T14:47:39.221742+0000","last_undegraded":"2024-09-13T14:47:39.221742+0000","last_fullsized":"2024-09-13T14:47:39.221742+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:57.220599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":216,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-13T14:47:40.766879+0000","last_change":"2024-09-13T14:44:54.821762+0000","last_active":"2024-09-13T14:47:40.766879+0000","last_peered":"2024-09-13T14:47:40.766879+0000","last_clean":"2024-09-13T14:47:40.766879+0000","last_became_active":"2024-09-13T14:44:54.721169+0000","last_became_peered":"2024-09-13T14:44:54.721169+0000","last_unstale":"2024-09-13T14:47:40.766879+0000","last_undegraded":"2024-09-13T14:47:40.766879+0000","last_fullsized":"2024-09-13T14:47:40.766879+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:26.845873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":11490,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":17314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12582988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234244,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":235776,"kb_used_data":106224,"kb_used_omap":1,"kb_used_meta":129534,"kb_avail":93509376,"statfs":{"total":95995035648,"available":95753601024,"internally_reserved":0,"allocated":108773376,"data_stored":107959295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":132643275},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495881,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119676,"kb_used_data":2408,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93625476,"statfs":{"total":95995035648,"available":95872487424,"internally_reserved":0,"allocated":2465792,"data_stored":1660085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":163208757518,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019155,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181428,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":93563724,"statfs":{"total":95995035648,"available":95809253376,"internally_reserved":0,"allocated":1875968,"data_stored":1069774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183892422},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468043,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100012,"kb_used_data":5916,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":93645140,"statfs":{"total":95995035648,"available":95892623360,"internally_reserved":0,"allocated":6057984,"data_stored":5259370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":96332964},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631569,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":168108,"kb_used_data":5920,"kb_used_omap":8,"kb_used_meta":162167,"kb_avail":93577044,"statfs":{"total":95995035648,"available":95822893056,"internally_reserved":0,"allocated":6062080,"data_stored":5259848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":166059999},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795096,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":343904,"kb_used_data":203928,"kb_used_omap":7,"kb_used_meta":139960,"kb_avail":93401248,"statfs":{"total":95995035648,"available":95642877952,"internally_reserved":0,"allocated":208822272,"data_stored":208027613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":143319666},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958623,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190896,"kb_used_data":6500,"kb_used_omap":7,"kb_used_meta":184376,"kb_avail":93554256,"statfs":{"total":95995035648,"available":95799558144,"internally_reserved":0,"allocated":6656000,"data_stored":5850201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":188801511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:47:51.681 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:47:51.681 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:47:51.682 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 7, 6] 2024-09-13T14:47:51.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-13T14:47:51.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:51.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:51.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:51.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:51.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:51.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:51.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:51.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:47:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:47:51] "GET /metrics HTTP/1.1" 200 36445 "" "Prometheus/2.51.0" 2024-09-13T14:47:52.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:52.078 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:52.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:51 smithi172 ceph-mon[100696]: pgmap v486: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:52.264 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:51 smithi154 ceph-mon[102314]: pgmap v486: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:52.264 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:51 smithi154 ceph-mon[107038]: pgmap v486: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:52.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:52.332 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:52.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:52.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:52.538 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:52.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:52.575 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:52.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:52.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:52.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:52.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:52.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:52.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:52.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:52.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:53.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:53.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:53.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:53.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:53.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:53.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:53.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:47:53.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:53.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:53.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:53.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:53.592 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:53.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:53.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:53.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:53.843 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:53.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:53.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:53.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:54.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:54.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:54.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:47:54.171 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:53 smithi154 ceph-mon[102314]: pgmap v487: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:54.172 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:53 smithi154 ceph-mon[107038]: pgmap v487: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:54.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:54.221 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:54.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:54.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:53 smithi172 ceph-mon[100696]: pgmap v487: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:54.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:54.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:54.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:54.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:54.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:54.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:54.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:54.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:54.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:54.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:54.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:54.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:54.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:54.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:54.986 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-13T14:47:55.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:55.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:55.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2553769520' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:55.128 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:54 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:55.128 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2553769520' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:55.128 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:54 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:55.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:55.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:55.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2553769520' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:55.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:54 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:47:55.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:55.346 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:55.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:55.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:55.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:55.598 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:55.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:55.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:55.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:55.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:55.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:55.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:55.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:56.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:56.099 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:56.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:56.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:56.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:55 smithi172 ceph-mon[100696]: pgmap v488: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:55 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:47:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:55 smithi172 ceph-mon[100696]: osdmap e164: 8 total, 8 up, 7 in 2024-09-13T14:47:56.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:55 smithi154 ceph-mon[102314]: pgmap v488: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:56.307 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:55 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:47:56.307 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:55 smithi154 ceph-mon[102314]: osdmap e164: 8 total, 8 up, 7 in 2024-09-13T14:47:56.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:55 smithi154 ceph-mon[107038]: pgmap v488: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:56.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:55 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:47:56.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:55 smithi154 ceph-mon[107038]: osdmap e164: 8 total, 8 up, 7 in 2024-09-13T14:47:56.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.354 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.425 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:56.425 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:47:56.433 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:56.433 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:56.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:56.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:56.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:56.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:56.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:56.732 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:56.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:56.859 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:56.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:47:56.986 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:56.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:47:57.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:57.111 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:57.155 DEBUG:teuthology.orchestra.run.smithi154:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-13T14:47:57.158 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:47:57.158 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:47:57.207 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:hints = 1 2024-09-13T14:47:57.207 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:47:57.207 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object prefix: benchmark_data_smithi154_175640 2024-09-13T14:47:57.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:47:57.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:47:57.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:57.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:57.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:57.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:57.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:57 smithi154 ceph-mon[102314]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:47:57.469 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:57 smithi154 ceph-mon[107038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:47:57.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:57 smithi172 ceph-mon[100696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:47:57.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:57.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:47:57.534 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:57.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:57.672 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:47:57.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:57.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:57.813 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:47:57.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:57.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:57.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:58.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:58.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:58.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:58.199 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:58 smithi172 ceph-mon[100696]: pgmap v490: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:58.200 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:58.200 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:47:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:58.207 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 1 15 3527 3512 219.487 219.5 0.00334364 0.00454028 2024-09-13T14:47:58.222 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:58.252 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:58 smithi154 ceph-mon[102314]: pgmap v490: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:58.252 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:58.252 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:47:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:58.253 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:58 smithi154 ceph-mon[107038]: pgmap v490: 89 pgs: 89 active+clean; 4.6 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail 2024-09-13T14:47:58.253 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:47:58.253 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:47:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:47:58.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:47:58.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:47:58.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:58.494 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:58.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:58.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:58.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:58.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:58.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:58.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:47:59.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:47:59.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:59.162 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:59.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 2 16 7013 6997 218.64 217.812 0.00438573 0.00456503 2024-09-13T14:47:59.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:59.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:59.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:47:59.432 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:47:59.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:47:59.556 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:47:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:47:59.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:47:59.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:47:59.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:47:59.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:47:59.884 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:47:59.885 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:47:59.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:47:59.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:47:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:47:59.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:00.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:00.102 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:00 smithi172 ceph-mon[100696]: pgmap v491: 89 pgs: 89 active+clean; 45 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s wr, 64 op/s 2024-09-13T14:48:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:00.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:00.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 3 16 9778 9762 203.361 172.812 0.00837986 0.00490734 2024-09-13T14:48:00.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:00.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:00.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:00.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:00.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:00.409 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:00 smithi154 ceph-mon[102314]: pgmap v491: 89 pgs: 89 active+clean; 45 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s wr, 64 op/s 2024-09-13T14:48:00.409 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:00 smithi154 ceph-mon[107038]: pgmap v491: 89 pgs: 89 active+clean; 45 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s wr, 64 op/s 2024-09-13T14:48:00.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:00.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:00.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:00.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:00.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:00.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:00.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:00.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:48:00.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:00.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:00.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:00.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:00 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T14:48:00.743+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e164 definitely_dead 0 2024-09-13T14:48:01.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:01.061 INFO:teuthology.orchestra.run.smithi154.stderr:marked out osd.7. 2024-09-13T14:48:01.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:01.084 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:48:01.085 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:01.108 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:48:01.108 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:01.125 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:48:01.126 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:01.142 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:48:01.142 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:01.143 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:01.159 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:48:01.159 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:01.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:01.178 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:48:01.178 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:01.194 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:48:01.195 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:01.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:01.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 4 16 11876 11860 185.299 131.125 0.00189777 0.00521447 2024-09-13T14:48:01.218 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:48:01.218 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:01.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:01.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:01.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/903024805' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T14:48:01.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:01 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T14:48:01.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:01 smithi172 ceph-mon[100696]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T14:48:01.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/323395025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:01.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:01.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/903024805' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T14:48:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T14:48:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[102314]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T14:48:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/323395025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:01.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/903024805' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T14:48:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T14:48:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[107038]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T14:48:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/323395025' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:01.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:01.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:01.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:01.693 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:01.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:01.816 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:01.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:01.938 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:01.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:02.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:02.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:02.069 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:48:02.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:02.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:02.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 5 16 11876 11860 148.239 0 - 0.00521447 2024-09-13T14:48:02.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[102314]: pgmap v492: 89 pgs: 89 active+clean; 437 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 43 MiB/s wr, 691 op/s 2024-09-13T14:48:02.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T14:48:02.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[102314]: osdmap e165: 8 total, 8 up, 6 in 2024-09-13T14:48:02.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1243133302' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:02.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:48:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:48:01] "GET /metrics HTTP/1.1" 200 36444 "" "Prometheus/2.51.0" 2024-09-13T14:48:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[107038]: pgmap v492: 89 pgs: 89 active+clean; 437 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 43 MiB/s wr, 691 op/s 2024-09-13T14:48:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T14:48:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[107038]: osdmap e165: 8 total, 8 up, 6 in 2024-09-13T14:48:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1243133302' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:02.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:02.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:02.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:02.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:02.457 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:02 smithi172 ceph-mon[100696]: pgmap v492: 89 pgs: 89 active+clean; 437 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 43 MiB/s wr, 691 op/s 2024-09-13T14:48:02.457 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:02 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T14:48:02.457 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:02 smithi172 ceph-mon[100696]: osdmap e165: 8 total, 8 up, 6 in 2024-09-13T14:48:02.457 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1243133302' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:02.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:02.562 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:02.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:02.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:02.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:02.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:02.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:02.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:02.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:02.944 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:48:02.944 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:48:02.944 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:48:02.945 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:48:02.945 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:48:03.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.059 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:03.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:03.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:03.198 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:03.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 6 16 12028 12012 125.116 4.75 0.0076054 0.00798413 2024-09-13T14:48:03.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:03.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:03.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1243133302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[102314]: osdmap e166: 8 total, 8 up, 6 in 2024-09-13T14:48:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[102314]: osdmap e167: 8 total, 8 up, 6 in 2024-09-13T14:48:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1243133302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[107038]: osdmap e166: 8 total, 8 up, 6 in 2024-09-13T14:48:03.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:03 smithi154 ceph-mon[107038]: osdmap e167: 8 total, 8 up, 6 in 2024-09-13T14:48:03.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:03 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1243133302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:03 smithi172 ceph-mon[100696]: osdmap e166: 8 total, 8 up, 6 in 2024-09-13T14:48:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:03 smithi172 ceph-mon[100696]: osdmap e167: 8 total, 8 up, 6 in 2024-09-13T14:48:03.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:03.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:03.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:03.732 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:03.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:03.873 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:03.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:03.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:03.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:48:03.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:04.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:04.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:04.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:04.164 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:04.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 7 16 14165 14149 126.321 133.562 0.00665621 0.007908 2024-09-13T14:48:04.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:04 smithi154 ceph-mon[102314]: pgmap v495: 89 pgs: 2 remapped+peering, 87 active+clean; 513 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:48:04.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:04 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:48:04.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:04 smithi154 ceph-mon[107038]: pgmap v495: 89 pgs: 2 remapped+peering, 87 active+clean; 513 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:48:04.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:04 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:48:04.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:04 smithi172 ceph-mon[100696]: pgmap v495: 89 pgs: 2 remapped+peering, 87 active+clean; 513 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s; 0 B/s, 0 objects/s recovering 2024-09-13T14:48:04.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:04 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:48:04.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.294 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:04.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:04.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:04.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:04.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:04.571 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:04.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:04.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:04.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:04.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:04.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:04.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:04.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:05.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:05.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:05.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:05.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:05.208 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 8 16 16185 16169 126.311 126.25 0.013239 0.00790878 2024-09-13T14:48:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:05.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:05.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:05.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:05.458 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:05 smithi172 ceph-mon[100696]: osdmap e168: 8 total, 8 up, 6 in 2024-09-13T14:48:05.458 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:05 smithi172 ceph-mon[100696]: osdmap e169: 8 total, 8 up, 6 in 2024-09-13T14:48:05.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:05 smithi154 ceph-mon[102314]: osdmap e168: 8 total, 8 up, 6 in 2024-09-13T14:48:05.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:05 smithi154 ceph-mon[102314]: osdmap e169: 8 total, 8 up, 6 in 2024-09-13T14:48:05.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:05 smithi154 ceph-mon[107038]: osdmap e168: 8 total, 8 up, 6 in 2024-09-13T14:48:05.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:05 smithi154 ceph-mon[107038]: osdmap e169: 8 total, 8 up, 6 in 2024-09-13T14:48:05.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:05.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:05.554 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:05.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:05.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:05.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:05.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:05.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:05.928 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:05.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:05.945 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:48:06.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:06.036 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:48:06.036 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:48:06.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:48:06.036 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:48:06.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.036 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:114968 5.f 5:fdbb20b9:::benchmark_data_smithi154_144140_object4615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:48:06.036 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:53.938605+0000", 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 192.05098725100001, 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89559311799999997, 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114968 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.938605+0000", 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.938605+0000", 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.938604+0000", 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.938613+0000", 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9939999999999995e-06 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.938625+0000", 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1695999999999999e-05 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.938632+0000", 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1439999999999997e-06 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.946327+0000", 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0076949749999999997 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.946338+0000", 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-09-13T14:48:06.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.825173+0000", 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87883499399999998 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.825306+0000", 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133118 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.825436+0000", 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129818 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830692+0000", 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052559160000000002 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830763+0000", 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1375e-05 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.832536+0000", 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017727559999999999 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.832595+0000", 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9423999999999999e-05 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834198+0000", 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016032520000000001 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117851 5.e 5:700ea90e:::benchmark_data_smithi154_144140_object7498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.096204+0000", 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 187.89338860199999, 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.26560158900000003, 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.044 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117851 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096204+0000", 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096204+0000", 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096203+0000", 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096210+0000", 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.99e-06 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096226+0000", 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.605e-05 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096233+0000", 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439999999999996e-06 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096276+0000", 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3089e-05 2024-09-13T14:48:06.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096400+0000", 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012452200000000001 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.096510+0000", 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109483 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.102796+0000", 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062864669999999996 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.102849+0000", 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2117999999999998e-05 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.361711+0000", 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.25886240500000002 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.361762+0000", 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0726999999999997e-05 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.361806+0000", 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3843e-05 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117856 5.e 5:70cff86c:::benchmark_data_smithi154_144140_object7503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.099094+0000", 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 187.89049872800001, 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.26889476099999998, 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117856 2024-09-13T14:48:06.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099094+0000", 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099094+0000", 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099093+0000", 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099097+0000", 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5859999999999998e-06 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099108+0000", 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0834e-05 2024-09-13T14:48:06.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099113+0000", 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.78e-06 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099462+0000", 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034937999999999999 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099558+0000", 2024-09-13T14:48:06.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6024000000000001e-05 2024-09-13T14:48:06.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.099666+0000", 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107243 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.102834+0000", 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031682699999999999 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.102987+0000", 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000152905 2024-09-13T14:48:06.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.367889+0000", 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.26490201600000002 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.367945+0000", 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6059e-05 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.367989+0000", 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3655999999999997e-05 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117859 5.f 5:fa381b5f:::benchmark_data_smithi154_144140_object7506:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.103623+0000", 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 187.88597004100001, 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.252822881, 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117859 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103623+0000", 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103623+0000", 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103621+0000", 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103628+0000", 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7320000000000001e-06 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103639+0000", 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1319e-05 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103645+0000", 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4620000000000001e-06 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103696+0000", 2024-09-13T14:48:06.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0281999999999997e-05 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103839+0000", 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014375100000000001 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103983+0000", 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000143685 2024-09-13T14:48:06.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.111169+0000", 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071854320000000003 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.111331+0000", 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237900000000001 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.355115+0000", 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.24378411 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.355159+0000", 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4047e-05 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.356445+0000", 2024-09-13T14:48:06.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001286305 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133440 5.f 5:fc5e6303:::benchmark_data_smithi154_144140_object23087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:15.983182+0000", 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 170.006410888, 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0259140289999999, 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.065 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133440 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983182+0000", 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983182+0000", 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983181+0000", 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983184+0000", 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6440000000000002e-06 2024-09-13T14:48:06.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983193+0000", 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0679999999999997e-06 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983196+0000", 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9670000000000003e-06 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983214+0000", 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7781000000000001e-05 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983325+0000", 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110649 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.983465+0000", 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014029299999999999 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986355+0000", 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028898029999999998 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986435+0000", 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9850000000000003e-05 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.998276+0000", 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011840741 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.998295+0000", 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9004e-05 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.009010+0000", 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0107154380000001 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.009096+0000", 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5650999999999997e-05 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133441 5.f 5:fa8b970c:::benchmark_data_smithi154_144140_object23088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:15.986621+0000", 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 170.00297110700001, 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0228510799999999, 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.073 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.074 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133441 2024-09-13T14:48:06.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.074 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986621+0000", 2024-09-13T14:48:06.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986621+0000", 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986620+0000", 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986625+0000", 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2009999999999998e-06 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986634+0000", 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1880000000000008e-06 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986639+0000", 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7389999999999999e-06 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986677+0000", 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8380000000000002e-05 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986779+0000", 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101434 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.986861+0000", 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2162000000000004e-05 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.992014+0000", 2024-09-13T14:48:06.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051529979999999998 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.993081+0000", 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001066839 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.998325+0000", 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052439139999999997 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:15.998336+0000", 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1521e-05 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.009409+0000", 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.01107273 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.009473+0000", 2024-09-13T14:48:06.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3505999999999998e-05 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133460 5.f 5:fa971c39:::benchmark_data_smithi154_144140_object23107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.012612+0000", 2024-09-13T14:48:06.083 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.976980518, 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0065098299999999, 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133460 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012612+0000", 2024-09-13T14:48:06.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012612+0000", 2024-09-13T14:48:06.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012610+0000", 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012619+0000", 2024-09-13T14:48:06.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1190000000000003e-06 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012644+0000", 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4538000000000001e-05 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012654+0000", 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0020000000000001e-05 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012798+0000", 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144408 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012808+0000", 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6670000000000008e-06 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.009512+0000", 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99670376299999996 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.009644+0000", 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000132445 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.009774+0000", 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129865 2024-09-13T14:48:06.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.016156+0000", 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063825690000000003 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.018998+0000", 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028415939999999998 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.019039+0000", 2024-09-13T14:48:06.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1337000000000001e-05 2024-09-13T14:48:06.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.019086+0000", 2024-09-13T14:48:06.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6164e-05 2024-09-13T14:48:06.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.019122+0000", 2024-09-13T14:48:06.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6300999999999997e-05 2024-09-13T14:48:06.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.093 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.093 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133461 5.f 5:fd1f94a0:::benchmark_data_smithi154_144140_object23108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.094 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.012678+0000", 2024-09-13T14:48:06.094 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.976914513, 2024-09-13T14:48:06.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0100811059999999, 2024-09-13T14:48:06.094 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.094 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.095 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.095 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.095 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133461 2024-09-13T14:48:06.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.095 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012678+0000", 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012678+0000", 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012677+0000", 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012682+0000", 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.233e-06 2024-09-13T14:48:06.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012689+0000", 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7549999999999994e-06 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012695+0000", 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4090000000000002e-06 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012830+0000", 2024-09-13T14:48:06.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013547799999999999 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.012837+0000", 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1890000000000001e-06 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.012898+0000", 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0000606919999999 2024-09-13T14:48:06.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.013018+0000", 2024-09-13T14:48:06.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120153 2024-09-13T14:48:06.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.013136+0000", 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011807900000000001 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.016203+0000", 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030671679999999999 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.019134+0000", 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029301179999999998 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022653+0000", 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035196469999999999 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022717+0000", 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3677999999999996e-05 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022759+0000", 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2261000000000001e-05 2024-09-13T14:48:06.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133480 5.f 5:f8c46fd0:::benchmark_data_smithi154_144140_object23127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.035524+0000", 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.954069046, 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99344206599999996, 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133480 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035524+0000", 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035524+0000", 2024-09-13T14:48:06.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035522+0000", 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035529+0000", 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035542+0000", 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3006e-05 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035549+0000", 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2840000000000001e-06 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035604+0000", 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5120000000000001e-05 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.035613+0000", 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.225e-06 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.016217+0000", 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.980604857 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.016291+0000", 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3977e-05 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.016359+0000", 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7234e-05 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022560+0000", 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062011410000000003 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022566+0000", 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9159999999999998e-06 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.028852+0000", 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062863770000000001 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.028921+0000", 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8779e-05 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.028966+0000", 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4759999999999998e-05 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133490 5.f 5:f2c6cf58:::benchmark_data_smithi154_144140_object23137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.048034+0000", 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.941558824, 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98105253199999998, 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133490 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.111 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048034+0000", 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048034+0000", 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048033+0000", 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048037+0000", 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3939999999999998e-06 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048046+0000", 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6100000000000006e-06 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048053+0000", 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4229999999999999e-06 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048111+0000", 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7336000000000001e-05 2024-09-13T14:48:06.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.048120+0000", 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8910000000000001e-06 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.019168+0000", 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97104846899999997 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.019240+0000", 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1701999999999999e-05 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.019309+0000", 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9695000000000003e-05 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022796+0000", 2024-09-13T14:48:06.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034861800000000002 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022803+0000", 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0099999999999998e-06 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029015+0000", 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062128050000000001 2024-09-13T14:48:06.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029055+0000", 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9190999999999998e-05 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029086+0000", 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1590000000000001e-05 2024-09-13T14:48:06.118 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133511 5.f 5:f8aa0fab:::benchmark_data_smithi154_144140_object23158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.075604+0000", 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.91398849199999, 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95358924099999998, 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.119 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133511 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075604+0000", 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075604+0000", 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075602+0000", 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075613+0000", 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1029999999999999e-05 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075635+0000", 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2254000000000001e-05 2024-09-13T14:48:06.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075646+0000", 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0869999999999999e-05 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075728+0000", 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2324999999999995e-05 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.075741+0000", 2024-09-13T14:48:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.269e-05 2024-09-13T14:48:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022823+0000", 2024-09-13T14:48:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.947082333 2024-09-13T14:48:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.022944+0000", 2024-09-13T14:48:06.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121099 2024-09-13T14:48:06.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.023068+0000", 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123189 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.025550+0000", 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024825400000000001 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029138+0000", 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035882069999999999 2024-09-13T14:48:06.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029146+0000", 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2139999999999999e-06 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029173+0000", 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7132e-05 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029193+0000", 2024-09-13T14:48:06.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0537999999999999e-05 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133606 5.f 5:fdf65816:::benchmark_data_smithi154_144140_object23253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.205637+0000", 2024-09-13T14:48:06.128 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.783955537, 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.82685999799999998, 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133606 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205637+0000", 2024-09-13T14:48:06.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205637+0000", 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205636+0000", 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205641+0000", 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5269999999999998e-06 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205649+0000", 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4070000000000004e-06 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205653+0000", 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5630000000000002e-06 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205698+0000", 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4289999999999998e-05 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.205707+0000", 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8560000000000002e-06 2024-09-13T14:48:06.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.025563+0000", 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81985566899999995 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.025717+0000", 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015384599999999999 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.025825+0000", 2024-09-13T14:48:06.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107638 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029222+0000", 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033977109999999999 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029256+0000", 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4128999999999998e-05 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.032411+0000", 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031540460000000002 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.032460+0000", 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9768000000000002e-05 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.032497+0000", 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6718999999999999e-05 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133613 5.f 5:f4abc695:::benchmark_data_smithi154_144140_object23260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.210893+0000", 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.77869944599999, 2024-09-13T14:48:06.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.82745507100000004, 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133613 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210893+0000", 2024-09-13T14:48:06.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210893+0000", 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210892+0000", 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210898+0000", 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9649999999999996e-06 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210910+0000", 2024-09-13T14:48:06.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2576e-05 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210917+0000", 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210975+0000", 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8396999999999997e-05 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.210986+0000", 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.094e-05 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029278+0000", 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81829247299999996 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029343+0000", 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5018000000000007e-05 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.029403+0000", 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9106e-05 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.035851+0000", 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064488810000000001 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.038211+0000", 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002359047 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.038218+0000", 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6669999999999996e-06 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.038310+0000", 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1385999999999995e-05 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.038348+0000", 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8600999999999999e-05 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133633 5.f 5:f1dfc866:::benchmark_data_smithi154_144140_object23280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.239215+0000", 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.750377184, 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80220314100000001, 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.145 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133633 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239215+0000", 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239215+0000", 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239214+0000", 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239220+0000", 2024-09-13T14:48:06.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3559999999999998e-06 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239238+0000", 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7544999999999999e-05 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239247+0000", 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9600000000000006e-06 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239304+0000", 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7173999999999998e-05 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.239315+0000", 2024-09-13T14:48:06.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0693000000000001e-05 2024-09-13T14:48:06.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.032580+0000", 2024-09-13T14:48:06.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.79326500099999997 2024-09-13T14:48:06.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.032695+0000", 2024-09-13T14:48:06.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115257 2024-09-13T14:48:06.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.032788+0000", 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3012999999999999e-05 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.038355+0000", 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005567341 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.038390+0000", 2024-09-13T14:48:06.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4486999999999997e-05 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.041342+0000", 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029518650000000001 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.041385+0000", 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3294999999999998e-05 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.041419+0000", 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3609999999999998e-05 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133639 5.f 5:f7fefb73:::benchmark_data_smithi154_144140_object23286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.246810+0000", 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.74278228599999, 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80673705799999995, 2024-09-13T14:48:06.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133639 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246810+0000", 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246810+0000", 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246809+0000", 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246814+0000", 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.259e-06 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246823+0000", 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9309999999999997e-06 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246830+0000", 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5010000000000001e-06 2024-09-13T14:48:06.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246879+0000", 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9379000000000003e-05 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.246892+0000", 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2482e-05 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.041440+0000", 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.79454853400000003 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.041551+0000", 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110936 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.041686+0000", 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013461500000000001 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.050937+0000", 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092517180000000008 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.050998+0000", 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0114000000000001e-05 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.053472+0000", 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024739139999999998 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.053501+0000", 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9456999999999999e-05 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.053547+0000", 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6343e-05 2024-09-13T14:48:06.161 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133642 5.f 5:f632eecb:::benchmark_data_smithi154_144140_object23289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.254070+0000", 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.73552213299999, 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80254834200000003, 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.162 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133642 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254070+0000", 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254070+0000", 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254069+0000", 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254076+0000", 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2910000000000002e-06 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254089+0000", 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2282e-05 2024-09-13T14:48:06.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254096+0000", 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7160000000000003e-06 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254150+0000", 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3415000000000003e-05 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.254158+0000", 2024-09-13T14:48:06.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2930000000000001e-06 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.044322+0000", 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.79016422200000003 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.044431+0000", 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010819600000000001 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.044516+0000", 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5288000000000001e-05 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.050973+0000", 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064569099999999997 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056464+0000", 2024-09-13T14:48:06.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054912299999999997 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056519+0000", 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5269000000000002e-05 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056566+0000", 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7169000000000002e-05 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056619+0000", 2024-09-13T14:48:06.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2309999999999999e-05 2024-09-13T14:48:06.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.169 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.169 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133645 5.f 5:f7e3cc0b:::benchmark_data_smithi154_144140_object23292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.258891+0000", 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.73070162299999, 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.79800980300000002, 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.170 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133645 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258891+0000", 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258891+0000", 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258890+0000", 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258894+0000", 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3560000000000003e-06 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258902+0000", 2024-09-13T14:48:06.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9189999999999999e-06 2024-09-13T14:48:06.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258906+0000", 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.172e-06 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258946+0000", 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9335000000000002e-05 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.258954+0000", 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8679999999999999e-06 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.047600+0000", 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.78864648599999998 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.047738+0000", 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000138307 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.047861+0000", 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012291299999999999 2024-09-13T14:48:06.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056453+0000", 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085916320000000001 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056677+0000", 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022410099999999999 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056697+0000", 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0335000000000001e-05 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056767+0000", 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9224e-05 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056901+0000", 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013414899999999999 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133647 5.f 5:f1b5737f:::benchmark_data_smithi154_144140_object23294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.260044+0000", 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.729548572, 2024-09-13T14:48:06.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.816288927, 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133647 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260044+0000", 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260044+0000", 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260043+0000", 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260048+0000", 2024-09-13T14:48:06.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6600000000000003e-06 2024-09-13T14:48:06.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260060+0000", 2024-09-13T14:48:06.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.19e-05 2024-09-13T14:48:06.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260066+0000", 2024-09-13T14:48:06.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3720000000000001e-06 2024-09-13T14:48:06.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260174+0000", 2024-09-13T14:48:06.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010753599999999999 2024-09-13T14:48:06.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.260184+0000", 2024-09-13T14:48:06.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0271e-05 2024-09-13T14:48:06.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.056929+0000", 2024-09-13T14:48:06.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.79674534900000005 2024-09-13T14:48:06.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.057048+0000", 2024-09-13T14:48:06.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118584 2024-09-13T14:48:06.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.057153+0000", 2024-09-13T14:48:06.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010505000000000001 2024-09-13T14:48:06.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.075918+0000", 2024-09-13T14:48:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018765148999999998 2024-09-13T14:48:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.075981+0000", 2024-09-13T14:48:06.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2700000000000006e-05 2024-09-13T14:48:06.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076225+0000", 2024-09-13T14:48:06.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000244219 2024-09-13T14:48:06.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076281+0000", 2024-09-13T14:48:06.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5807000000000003e-05 2024-09-13T14:48:06.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076333+0000", 2024-09-13T14:48:06.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2200000000000002e-05 2024-09-13T14:48:06.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.190 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.191 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133661 5.f 5:f5f628d5:::benchmark_data_smithi154_144140_object23308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.191 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.300224+0000", 2024-09-13T14:48:06.191 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.689368059, 2024-09-13T14:48:06.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.77617865200000002, 2024-09-13T14:48:06.191 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.192 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.192 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.192 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.192 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.192 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133661 2024-09-13T14:48:06.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.193 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300224+0000", 2024-09-13T14:48:06.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300224+0000", 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300223+0000", 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300230+0000", 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3100000000000003e-06 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300243+0000", 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2334000000000001e-05 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300250+0000", 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5549999999999997e-06 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300299+0000", 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8903999999999998e-05 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.300307+0000", 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.060285+0000", 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.75997778699999996 2024-09-13T14:48:06.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.060379+0000", 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4234999999999995e-05 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.060472+0000", 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2749000000000003e-05 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.075952+0000", 2024-09-13T14:48:06.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015480521000000001 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076010+0000", 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8269000000000001e-05 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076342+0000", 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033117800000000001 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076374+0000", 2024-09-13T14:48:06.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2762999999999998e-05 2024-09-13T14:48:06.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076403+0000", 2024-09-13T14:48:06.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8787999999999999e-05 2024-09-13T14:48:06.200 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.200 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.200 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:133663 5.f 5:fc91c3ee:::benchmark_data_smithi154_144140_object23310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:45:16.307268+0000", 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.68232498, 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.76921289599999998, 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 133663 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.201 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307268+0000", 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307268+0000", 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307267+0000", 2024-09-13T14:48:06.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307271+0000", 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7709999999999997e-06 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307280+0000", 2024-09-13T14:48:06.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0880000000000001e-06 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307285+0000", 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8629999999999999e-06 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307325+0000", 2024-09-13T14:48:06.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9295999999999998e-05 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:16.307331+0000", 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.063751+0000", 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.75641970700000005 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.063867+0000", 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011636799999999999 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,6", 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.063976+0000", 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108578 2024-09-13T14:48:06.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076042+0000", 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012066429 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076070+0000", 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8000999999999998e-05 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076412+0000", 2024-09-13T14:48:06.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034223000000000001 2024-09-13T14:48:06.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:06.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076446+0000", 2024-09-13T14:48:06.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3399999999999999e-05 2024-09-13T14:48:06.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:45:17.076480+0000", 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4804000000000001e-05 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:06.209 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:48:06.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:06.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:06.210 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 9 16 17934 17918 124.398 109.312 0.00916586 0.00803129 2024-09-13T14:48:06.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.339 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:06 smithi154 ceph-mon[102314]: pgmap v498: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 79 active+clean; 824 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 97 MiB/s wr, 1.54k op/s; 3801/39393 objects degraded (9.649%); 540/39393 objects misplaced (1.371%); 2.5 MiB/s, 2 keys/s, 53 objects/s recovering 2024-09-13T14:48:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:06 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 3801/39393 objects degraded (9.649%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:06.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:06 smithi154 ceph-mon[107038]: pgmap v498: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 79 active+clean; 824 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 97 MiB/s wr, 1.54k op/s; 3801/39393 objects degraded (9.649%); 540/39393 objects misplaced (1.371%); 2.5 MiB/s, 2 keys/s, 53 objects/s recovering 2024-09-13T14:48:06.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:06 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 3801/39393 objects degraded (9.649%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:06.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:06 smithi172 ceph-mon[100696]: pgmap v498: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 79 active+clean; 824 MiB data, 2.9 GiB used, 533 GiB / 536 GiB avail; 97 MiB/s wr, 1.54k op/s; 3801/39393 objects degraded (9.649%); 540/39393 objects misplaced (1.371%); 2.5 MiB/s, 2 keys/s, 53 objects/s recovering 2024-09-13T14:48:06.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:06 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 3801/39393 objects degraded (9.649%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:06.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:06.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:06.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:06.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:07.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:07.210 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 10 16 19571 19555 122.188 102.312 0.00419608 0.00810229 2024-09-13T14:48:07.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:07.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:07.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:07.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:07.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:07.523 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:07.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:07.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:07.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:48:07.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:07.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:07.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:07.920 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:07.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:07.943 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:08.022 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:08.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:08.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:08.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:08.210 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 11 16 21146 21130 120.029 98.4375 0.0142747 0.0083226 2024-09-13T14:48:08.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:08.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:08.339 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:08 smithi172 ceph-mon[100696]: pgmap v500: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 79 active+clean; 900 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 96 MiB/s wr, 1.53k op/s; 3801/43065 objects degraded (8.826%); 540/43065 objects misplaced (1.254%); 2.0 MiB/s, 2 keys/s, 48 objects/s recovering 2024-09-13T14:48:08.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:08.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:08 smithi154 ceph-mon[102314]: pgmap v500: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 79 active+clean; 900 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 96 MiB/s wr, 1.53k op/s; 3801/43065 objects degraded (8.826%); 540/43065 objects misplaced (1.254%); 2.0 MiB/s, 2 keys/s, 48 objects/s recovering 2024-09-13T14:48:08.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:08 smithi154 ceph-mon[107038]: pgmap v500: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 79 active+clean; 900 MiB data, 3.1 GiB used, 533 GiB / 536 GiB avail; 96 MiB/s wr, 1.53k op/s; 3801/43065 objects degraded (8.826%); 540/43065 objects misplaced (1.254%); 2.0 MiB/s, 2 keys/s, 48 objects/s recovering 2024-09-13T14:48:08.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:08.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:08.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:08.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:08.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:08.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:08.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:08.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:08.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:09.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:09.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:09.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:09.210 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 12 16 22652 22636 117.87 94.125 0.0055393 0.0084782 2024-09-13T14:48:09.221 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:09.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:09.335 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:48:09.335 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:48:09.335 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-13T14:48:09.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd in 5 2024-09-13T14:48:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:09.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:09.418 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:09 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:48:09.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:09 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:48:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:09 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:48:09.497 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:09.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:09.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:09.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:09.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:09.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:09.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:09.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:10.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:10.035 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:48:10.036 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:48:10.036 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T14:48:10.036 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:48:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:10.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:10.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:10.173 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:10.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 13 16 23868 23852 114.649 76 0.00190005 0.00852391 2024-09-13T14:48:10.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:10.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:10.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:10.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:10.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:10.546 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:10.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:10.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:10.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:10.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.798 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:10.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:10.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:10.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:10.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:10 smithi154 ceph-mon[102314]: pgmap v501: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.0 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s; 4504/51237 objects degraded (8.791%); 540/51237 objects misplaced (1.054%); 1.9 MiB/s, 1 keys/s, 38 objects/s recovering 2024-09-13T14:48:10.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:10 smithi154 ceph-mon[102314]: osdmap e170: 8 total, 8 up, 6 in 2024-09-13T14:48:10.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:10 smithi154 ceph-mon[107038]: pgmap v501: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.0 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s; 4504/51237 objects degraded (8.791%); 540/51237 objects misplaced (1.054%); 1.9 MiB/s, 1 keys/s, 38 objects/s recovering 2024-09-13T14:48:10.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:10 smithi154 ceph-mon[107038]: osdmap e170: 8 total, 8 up, 6 in 2024-09-13T14:48:11.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:11.065 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:11.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:11.202 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:10 smithi172 ceph-mon[100696]: pgmap v501: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 1.0 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 93 MiB/s wr, 1.49k op/s; 4504/51237 objects degraded (8.791%); 540/51237 objects misplaced (1.054%); 1.9 MiB/s, 1 keys/s, 38 objects/s recovering 2024-09-13T14:48:11.203 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:10 smithi172 ceph-mon[100696]: osdmap e170: 8 total, 8 up, 6 in 2024-09-13T14:48:11.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:11.210 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 14 16 24672 24656 110.049 50.25 0.0168451 0.00907946 2024-09-13T14:48:11.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:48:11.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:11.314 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:11.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:11.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:11.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:11.486 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:11.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:11.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:11.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:11.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:11.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:11.872 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:11 smithi172 ceph-mon[100696]: osdmap e171: 8 total, 8 up, 6 in 2024-09-13T14:48:11.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:11.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:11.921 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:11.992 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:48:11.992 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:12.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:12.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:12.037 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:48:12.037 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:11 smithi154 ceph-mon[102314]: osdmap e171: 8 total, 8 up, 6 in 2024-09-13T14:48:12.038 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:48:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:48:11] "GET /metrics HTTP/1.1" 200 36444 "" "Prometheus/2.51.0" 2024-09-13T14:48:12.038 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:11 smithi154 ceph-mon[107038]: osdmap e171: 8 total, 8 up, 6 in 2024-09-13T14:48:12.038 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:12.078 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:48:12.078 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:12.122 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:48:12.122 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:12.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:12.158 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:48:12.159 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:12.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:12.193 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:48:12.193 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:12.210 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 15 15 26119 26104 108.745 90.5 0.00680025 0.00918786 2024-09-13T14:48:12.227 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:48:12.227 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:12.261 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:48:12.262 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:12.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.307 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:12.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:12.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:12.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:12.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:12.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:12.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:12.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:12.687 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:12.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:12.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:12.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:12 smithi172 ceph-mon[100696]: pgmap v504: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.4 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s; 3754/68205 objects degraded (5.504%); 717/68205 objects misplaced (1.051%); 6.1 MiB/s, 99 objects/s recovering 2024-09-13T14:48:12.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:12 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3754/68205 objects degraded (5.504%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:12.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1198560701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:12.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:12 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:12.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:12.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:12.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[102314]: pgmap v504: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.4 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s; 3754/68205 objects degraded (5.504%); 717/68205 objects misplaced (1.051%); 6.1 MiB/s, 99 objects/s recovering 2024-09-13T14:48:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3754/68205 objects degraded (5.504%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1198560701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:12.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[107038]: pgmap v504: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.4 GiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s; 3754/68205 objects degraded (5.504%); 717/68205 objects misplaced (1.051%); 6.1 MiB/s, 99 objects/s recovering 2024-09-13T14:48:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3754/68205 objects degraded (5.504%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1198560701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:12 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:48:13.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:13.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:13.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:13.210 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 16 16 27798 27782 108.502 104.875 0.00775789 0.00920966 2024-09-13T14:48:13.211 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:13.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:13.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:13.367 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:13.367 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:13.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:13.461 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:13.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:13.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:13.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:13.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:13.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:13.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:13.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:13.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:14.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:14.055 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:13 smithi172 ceph-mon[100696]: pgmap v505: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.4 GiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 84 MiB/s wr, 1.34k op/s; 3754/70701 objects degraded (5.310%); 717/70701 objects misplaced (1.014%); 4.7 MiB/s, 76 objects/s recovering 2024-09-13T14:48:14.055 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:14.055 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:14.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:48:14.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:14.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 17 16 29280 29264 107.568 92.625 0.00207365 0.0092371 2024-09-13T14:48:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:13 smithi154 ceph-mon[102314]: pgmap v505: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.4 GiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 84 MiB/s wr, 1.34k op/s; 3754/70701 objects degraded (5.310%); 717/70701 objects misplaced (1.014%); 4.7 MiB/s, 76 objects/s recovering 2024-09-13T14:48:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:13 smithi154 ceph-mon[107038]: pgmap v505: 89 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 1.4 GiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 84 MiB/s wr, 1.34k op/s; 3754/70701 objects degraded (5.310%); 717/70701 objects misplaced (1.014%); 4.7 MiB/s, 76 objects/s recovering 2024-09-13T14:48:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:14.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:14.285 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:14.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:14.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:14.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.534 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:14.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:14.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:14.657 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:14.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:14.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:14.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:14.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:14.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:14.964 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:15.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:15.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:15.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:15.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:15.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 18 16 29682 29666 102.988 25.125 0.00837201 0.00970243 2024-09-13T14:48:15.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:14 smithi172 ceph-mon[100696]: osdmap e172: 8 total, 8 up, 6 in 2024-09-13T14:48:15.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:15.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:15.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:14 smithi154 ceph-mon[102314]: osdmap e172: 8 total, 8 up, 6 in 2024-09-13T14:48:15.353 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:14 smithi154 ceph-mon[107038]: osdmap e172: 8 total, 8 up, 6 in 2024-09-13T14:48:15.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:15.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:15.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:15.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:15.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:15.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:15.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:15.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:15.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:15.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:15.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:15.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:16.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:16.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:16.118 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:15 smithi154 ceph-mon[107038]: pgmap v507: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 112 MiB/s wr, 1.80k op/s; 2991/83637 objects degraded (3.576%); 700/83637 objects misplaced (0.837%); 14 MiB/s, 225 objects/s recovering 2024-09-13T14:48:16.118 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:15 smithi154 ceph-mon[107038]: osdmap e173: 8 total, 8 up, 6 in 2024-09-13T14:48:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:16.119 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:15 smithi154 ceph-mon[102314]: pgmap v507: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 112 MiB/s wr, 1.80k op/s; 2991/83637 objects degraded (3.576%); 700/83637 objects misplaced (0.837%); 14 MiB/s, 225 objects/s recovering 2024-09-13T14:48:16.119 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:15 smithi154 ceph-mon[102314]: osdmap e173: 8 total, 8 up, 6 in 2024-09-13T14:48:16.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 19 16 31298 31282 102.883 101 0.0159224 0.0097133 2024-09-13T14:48:16.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:16.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:15 smithi172 ceph-mon[100696]: pgmap v507: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.7 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 112 MiB/s wr, 1.80k op/s; 2991/83637 objects degraded (3.576%); 700/83637 objects misplaced (0.837%); 14 MiB/s, 225 objects/s recovering 2024-09-13T14:48:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:15 smithi172 ceph-mon[100696]: osdmap e173: 8 total, 8 up, 6 in 2024-09-13T14:48:16.244 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:16.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:16.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:16.505 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.546 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:16.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:16.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:16.671 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:16.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:16.800 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:16.902 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:16.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:16.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:17.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:17.064 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:17.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:17.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:17.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:48:17.211530+0000 min lat: 0.001048 max lat: 2.09981 avg lat: 0.00976559 2024-09-13T14:48:17.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:48:17.212 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 20 15 32768 32753 102.333 91.9375 0.0136003 0.00976559 2024-09-13T14:48:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:17.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:17.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:17.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:17.350 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:17 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2991/86973 objects degraded (3.439%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:17.378 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:17.378 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:17.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:17.458 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:17 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2991/86973 objects degraded (3.439%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:17.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:17 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2991/86973 objects degraded (3.439%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:17.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:17.505 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:17.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:17.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:17.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:17.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:17.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:17.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:17.881 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:17.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:17.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:18.008 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:18.047 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:18 smithi154 ceph-mon[102314]: pgmap v509: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 2991/86973 objects degraded (3.439%); 700/86973 objects misplaced (0.805%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:48:18.047 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/475327178' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:18.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:18 smithi154 ceph-mon[107038]: pgmap v509: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 2991/86973 objects degraded (3.439%); 700/86973 objects misplaced (0.805%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:48:18.048 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/475327178' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:18.057 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:48:18.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:18.134 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:18.156 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:18 smithi172 ceph-mon[100696]: pgmap v509: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.8 GiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s; 2991/86973 objects degraded (3.439%); 700/86973 objects misplaced (0.805%); 7.9 MiB/s, 126 objects/s recovering 2024-09-13T14:48:18.156 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/475327178' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:48:18.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:48:18.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 21 16 34231 34215 101.811 91.375 0.018655 0.00981616 2024-09-13T14:48:18.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:18.273 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:18.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:18.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:18.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:18.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:18.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:18.665 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:18.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:18.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:18.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:18.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:18.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:18.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:18.991 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:48:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=sqlstore.transactions t=2024-09-13T14:48:18.773961889Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-13T14:48:19.016 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:19.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:19.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:19.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:19.212 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 22 16 35657 35641 101.234 89.125 0.00587417 0.00984769 2024-09-13T14:48:19.217 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:19.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:19.300 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:19 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:19.300 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/475327178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:19.300 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:19 smithi172 ceph-mon[100696]: osdmap e174: 8 total, 8 up, 6 in 2024-09-13T14:48:19.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:19.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:19 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:19.342 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/475327178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:19.342 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:19 smithi154 ceph-mon[102314]: osdmap e174: 8 total, 8 up, 6 in 2024-09-13T14:48:19.342 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:19 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:19.343 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/475327178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:48:19.343 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:19 smithi154 ceph-mon[107038]: osdmap e174: 8 total, 8 up, 6 in 2024-09-13T14:48:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.344 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:19.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:19.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:19.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:19.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:19.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:19.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:19.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:19.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:19 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T14:48:19.339+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e175 definitely_dead 0 2024-09-13T14:48:19.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:19.840 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:19.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:19.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:19.964 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:19.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:20.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:20.083 INFO:teuthology.orchestra.run.smithi154.stderr:marked in osd.5. 2024-09-13T14:48:20.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:20.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:20.212 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 23 16 35717 35701 96.9962 3.75 0.0021088 0.0102665 2024-09-13T14:48:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.216 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:20.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:20.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:20.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:20.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:20.462 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:20 smithi172 ceph-mon[100696]: pgmap v511: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 77 MiB/s wr, 1.24k op/s; 2991/93024 objects degraded (3.215%); 700/93024 objects misplaced (0.752%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:48:20.463 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:20 smithi172 ceph-mon[100696]: osdmap e175: 8 total, 8 up, 6 in 2024-09-13T14:48:20.463 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2176727427' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:48:20.463 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:48:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:20.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:20.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[102314]: pgmap v511: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 77 MiB/s wr, 1.24k op/s; 2991/93024 objects degraded (3.215%); 700/93024 objects misplaced (0.752%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:48:20.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[102314]: osdmap e175: 8 total, 8 up, 6 in 2024-09-13T14:48:20.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2176727427' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:48:20.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:48:20.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[107038]: pgmap v511: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 1.9 GiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 77 MiB/s wr, 1.24k op/s; 2991/93024 objects degraded (3.215%); 700/93024 objects misplaced (0.752%); 8.1 MiB/s, 129 objects/s recovering 2024-09-13T14:48:20.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[107038]: osdmap e175: 8 total, 8 up, 6 in 2024-09-13T14:48:20.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2176727427' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:48:20.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-13T14:48:20.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:20.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:20.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:20.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:20.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:20.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:20.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:20.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:20.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:21.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:21.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:21.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:21.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:21.211 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 24 16 35729 35713 92.9863 0.75 0.00287593 0.0103501 2024-09-13T14:48:21.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:21.333 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:21.374 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:21 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:48:21.374 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:21 smithi172 ceph-mon[100696]: osdmap e176: 8 total, 8 up, 7 in 2024-09-13T14:48:21.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:21 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:48:21.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:21 smithi154 ceph-mon[102314]: osdmap e176: 8 total, 8 up, 7 in 2024-09-13T14:48:21.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:21 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-13T14:48:21.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:21 smithi154 ceph-mon[107038]: osdmap e176: 8 total, 8 up, 7 in 2024-09-13T14:48:21.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:21.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:21.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:21.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:21.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:21.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:21.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:21.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:21.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:21.849 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-13T14:48:21.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:21.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:21.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:22.049 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:22.049 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:22.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:22.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:22.086 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:48:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:48:21] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:48:22.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:22.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:22.212 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 25 16 35762 35746 89.3496 2.0625 0.010498 0.0111274 2024-09-13T14:48:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:22 smithi172 ceph-mon[100696]: pgmap v514: 89 pgs: 4 peering, 4 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s; 2991/104868 objects degraded (2.852%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:48:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:22 smithi172 ceph-mon[100696]: osdmap e177: 8 total, 8 up, 7 in 2024-09-13T14:48:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:22.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:22.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:22.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[102314]: pgmap v514: 89 pgs: 4 peering, 4 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s; 2991/104868 objects degraded (2.852%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:48:22.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[102314]: osdmap e177: 8 total, 8 up, 7 in 2024-09-13T14:48:22.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:22.353 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:22.353 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[107038]: pgmap v514: 89 pgs: 4 peering, 4 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 75 MiB/s wr, 1.19k op/s; 2991/104868 objects degraded (2.852%); 7.4 MiB/s, 118 objects/s recovering 2024-09-13T14:48:22.354 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[107038]: osdmap e177: 8 total, 8 up, 7 in 2024-09-13T14:48:22.354 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:22.354 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:22.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:22.468 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:22.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:22.596 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:22.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:22.745 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:22.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:22.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:22.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:22.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:22.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:23.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:23.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:48:23.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:23.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:23 smithi154 ceph-mon[102314]: osdmap e178: 8 total, 8 up, 7 in 2024-09-13T14:48:23.121 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:23 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1605745510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:23.121 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:48:23.122 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:23.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:23 smithi154 ceph-mon[107038]: osdmap e178: 8 total, 8 up, 7 in 2024-09-13T14:48:23.122 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:23 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1605745510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:23.128 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:23.167 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:48:23.167 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:23.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 26 16 36770 36754 88.3333 63 0.0203887 0.011308 2024-09-13T14:48:23.222 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:48:23.222 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:23.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:23.258 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:48:23.258 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.287 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:23.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:23.293 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:48:23.293 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:23.324 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:48:23.325 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:23.356 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:48:23.356 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:23.386 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:48:23.387 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:23.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:23.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:23.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:23.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:23 smithi172 ceph-mon[100696]: osdmap e178: 8 total, 8 up, 7 in 2024-09-13T14:48:23.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:23 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1605745510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:23.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:23.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:23.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:23.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:23.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:23.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:23.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:23.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:23.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:24.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:24.076 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:24.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:24.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 27 16 37884 37868 87.6402 69.625 0.00647115 0.0114034 2024-09-13T14:48:24.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:24.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:24.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:24.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:24 smithi154 ceph-mon[102314]: pgmap v517: 89 pgs: 4 peering, 5 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 74 MiB/s wr, 1.19k op/s; 6694/107316 objects degraded (6.238%); 22 MiB/s, 350 objects/s recovering 2024-09-13T14:48:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:24 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6694/107316 objects degraded (6.238%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:24.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:24 smithi154 ceph-mon[107038]: pgmap v517: 89 pgs: 4 peering, 5 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 74 MiB/s wr, 1.19k op/s; 6694/107316 objects degraded (6.238%); 22 MiB/s, 350 objects/s recovering 2024-09-13T14:48:24.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:24 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6694/107316 objects degraded (6.238%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:24.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:24.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:24 smithi172 ceph-mon[100696]: pgmap v517: 89 pgs: 4 peering, 5 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.2 GiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 74 MiB/s wr, 1.19k op/s; 6694/107316 objects degraded (6.238%); 22 MiB/s, 350 objects/s recovering 2024-09-13T14:48:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:24 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6694/107316 objects degraded (6.238%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:24.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:24.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:24.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:24.752 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:24.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:48:24.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:24.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:24.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:24.992 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:25.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:25.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:25.108 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:25.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:25.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:25.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 28 16 39012 38996 87.0279 70.5 0.00494595 0.0114846 2024-09-13T14:48:25.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:25.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:25.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:25.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:25.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:25.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:25.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:25.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:25.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:25.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:25.996 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:25.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:26.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:26.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:26.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 29 16 40278 40262 86.7552 79.125 0.00709061 0.0115198 2024-09-13T14:48:26.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:26.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:26.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:48:26.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:26.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:26.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:26 smithi154 ceph-mon[107038]: pgmap v518: 89 pgs: 2 active+recovering+undersized+remapped, 3 peering, 4 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 57 MiB/s wr, 912 op/s; 4973/111372 objects degraded (4.465%); 2251/111372 objects misplaced (2.021%); 26 MiB/s, 405 objects/s recovering 2024-09-13T14:48:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:26 smithi154 ceph-mon[102314]: pgmap v518: 89 pgs: 2 active+recovering+undersized+remapped, 3 peering, 4 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 57 MiB/s wr, 912 op/s; 4973/111372 objects degraded (4.465%); 2251/111372 objects misplaced (2.021%); 26 MiB/s, 405 objects/s recovering 2024-09-13T14:48:26.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:26 smithi172 ceph-mon[100696]: pgmap v518: 89 pgs: 2 active+recovering+undersized+remapped, 3 peering, 4 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 57 MiB/s wr, 912 op/s; 4973/111372 objects degraded (4.465%); 2251/111372 objects misplaced (2.021%); 26 MiB/s, 405 objects/s recovering 2024-09-13T14:48:26.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:26.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:26.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:26.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:26.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:26.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:26.850 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:48:26.850 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:48:26.850 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T14:48:26.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-09-13T14:48:26.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:26.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:26.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:27.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:27.135 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:27.166 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:48:27.166 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:48:27.166 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:48:27.166 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:48:27.166 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:48:27.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 30 16 41475 41459 86.3569 74.8125 0.0124332 0.0115693 2024-09-13T14:48:27.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:27.271 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:27.373 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:27.395 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:27.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:27.519 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:27.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:27.642 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:27.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:27.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:27.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:48:27.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:27.890 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:27.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:27.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:28.005 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:28.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:28.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:28.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 31 16 41696 41680 84.017 13.8125 0.00470124 0.011896 2024-09-13T14:48:28.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:28.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:28.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:28.399 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:28 smithi172 ceph-mon[100696]: pgmap v519: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 33 MiB/s wr, 529 op/s; 7255/113259 objects degraded (6.406%); 4420/113259 objects misplaced (3.903%); 15 MiB/s, 244 objects/s recovering 2024-09-13T14:48:28.400 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:28 smithi172 ceph-mon[100696]: osdmap e179: 8 total, 8 up, 7 in 2024-09-13T14:48:28.400 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:28.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:28 smithi154 ceph-mon[102314]: pgmap v519: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 33 MiB/s wr, 529 op/s; 7255/113259 objects degraded (6.406%); 4420/113259 objects misplaced (3.903%); 15 MiB/s, 244 objects/s recovering 2024-09-13T14:48:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:28 smithi154 ceph-mon[102314]: osdmap e179: 8 total, 8 up, 7 in 2024-09-13T14:48:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:28 smithi154 ceph-mon[107038]: pgmap v519: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 33 MiB/s wr, 529 op/s; 7255/113259 objects degraded (6.406%); 4420/113259 objects misplaced (3.903%); 15 MiB/s, 244 objects/s recovering 2024-09-13T14:48:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:28 smithi154 ceph-mon[107038]: osdmap e179: 8 total, 8 up, 7 in 2024-09-13T14:48:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:28.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:28.569 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:28.610 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:28.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:28.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:28.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:28.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:28.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:28.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:29.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:29.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:29.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:29.185 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:29.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 32 16 42901 42885 83.7447 75.3125 0.0149851 0.011934 2024-09-13T14:48:29.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:29 smithi172 ceph-mon[100696]: osdmap e180: 8 total, 8 up, 7 in 2024-09-13T14:48:29.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:29.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:29 smithi154 ceph-mon[102314]: osdmap e180: 8 total, 8 up, 7 in 2024-09-13T14:48:29.375 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:29 smithi154 ceph-mon[107038]: osdmap e180: 8 total, 8 up, 7 in 2024-09-13T14:48:29.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:29.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:29.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:29.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:29.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:29.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:29.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:29.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:29.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:29.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:30.012 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:30.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:30.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.174 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:48:30.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 33 16 44052 44036 83.3868 71.9375 0.0103241 0.0119862 2024-09-13T14:48:30.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:30.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:30.386 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[102314]: pgmap v522: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 44 MiB/s wr, 698 op/s; 7255/119172 objects degraded (6.088%); 4160/119172 objects misplaced (3.491%); 18 MiB/s, 282 objects/s recovering 2024-09-13T14:48:30.386 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7255/119172 objects degraded (6.088%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:30.386 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/296024555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:30.386 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:30.386 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:30.386 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:30.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[107038]: pgmap v522: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 44 MiB/s wr, 698 op/s; 7255/119172 objects degraded (6.088%); 4160/119172 objects misplaced (3.491%); 18 MiB/s, 282 objects/s recovering 2024-09-13T14:48:30.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7255/119172 objects degraded (6.088%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:30.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/296024555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:30.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:30.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:30.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:30.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:30.409 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:30 smithi172 ceph-mon[100696]: pgmap v522: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 44 MiB/s wr, 698 op/s; 7255/119172 objects degraded (6.088%); 4160/119172 objects misplaced (3.491%); 18 MiB/s, 282 objects/s recovering 2024-09-13T14:48:30.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7255/119172 objects degraded (6.088%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:30.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:30 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/296024555' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:30.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:30 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:48:30.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:30.410 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:30.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:27229 5.2 5:47a3bf38:::benchmark_data_smithi154_144140_object27228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.669459+0000", 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 302.77105117500003, 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.16906344300000001, 2024-09-13T14:48:30.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 27229 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669459+0000", 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669459+0000", 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669457+0000", 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669491+0000", 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3810000000000003e-05 2024-09-13T14:48:30.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669500+0000", 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9339999999999995e-06 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669506+0000", 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0839999999999997e-06 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669547+0000", 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0438999999999997e-05 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669666+0000", 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011913600000000001 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.669880+0000", 2024-09-13T14:48:30.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000214195 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.671121+0000", 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001240653 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.672401+0000", 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012802569999999999 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838431+0000", 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.16602999700000001 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838482+0000", 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0695999999999997e-05 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838522+0000", 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0590999999999998e-05 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:27239 5.2 5:4246bfbf:::benchmark_data_smithi154_144140_object27238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.674859+0000", 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 302.76565141200001, 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.163782447, 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.451 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 27239 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674859+0000", 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674859+0000", 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674857+0000", 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674958+0000", 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101588 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674974+0000", 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5661000000000001e-05 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.674982+0000", 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8970000000000008e-06 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.675031+0000", 2024-09-13T14:48:30.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9191999999999999e-05 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.675172+0000", 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140345 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.675332+0000", 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160089 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676497+0000", 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011657250000000001 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.677801+0000", 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013040370000000001 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838574+0000", 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.160772633 2024-09-13T14:48:30.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838607+0000", 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2879000000000001e-05 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.838641+0000", 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4063000000000002e-05 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:27242 5.2 5:482bba7a:::benchmark_data_smithi154_144140_object27241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:27.676639+0000", 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 302.76387122400001, 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.162555276, 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 27242 2024-09-13T14:48:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676639+0000", 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676639+0000", 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676638+0000", 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676723+0000", 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4969999999999995e-05 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676734+0000", 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1841000000000001e-05 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676742+0000", 2024-09-13T14:48:30.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676790+0000", 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8687999999999999e-05 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.676933+0000", 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000143008 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.677076+0000", 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014246899999999999 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.677906+0000", 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083029999999999996 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.678301+0000", 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000394607 2024-09-13T14:48:30.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.839143+0000", 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.160842136 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.839173+0000", 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0505999999999999e-05 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:27.839194+0000", 2024-09-13T14:48:30.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0752999999999999e-05 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:33386 5.2 5:45c88780:::benchmark_data_smithi154_144140_object33385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:43:31.021828+0000", 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 299.418681757, 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.108195734, 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:48:30.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 33386 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021828+0000", 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021828+0000", 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021827+0000", 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021872+0000", 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5302000000000001e-05 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021888+0000", 2024-09-13T14:48:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5984000000000001e-05 2024-09-13T14:48:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021897+0000", 2024-09-13T14:48:30.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5499999999999995e-06 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.021935+0000", 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8494000000000001e-05 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.022059+0000", 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012427299999999999 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.022200+0000", 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014108999999999999 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.023483+0000", 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001283001 2024-09-13T14:48:30.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.027180+0000", 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036963 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.129924+0000", 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.102744157 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.129979+0000", 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5599000000000001e-05 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:43:31.130024+0000", 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4467e-05 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41401 6.5 6:a0cee5c3:::benchmark_data_smithi154_175640_object41400:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.154326+0000", 2024-09-13T14:48:30.472 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2861837760000001, 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.987267173, 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41401 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154326+0000", 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154326+0000", 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154325+0000", 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154365+0000", 2024-09-13T14:48:30.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9858999999999999e-05 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154373+0000", 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5169999999999999e-06 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154377+0000", 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.386e-06 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154429+0000", 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1792999999999997e-05 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154440+0000", 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1009000000000001e-05 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.136835+0000", 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98239497799999997 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.136984+0000", 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149166 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.137156+0000", 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171724 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141391+0000", 2024-09-13T14:48:30.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042347770000000003 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141397+0000", 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3180000000000003e-06 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141508+0000", 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110745 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141555+0000", 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6873000000000003e-05 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141593+0000", 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8525000000000001e-05 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41409 6.5 6:a7afbe70:::benchmark_data_smithi154_175640_object41408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.154474+0000", 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2860361199999999, 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98730288099999997, 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41409 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154474+0000", 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154474+0000", 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154473+0000", 2024-09-13T14:48:30.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154547+0000", 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4348e-05 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154560+0000", 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2741e-05 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154567+0000", 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.601e-06 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154613+0000", 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6465999999999999e-05 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154626+0000", 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2619999999999999e-05 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.138993+0000", 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98436692299999995 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.139102+0000", 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109354 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.139209+0000", 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107131 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141630+0000", 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024207909999999998 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141637+0000", 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6819999999999997e-06 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141695+0000", 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8863999999999997e-05 2024-09-13T14:48:30.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141740+0000", 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4885999999999997e-05 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141777+0000", 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6353000000000002e-05 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41423 6.5 6:adf64e52:::benchmark_data_smithi154_175640_object41422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.170481+0000", 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2700286310000002, 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97225323200000002, 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41423 2024-09-13T14:48:30.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170481+0000", 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170481+0000", 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170480+0000", 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170552+0000", 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1755999999999995e-05 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170564+0000", 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2017e-05 2024-09-13T14:48:30.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170572+0000", 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2770000000000005e-06 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170631+0000", 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9401999999999998e-05 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170643+0000", 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.188e-05 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.140287+0000", 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96964348 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.140399+0000", 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243600000000001 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.140552+0000", 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000152318 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142583+0000", 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020316010000000001 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142620+0000", 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6758000000000003e-05 2024-09-13T14:48:30.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142667+0000", 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7546999999999997e-05 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142714+0000", 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6428999999999999e-05 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142735+0000", 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0611e-05 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41430 6.5 6:a33d844f:::benchmark_data_smithi154_175640_object41429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.176451+0000", 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2640589260000001, 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98017701400000001, 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.494 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41430 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176451+0000", 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176451+0000", 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176450+0000", 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176568+0000", 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118829 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176585+0000", 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6586e-05 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176592+0000", 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4610000000000002e-06 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176720+0000", 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012720900000000001 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176730+0000", 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-05 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141823+0000", 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96509276600000005 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141911+0000", 2024-09-13T14:48:30.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8411000000000004e-05 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141991+0000", 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0037999999999995e-05 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144236+0000", 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022446940000000002 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154118+0000", 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098819530000000006 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156564+0000", 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002446181 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156602+0000", 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8198000000000003e-05 2024-09-13T14:48:30.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156628+0000", 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5703000000000001e-05 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41449 6.5 6:ab6f9780:::benchmark_data_smithi154_175640_object41448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.192954+0000", 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.247555534, 2024-09-13T14:48:30.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96378176500000001, 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41449 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192954+0000", 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192954+0000", 2024-09-13T14:48:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192953+0000", 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192996+0000", 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3519000000000001e-05 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193008+0000", 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1378e-05 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193014+0000", 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0850000000000002e-06 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193069+0000", 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5018000000000001e-05 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193080+0000", 2024-09-13T14:48:30.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1386000000000001e-05 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142763+0000", 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94968323099999996 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142830+0000", 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7049000000000001e-05 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142908+0000", 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7990000000000004e-05 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146064+0000", 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031560289999999999 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154178+0000", 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081134980000000002 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156666+0000", 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024880240000000001 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156716+0000", 2024-09-13T14:48:30.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9897e-05 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156736+0000", 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0267000000000001e-05 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41467 6.5 6:aa02d758:::benchmark_data_smithi154_175640_object41466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.510 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.208447+0000", 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2320626419999998, 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.948363972, 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41467 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208447+0000", 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208447+0000", 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208445+0000", 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:30.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208491+0000", 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5204999999999998e-05 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208510+0000", 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9436000000000002e-05 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208520+0000", 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0275000000000001e-05 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208572+0000", 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1277999999999999e-05 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208579+0000", 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5159999999999997e-06 2024-09-13T14:48:30.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.143489+0000", 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.93491021900000004 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.143594+0000", 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010509999999999999 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.143678+0000", 2024-09-13T14:48:30.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3837000000000004e-05 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147708+0000", 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040294559999999998 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154210+0000", 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065019739999999998 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156767+0000", 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025577299999999998 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156793+0000", 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5380000000000001e-05 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156811+0000", 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8502e-05 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41494 6.5 6:a01b1af7:::benchmark_data_smithi154_175640_object41493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.224013+0000", 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2164973309999998, 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.93286539300000004, 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.518 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41494 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224013+0000", 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224013+0000", 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224012+0000", 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224087+0000", 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5371999999999996e-05 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224096+0000", 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1260000000000004e-06 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224101+0000", 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.899e-06 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224165+0000", 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3408e-05 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224176+0000", 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1522000000000001e-05 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144255+0000", 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92007880099999995 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144322+0000", 2024-09-13T14:48:30.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6680000000000005e-05 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144399+0000", 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6947999999999996e-05 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147713+0000", 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003314053 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154258+0000", 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065451260000000001 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156839+0000", 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025813770000000002 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156861+0000", 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2056e-05 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156878+0000", 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6735000000000002e-05 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41499 6.5 6:a6ba2bb0:::benchmark_data_smithi154_175640_object41498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.228394+0000", 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.2121159910000001, 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.930512809, 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41499 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.526 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228394+0000", 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228394+0000", 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228392+0000", 2024-09-13T14:48:30.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228458+0000", 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.635e-05 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228469+0000", 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1146e-05 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228476+0000", 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9789999999999996e-06 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228531+0000", 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4364999999999999e-05 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228538+0000", 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9190000000000002e-06 2024-09-13T14:48:30.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.145038+0000", 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91650072000000005 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.145153+0000", 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114875 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.145286+0000", 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133063 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.149861+0000", 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045745200000000003 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156905+0000", 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070438710000000002 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158835+0000", 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019305539999999999 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158871+0000", 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5491999999999998e-05 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158907+0000", 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6086e-05 2024-09-13T14:48:30.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41505 6.5 6:a4d4b87f:::benchmark_data_smithi154_175640_object41504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.234857+0000", 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.205652546, 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92416932399999996, 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41505 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234857+0000", 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234857+0000", 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234856+0000", 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234914+0000", 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8418999999999998e-05 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234927+0000", 2024-09-13T14:48:30.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2394e-05 2024-09-13T14:48:30.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234933+0000", 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1040000000000003e-06 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234982+0000", 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8931999999999998e-05 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234991+0000", 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8680000000000005e-06 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146100+0000", 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91110977800000004 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146177+0000", 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6884000000000005e-05 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146267+0000", 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9871999999999996e-05 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.149867+0000", 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035993420000000002 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156923+0000", 2024-09-13T14:48:30.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070560579999999996 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158959+0000", 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020363640000000001 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.159005+0000", 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6079999999999999e-05 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.159027+0000", 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1668999999999999e-05 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41545 6.5 6:afbdcd81:::benchmark_data_smithi154_175640_object41544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.269203+0000", 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.1713072900000001, 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89307002400000002, 2024-09-13T14:48:30.541 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41545 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269203+0000", 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269203+0000", 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269201+0000", 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269242+0000", 2024-09-13T14:48:30.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0963000000000001e-05 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269254+0000", 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1636e-05 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269259+0000", 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9620000000000001e-06 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269321+0000", 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2120999999999997e-05 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.269332+0000", 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.155e-05 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146971+0000", 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87763862199999998 2024-09-13T14:48:30.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147035+0000", 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3757000000000006e-05 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147118+0000", 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3030999999999998e-05 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.151224+0000", 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041061739999999998 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.159062+0000", 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078385009999999995 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162222+0000", 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031592899999999999 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162250+0000", 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7940999999999999e-05 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162273+0000", 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3108999999999998e-05 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.548 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41561 6.5 6:a211154d:::benchmark_data_smithi154_175640_object41560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.284423+0000", 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.1560868590000002, 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87797212099999999, 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41561 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284423+0000", 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284423+0000", 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284423+0000", 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284495+0000", 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1860999999999995e-05 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284500+0000", 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.062e-06 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284503+0000", 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2200000000000001e-06 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284543+0000", 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9845999999999998e-05 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.284553+0000", 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0397999999999999e-05 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147740+0000", 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86318737899999998 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147812+0000", 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1595999999999997e-05 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147897+0000", 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5272000000000006e-05 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.151235+0000", 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003337811 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.159092+0000", 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078566910000000007 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162322+0000", 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032300390000000001 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162362+0000", 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0207999999999999e-05 2024-09-13T14:48:30.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162395+0000", 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3170000000000003e-05 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41565 6.5 6:add7dea2:::benchmark_data_smithi154_175640_object41564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.287835+0000", 2024-09-13T14:48:30.556 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.1526751200000001, 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87467696699999997, 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41565 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287835+0000", 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287835+0000", 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287833+0000", 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287887+0000", 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3641999999999999e-05 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287897+0000", 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7669999999999998e-06 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287902+0000", 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3499999999999996e-06 2024-09-13T14:48:30.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287952+0000", 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0352999999999997e-05 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287959+0000", 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6460000000000004e-06 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.148652+0000", 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86069265500000003 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.148769+0000", 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117346 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.148923+0000", 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153695 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154124+0000", 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052014779999999998 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.159109+0000", 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049848729999999999 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162438+0000", 2024-09-13T14:48:30.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003328892 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162480+0000", 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1662999999999997e-05 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.162512+0000", 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2076999999999998e-05 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.563 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41566 6.5 6:a9d720bf:::benchmark_data_smithi154_175640_object41565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.287911+0000", 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.152598888, 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87690132200000004, 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41566 2024-09-13T14:48:30.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287911+0000", 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287911+0000", 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287911+0000", 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287954+0000", 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3037999999999997e-05 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287960+0000", 2024-09-13T14:48:30.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1210000000000003e-06 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287963+0000", 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6880000000000001e-06 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.287995+0000", 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1585999999999999e-05 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.288001+0000", 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8119999999999996e-06 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.149900+0000", 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86189893699999998 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.150011+0000", 2024-09-13T14:48:30.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111548 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.150163+0000", 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151977 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154214+0000", 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040507920000000001 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.164629+0000", 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010415115000000001 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.164725+0000", 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5951000000000003e-05 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.164775+0000", 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9437000000000001e-05 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.164812+0000", 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7731000000000002e-05 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41568 6.5 6:aa94799d:::benchmark_data_smithi154_175640_object41567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.290211+0000", 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.1502986009999998, 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87589746700000004, 2024-09-13T14:48:30.571 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41568 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290211+0000", 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290211+0000", 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290211+0000", 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290268+0000", 2024-09-13T14:48:30.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6960999999999999e-05 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290280+0000", 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2911000000000001e-05 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290287+0000", 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6769999999999998e-06 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290325+0000", 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8250999999999997e-05 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.290332+0000", 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6460000000000004e-06 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.151281+0000", 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86094923000000001 2024-09-13T14:48:30.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.151423+0000", 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141784 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.151568+0000", 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014479200000000001 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154218+0000", 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026504250000000001 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.164882+0000", 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010663972000000001 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.166016+0000", 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011341820000000001 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.166071+0000", 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4858999999999998e-05 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.166109+0000", 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7571999999999999e-05 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.578 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41601 6.5 6:a44fbd16:::benchmark_data_smithi154_175640_object41600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.342185+0000", 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.098324877, 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.82406465600000001, 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41601 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342185+0000", 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342185+0000", 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342184+0000", 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342240+0000", 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5390000000000003e-05 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342250+0000", 2024-09-13T14:48:30.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0331e-05 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342255+0000", 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0889999999999999e-06 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342311+0000", 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5482999999999997e-05 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.342322+0000", 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1584e-05 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.152978+0000", 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81065618399999995 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.153089+0000", 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110192 2024-09-13T14:48:30.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.153209+0000", 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011990299999999999 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.155217+0000", 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020081489999999999 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.164931+0000", 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097142789999999993 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.166156+0000", 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012246049999999999 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.166206+0000", 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0705000000000003e-05 2024-09-13T14:48:30.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.166250+0000", 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3452999999999999e-05 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41604 6.5 6:ad905772:::benchmark_data_smithi154_175640_object41603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.347226+0000", 2024-09-13T14:48:30.586 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.0932837819999999, 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.82369809100000002, 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41604 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347226+0000", 2024-09-13T14:48:30.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347226+0000", 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347225+0000", 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347271+0000", 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5617e-05 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347282+0000", 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0977e-05 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347287+0000", 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0900000000000004e-06 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347326+0000", 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8460000000000001e-05 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.347331+0000", 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8930000000000002e-06 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154277+0000", 2024-09-13T14:48:30.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80694552799999997 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154368+0000", 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0697000000000003e-05 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154449+0000", 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1525000000000003e-05 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.155347+0000", 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089765199999999998 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.164971+0000", 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096240300000000004 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.170845+0000", 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058741460000000002 2024-09-13T14:48:30.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.170893+0000", 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7673999999999998e-05 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.170924+0000", 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1581000000000001e-05 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:48:30.594 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:48:30.594 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:48:30.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:30.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:30.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:30.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:30.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:30.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:30.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:31.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:31.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:31.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:31.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:31.213 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 34 16 45290 45274 83.2097 77.375 0.0386467 0.0120091 2024-09-13T14:48:31.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:31.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:31.425 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:31 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:31.425 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:31 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:31.425 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:31 smithi172 ceph-mon[100696]: osdmap e181: 8 total, 8 up, 7 in 2024-09-13T14:48:31.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:48:31.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:31 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:31.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:31 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:31.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:31 smithi154 ceph-mon[102314]: osdmap e181: 8 total, 8 up, 7 in 2024-09-13T14:48:31.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:31 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:48:31.451 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:31 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:48:31.451 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:31 smithi154 ceph-mon[107038]: osdmap e181: 8 total, 8 up, 7 in 2024-09-13T14:48:31.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:31.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:31.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:31.726 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:31.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:31.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:31.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:31.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:32.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:32.046 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:32.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.179 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:32.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:32.180 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:48:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:48:31] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-13T14:48:32.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 35 16 46569 46553 83.1161 79.9375 0.0117342 0.0120257 2024-09-13T14:48:32.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:32.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:32.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:32.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:32 smithi172 ceph-mon[100696]: pgmap v524: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s; 7255/128565 objects degraded (5.643%); 3775/128565 objects misplaced (2.936%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:48:32.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:32.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:32.431 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:32.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[102314]: pgmap v524: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s; 7255/128565 objects degraded (5.643%); 3775/128565 objects misplaced (2.936%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:48:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[107038]: pgmap v524: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.6 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 60 MiB/s wr, 954 op/s; 7255/128565 objects degraded (5.643%); 3775/128565 objects misplaced (2.936%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:48:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:48:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:48:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:32.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:32.577 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:32.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:32.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:32.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:32.841 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:32.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:32.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:32.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:33.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:33.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:33.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:33.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 36 16 47735 47719 82.8314 72.875 0.011272 0.0120663 2024-09-13T14:48:33.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:33.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:33.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:33.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:33.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:33.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:33.631 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:33.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:33.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:33.782 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:48:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:48:33.782 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T14:48:33.782 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:48:33.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:33.903 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:33.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:34.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:34.047 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:34.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:34.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:34.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 37 16 49016 49000 82.7564 80.0625 0.00562132 0.0120774 2024-09-13T14:48:34.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:34.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:34.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:34 smithi154 ceph-mon[102314]: pgmap v525: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s; 7255/132771 objects degraded (5.464%); 3775/132771 objects misplaced (2.843%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:48:34.376 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:34 smithi154 ceph-mon[107038]: pgmap v525: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s; 7255/132771 objects degraded (5.464%); 3775/132771 objects misplaced (2.843%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:48:34.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:34.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:48:34.446 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:34 smithi172 ceph-mon[100696]: pgmap v525: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s; 7255/132771 objects degraded (5.464%); 3775/132771 objects misplaced (2.843%); 6.7 MiB/s, 107 objects/s recovering 2024-09-13T14:48:34.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:34.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:34.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:34.697 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:34.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:34.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:34.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:34.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:34.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:35.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:35.027 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:35.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:35.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:35.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:35.194 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:48:35.194 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:35.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 38 16 50224 50208 82.5653 75.5 0.00508853 0.0121039 2024-09-13T14:48:35.235 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:48:35.235 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:35.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:35.275 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:48:35.276 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:35.276 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7255/132771 objects degraded (5.464%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:35.276 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3639815129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:35.276 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7255/132771 objects degraded (5.464%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:35.276 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3639815129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:35.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:35.333 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:48:35.334 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:35.371 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:48:35.371 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:35.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:35.404 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:48:35.404 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:35.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:35.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:35.442 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:48:35.443 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:35.476 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:48:35.476 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:35.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7255/132771 objects degraded (5.464%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:35.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3639815129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:35.543 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:35.568 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:35.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:35.696 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:35.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:35.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:35.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:35.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:35.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:36.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:36.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:36.216 INFO:teuthology.orchestra.run.smithi154.stdout:moved 23 / 267 (8.61423%) 2024-09-13T14:48:36.216 INFO:teuthology.orchestra.run.smithi154.stdout:avg 38.1429 2024-09-13T14:48:36.216 INFO:teuthology.orchestra.run.smithi154.stdout:stddev 6.42381 -> 5.91435 (expected baseline 5.71786) 2024-09-13T14:48:36.216 INFO:teuthology.orchestra.run.smithi154.stdout:min osd.5 with 28 -> 37 pgs (0.734082 -> 0.970037 * mean) 2024-09-13T14:48:36.216 INFO:teuthology.orchestra.run.smithi154.stdout:max osd.1 with 47 -> 49 pgs (1.23221 -> 1.28464 * mean) 2024-09-13T14:48:36.216 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:oload 110 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:max_change 3 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:max_change_osds 4 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:average_utilization 0.0160 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:overload_utilization 0.0177 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 weight 1.0000 -> 0.6541 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 weight 1.0000 -> 0.6552 2024-09-13T14:48:36.217 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 weight 1.0000 -> 0.8046 2024-09-13T14:48:36.217 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 39 16 51387 51371 82.3118 72.6875 0.0197704 0.0121432 2024-09-13T14:48:36.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:36.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:36.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:36.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:36.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:48:36.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:36 smithi154 ceph-mon[102314]: pgmap v526: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s; 7255/143247 objects degraded (5.065%); 3507/143247 objects misplaced (2.448%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:48:36.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:36 smithi154 ceph-mon[102314]: from='client.54673 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:48:36.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'42870','3':'42942','6':'52727'}"} : dispatch 2024-09-13T14:48:36.452 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:36 smithi154 ceph-mon[107038]: pgmap v526: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s; 7255/143247 objects degraded (5.065%); 3507/143247 objects misplaced (2.448%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:48:36.452 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:36 smithi154 ceph-mon[107038]: from='client.54673 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:48:36.452 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'42870','3':'42942','6':'52727'}"} : dispatch 2024-09-13T14:48:36.475 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:36 smithi172 ceph-mon[100696]: pgmap v526: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s; 7255/143247 objects degraded (5.065%); 3507/143247 objects misplaced (2.448%); 7.3 MiB/s, 117 objects/s recovering 2024-09-13T14:48:36.475 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:36 smithi172 ceph-mon[100696]: from='client.54673 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:48:36.475 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'2':'42870','3':'42942','6':'52727'}"} : dispatch 2024-09-13T14:48:36.483 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:36.483 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:36.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:36.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:36.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:36.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:36.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:36.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:36.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:36.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:36.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:37.004 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:37.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:37.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:48:37.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:37.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:48:37.214146+0000 min lat: 0.001048 max lat: 2.09981 avg lat: 0.012145 2024-09-13T14:48:37.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:48:37.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 40 16 51400 51384 80.2745 0.8125 0.00319737 0.012145 2024-09-13T14:48:37.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:37.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:37.335 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:48:37.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:37.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:37.456 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'42870','3':'42942','6':'52727'}"}]': finished 2024-09-13T14:48:37.457 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:37 smithi172 ceph-mon[100696]: osdmap e182: 8 total, 8 up, 7 in 2024-09-13T14:48:37.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'42870','3':'42942','6':'52727'}"}]': finished 2024-09-13T14:48:37.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:37 smithi154 ceph-mon[102314]: osdmap e182: 8 total, 8 up, 7 in 2024-09-13T14:48:37.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'2':'42870','3':'42942','6':'52727'}"}]': finished 2024-09-13T14:48:37.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:37 smithi154 ceph-mon[107038]: osdmap e182: 8 total, 8 up, 7 in 2024-09-13T14:48:37.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:37.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:37.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:37.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:37.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:37.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:37.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:37.880 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:37.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:37.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:38.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:38.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:38.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:38.216 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 41 16 51400 51384 78.3167 0 - 0.012145 2024-09-13T14:48:38.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:38.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:38.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:38.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:38.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:38 smithi172 ceph-mon[100696]: pgmap v528: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 7255/146631 objects degraded (4.948%); 3507/146631 objects misplaced (2.392%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T14:48:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:38 smithi172 ceph-mon[100696]: osdmap e183: 8 total, 8 up, 7 in 2024-09-13T14:48:38.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:38.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:38.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:38 smithi154 ceph-mon[102314]: pgmap v528: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 7255/146631 objects degraded (4.948%); 3507/146631 objects misplaced (2.392%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T14:48:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:38 smithi154 ceph-mon[102314]: osdmap e183: 8 total, 8 up, 7 in 2024-09-13T14:48:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:38 smithi154 ceph-mon[107038]: pgmap v528: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 7255/146631 objects degraded (4.948%); 3507/146631 objects misplaced (2.392%); 5.1 MiB/s, 81 objects/s recovering 2024-09-13T14:48:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:38 smithi154 ceph-mon[107038]: osdmap e183: 8 total, 8 up, 7 in 2024-09-13T14:48:38.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:48:38.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:38.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:38.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:38.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:38.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:39.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:39.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:39.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:39.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:39.161 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:39.161 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:39.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 42 16 51535 51519 76.653 4.21875 0.00299324 0.0127488 2024-09-13T14:48:39.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:39.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:39.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:39.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:39.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:39.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:39.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:39 smithi154 ceph-mon[102314]: osdmap e184: 8 total, 8 up, 7 in 2024-09-13T14:48:39.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:39 smithi154 ceph-mon[107038]: osdmap e184: 8 total, 8 up, 7 in 2024-09-13T14:48:39.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:39 smithi172 ceph-mon[100696]: osdmap e184: 8 total, 8 up, 7 in 2024-09-13T14:48:39.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:39.526 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:39.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:39.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:39.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:39.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:39.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:48:39.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:39.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:39.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:40.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:40.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:40.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:40.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:40.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 43 16 51535 51519 74.8705 0 - 0.0127488 2024-09-13T14:48:40.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:40.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:40.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:40.392 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:40 smithi172 ceph-mon[100696]: pgmap v531: 89 pgs: 2 remapped+peering, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 61 MiB/s wr, 974 op/s; 5107/150312 objects degraded (3.398%); 5357/150312 objects misplaced (3.564%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:48:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 5107/150312 objects degraded (3.398%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:40.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:40.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:40.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:40.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:40 smithi154 ceph-mon[102314]: pgmap v531: 89 pgs: 2 remapped+peering, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 61 MiB/s wr, 974 op/s; 5107/150312 objects degraded (3.398%); 5357/150312 objects misplaced (3.564%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:48:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 5107/150312 objects degraded (3.398%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:40.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:40 smithi154 ceph-mon[107038]: pgmap v531: 89 pgs: 2 remapped+peering, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 61 MiB/s wr, 974 op/s; 5107/150312 objects degraded (3.398%); 5357/150312 objects misplaced (3.564%); 5.9 MiB/s, 94 objects/s recovering 2024-09-13T14:48:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 5107/150312 objects degraded (3.398%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:40.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:40.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:40.824 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:48:40.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-13T14:48:40.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:40.868 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:40.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:40.897 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:40.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:40.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:41.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:41.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:41.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:41.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 44 16 51535 51519 73.1691 0 - 0.0127488 2024-09-13T14:48:41.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:41.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:41.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:41.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:41.499 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:41.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:41.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:41.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:41.747 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:41.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:41.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:41.891 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:41.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:41.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:42.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:42.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:42.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:42.213 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:48:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:48:41] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-13T14:48:42.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 45 16 51535 51519 71.5433 0 - 0.0127488 2024-09-13T14:48:42.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:42.263 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:42.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:42.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:42 smithi154 ceph-mon[102314]: pgmap v532: 89 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 32 MiB/s wr, 515 op/s; 14708/152529 objects degraded (9.643%); 5357/152529 objects misplaced (3.512%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:48:42.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:42 smithi154 ceph-mon[107038]: pgmap v532: 89 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 32 MiB/s wr, 515 op/s; 14708/152529 objects degraded (9.643%); 5357/152529 objects misplaced (3.512%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:48:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:42 smithi172 ceph-mon[100696]: pgmap v532: 89 pgs: 1 active+recovery_wait+remapped, 2 remapped+peering, 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 32 MiB/s wr, 515 op/s; 14708/152529 objects degraded (9.643%); 5357/152529 objects misplaced (3.512%); 3.1 MiB/s, 49 objects/s recovering 2024-09-13T14:48:42.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:42.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:42.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:42.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:42.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:42.804 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:42.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:42.947 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:42.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:43.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.088 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:43.214 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 46 16 52411 52395 71.1781 13.6875 0.0154537 0.0140429 2024-09-13T14:48:43.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:43.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:43.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:43.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.480 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:43.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:43.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:43.895 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:43.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:43.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:44.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:44.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:44.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:44.215 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 47 16 53431 53415 71.02 63.75 0.00603105 0.0140736 2024-09-13T14:48:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:43 smithi154 ceph-mon[102314]: pgmap v533: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 26 MiB/s wr, 414 op/s; 33794/154965 objects degraded (21.808%); 2947/154965 objects misplaced (1.902%); 14 MiB/s, 216 objects/s recovering 2024-09-13T14:48:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:44.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:43 smithi154 ceph-mon[107038]: pgmap v533: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 26 MiB/s wr, 414 op/s; 33794/154965 objects degraded (21.808%); 2947/154965 objects misplaced (1.902%); 14 MiB/s, 216 objects/s recovering 2024-09-13T14:48:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:44.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:43 smithi172 ceph-mon[100696]: pgmap v533: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 26 MiB/s wr, 414 op/s; 33794/154965 objects degraded (21.808%); 2947/154965 objects misplaced (1.902%); 14 MiB/s, 216 objects/s recovering 2024-09-13T14:48:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:44.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:44.306 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:44.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:44.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:44.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:44.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:44.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:44.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:44.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:44.850 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:44.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:44.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:44.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:45.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:45.215 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 48 16 54405 54389 70.8085 60.875 0.0249576 0.0141167 2024-09-13T14:48:45.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:45.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 33794/154965 objects degraded (21.808%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:45.264 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:45.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:45.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.400 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:45.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:45.448 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 33794/154965 objects degraded (21.808%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:45.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 33794/154965 objects degraded (21.808%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:45.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:45.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:45.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:45.672 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:45.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:45.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:45.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:45.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:45.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:45.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:46.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:46.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:46.143 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:46.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:46.216 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 49 16 55475 55459 70.7262 66.875 0.0072675 0.0141333 2024-09-13T14:48:46.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:45 smithi172 ceph-mon[100696]: pgmap v534: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 25 MiB/s wr, 401 op/s; 33794/156282 objects degraded (21.624%); 2947/156282 objects misplaced (1.886%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:48:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:46.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:46.253 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:45 smithi154 ceph-mon[107038]: pgmap v534: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 25 MiB/s wr, 401 op/s; 33794/156282 objects degraded (21.624%); 2947/156282 objects misplaced (1.886%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:48:46.253 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:45 smithi154 ceph-mon[102314]: pgmap v534: 89 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 25 MiB/s wr, 401 op/s; 33794/156282 objects degraded (21.624%); 2947/156282 objects misplaced (1.886%); 11 MiB/s, 181 objects/s recovering 2024-09-13T14:48:46.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:46.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:46.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:46.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:46.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:46.678 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:46.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:46.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:46.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:46.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:46.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:47.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:47.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:47.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:47.216 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 50 16 56540 56524 70.6428 66.5625 0.0245822 0.0141488 2024-09-13T14:48:47.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:47.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:47.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:47.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:47.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:47.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:47.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:47.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:47.654 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:47.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:47.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:47.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:47.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:47.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:48.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:47 smithi154 ceph-mon[102314]: pgmap v535: 89 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 24 MiB/s wr, 391 op/s; 33794/158043 objects degraded (21.383%); 2947/158043 objects misplaced (1.865%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:48:48.019 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:47 smithi154 ceph-mon[107038]: pgmap v535: 89 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 24 MiB/s wr, 391 op/s; 33794/158043 objects degraded (21.383%); 2947/158043 objects misplaced (1.865%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:48:48.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:48.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:48.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:48.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:48.216 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 51 16 57486 57470 70.4169 59.125 0.0131305 0.0141941 2024-09-13T14:48:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:48.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:47 smithi172 ceph-mon[100696]: pgmap v535: 89 pgs: 1 active+recovery_wait+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 24 MiB/s wr, 391 op/s; 33794/158043 objects degraded (21.383%); 2947/158043 objects misplaced (1.865%); 9.3 MiB/s, 149 objects/s recovering 2024-09-13T14:48:48.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:48.331 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:48:48.331 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:48:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:48.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:48.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:48.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:48.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:48.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:48.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:48.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:48.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:48.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:48.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:48.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:49.003 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:48:49.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:49.038 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:49.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:49.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:49.216 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 52 16 58454 58438 70.2261 60.5 0.0237181 0.014233 2024-09-13T14:48:49.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3241298345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:49.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:49.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:49.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3241298345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:49.351 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3241298345' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:48:49.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:49.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:49.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:49.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:49.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:49.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:49.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:49.863 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:49.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:49.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:50.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:50.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:50.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:50.188 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:50.216 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 53 16 59519 59503 70.1569 66.5625 0.0167047 0.0142459 2024-09-13T14:48:50.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:48:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:49 smithi172 ceph-mon[100696]: pgmap v536: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 13 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 33792/167574 objects degraded (20.165%); 2499/167574 objects misplaced (1.491%); 9.4 MiB/s, 149 objects/s recovering 2024-09-13T14:48:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:49 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:49 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3241298345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:49 smithi172 ceph-mon[100696]: osdmap e185: 8 total, 8 up, 7 in 2024-09-13T14:48:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:49 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 33792/167574 objects degraded (20.165%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:50.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:50.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:50.315 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[102314]: pgmap v536: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 13 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 33792/167574 objects degraded (20.165%); 2499/167574 objects misplaced (1.491%); 9.4 MiB/s, 149 objects/s recovering 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3241298345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[102314]: osdmap e185: 8 total, 8 up, 7 in 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 33792/167574 objects degraded (20.165%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[107038]: pgmap v536: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 13 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 35 MiB/s wr, 556 op/s; 33792/167574 objects degraded (20.165%); 2499/167574 objects misplaced (1.491%); 9.4 MiB/s, 149 objects/s recovering 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3241298345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:48:50.316 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[107038]: osdmap e185: 8 total, 8 up, 7 in 2024-09-13T14:48:50.317 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:49 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 33792/167574 objects degraded (20.165%), 13 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:50.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:50.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:50.520 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:50.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:50.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:50.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:50.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:50.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:50.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:50.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:50.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:51.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:51.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:51.218 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 54 16 60429 60413 69.9084 56.875 0.020174 0.0142804 2024-09-13T14:48:51.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:51.252 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:51.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:51.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:51.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:51.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:51.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:51.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:51.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:51.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:51.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:51.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:51.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:51.998 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:51.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:52.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.134 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:52.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:52.219 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 55 16 60683 60667 68.926 15.875 0.0121901 0.0145024 2024-09-13T14:48:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:52 smithi154 ceph-mon[102314]: pgmap v538: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 13 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 43 MiB/s wr, 693 op/s; 33792/173352 objects degraded (19.493%); 2499/173352 objects misplaced (1.442%); 10 MiB/s, 160 objects/s recovering 2024-09-13T14:48:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:52 smithi154 ceph-mon[102314]: osdmap e186: 8 total, 8 up, 7 in 2024-09-13T14:48:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:52 smithi154 ceph-mon[107038]: pgmap v538: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 13 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 43 MiB/s wr, 693 op/s; 33792/173352 objects degraded (19.493%); 2499/173352 objects misplaced (1.442%); 10 MiB/s, 160 objects/s recovering 2024-09-13T14:48:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:52 smithi154 ceph-mon[107038]: osdmap e186: 8 total, 8 up, 7 in 2024-09-13T14:48:52.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:48:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:48:51] "GET /metrics HTTP/1.1" 200 36583 "" "Prometheus/2.51.0" 2024-09-13T14:48:52.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:52.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:52.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:52.412 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:52 smithi172 ceph-mon[100696]: pgmap v538: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 13 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 43 MiB/s wr, 693 op/s; 33792/173352 objects degraded (19.493%); 2499/173352 objects misplaced (1.442%); 10 MiB/s, 160 objects/s recovering 2024-09-13T14:48:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:52 smithi172 ceph-mon[100696]: osdmap e186: 8 total, 8 up, 7 in 2024-09-13T14:48:52.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:52.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:52.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:52.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:52.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:52.844 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:52.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:52.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:52.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:53.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:53.103 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:53.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:48:53.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:53.219 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 56 16 61560 61544 68.673 54.8125 0.0168011 0.0145528 2024-09-13T14:48:53.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.256 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:53.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:53.275 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:53 smithi154 ceph-mon[102314]: osdmap e187: 8 total, 8 up, 7 in 2024-09-13T14:48:53.275 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:53 smithi154 ceph-mon[107038]: osdmap e187: 8 total, 8 up, 7 in 2024-09-13T14:48:53.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:53.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:53 smithi172 ceph-mon[100696]: osdmap e187: 8 total, 8 up, 7 in 2024-09-13T14:48:53.494 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:48:53.494 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:48:53.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:53.538 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:48:53.538 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:48:53.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:53.581 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:48:53.581 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:48:53.635 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:48:53.636 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:48:53.676 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:48:53.677 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:48:53.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:53.700 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:53.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:53.709 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:48:53.709 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:48:53.735 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:53.758 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:48:53.758 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:48:53.799 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:48:53.800 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:48:53.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:53.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:53.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:53.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:54.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:54.059 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:54 smithi154 ceph-mon[102314]: pgmap v541: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 30549/179673 objects degraded (17.003%); 5594/179673 objects misplaced (3.113%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:48:54.059 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2896456787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:54.060 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:54 smithi154 ceph-mon[107038]: pgmap v541: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 30549/179673 objects degraded (17.003%); 5594/179673 objects misplaced (3.113%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:48:54.060 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2896456787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:54.060 INFO:teuthology.orchestra.run.smithi154.stdout:moved 0 / 267 (0%) 2024-09-13T14:48:54.060 INFO:teuthology.orchestra.run.smithi154.stdout:avg 38.1429 2024-09-13T14:48:54.060 INFO:teuthology.orchestra.run.smithi154.stdout:stddev 5.91435 -> 5.91435 (expected baseline 5.71786) 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:min osd.2 with 27 -> 27 pgs (0.707865 -> 0.707865 * mean) 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:max osd.1 with 49 -> 49 pgs (1.28464 -> 1.28464 * mean) 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:oload 1000 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:max_change 0.05 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:max_change_osds 4 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:average_utilization 0.0198 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:overload_utilization 0.1976 2024-09-13T14:48:54.061 INFO:teuthology.orchestra.run.smithi154.stderr:no change 2024-09-13T14:48:54.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:54.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:54.219 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 57 16 62200 62184 68.17 40 0.0149659 0.0146616 2024-09-13T14:48:54.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:54.263 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:54.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:54.397 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:54 smithi172 ceph-mon[100696]: pgmap v541: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 12 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 30549/179673 objects degraded (17.003%); 5594/179673 objects misplaced (3.113%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:48:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2896456787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:48:54.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:54.547 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:54.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:54.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:54.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:54.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:54.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:54.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:54.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:55.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:55.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:55.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:55.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:55.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:55.219 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 58 16 62982 62966 67.8373 48.875 0.0113743 0.0147151 2024-09-13T14:48:55.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:55.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:55.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:55.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:55.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:55 smithi154 ceph-mon[102314]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:48:55.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 30549/179673 objects degraded (17.003%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:55.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:55 smithi154 ceph-mon[107038]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:48:55.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 30549/179673 objects degraded (17.003%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:55.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:55 smithi172 ceph-mon[100696]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:48:55.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 30549/179673 objects degraded (17.003%), 12 pgs degraded (PG_DEGRADED) 2024-09-13T14:48:55.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:55.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:55.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:55.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:55.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:55.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:55.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:55.950 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:55.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:56.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:56.083 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:56.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:56.220 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 59 16 63162 63146 66.8782 11.25 0.0164335 0.014946 2024-09-13T14:48:56.226 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:56.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:56.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:56.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:56 smithi154 ceph-mon[102314]: pgmap v542: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 55 MiB/s wr, 872 op/s; 29982/183276 objects degraded (16.359%); 5351/183276 objects misplaced (2.920%); 9.9 MiB/s, 159 objects/s recovering 2024-09-13T14:48:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:56 smithi154 ceph-mon[102314]: osdmap e188: 8 total, 8 up, 7 in 2024-09-13T14:48:56.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:56 smithi154 ceph-mon[107038]: pgmap v542: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 55 MiB/s wr, 872 op/s; 29982/183276 objects degraded (16.359%); 5351/183276 objects misplaced (2.920%); 9.9 MiB/s, 159 objects/s recovering 2024-09-13T14:48:56.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:56 smithi154 ceph-mon[107038]: osdmap e188: 8 total, 8 up, 7 in 2024-09-13T14:48:56.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:56.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:56 smithi172 ceph-mon[100696]: pgmap v542: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 55 MiB/s wr, 872 op/s; 29982/183276 objects degraded (16.359%); 5351/183276 objects misplaced (2.920%); 9.9 MiB/s, 159 objects/s recovering 2024-09-13T14:48:56.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:56 smithi172 ceph-mon[100696]: osdmap e188: 8 total, 8 up, 7 in 2024-09-13T14:48:56.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:56.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:56.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:56.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:56.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:56.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:56.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:56.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:56.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:57.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:57.052 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:57.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:57.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:57.220 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:48:57.219806+0000 min lat: 0.001048 max lat: 6.22442 avg lat: 0.0149717 2024-09-13T14:48:57.220 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:48:57.220 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 60 16 64119 64103 66.7604 59.8125 0.0119028 0.0149717 2024-09-13T14:48:57.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:57.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:57.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:57.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:57 smithi154 ceph-mon[102314]: osdmap e189: 8 total, 8 up, 7 in 2024-09-13T14:48:57.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:57 smithi154 ceph-mon[107038]: osdmap e189: 8 total, 8 up, 7 in 2024-09-13T14:48:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:57 smithi172 ceph-mon[100696]: osdmap e189: 8 total, 8 up, 7 in 2024-09-13T14:48:57.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:57.571 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-13T14:48:57.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-13T14:48:57.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:57.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.742 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:57.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:57.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:57.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:58.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:58.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:48:58.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:48:58.220 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 61 16 65046 65030 66.6158 57.9375 0.0164613 0.0150052 2024-09-13T14:48:58.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:58.296 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:58.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:58 smithi154 ceph-mon[102314]: pgmap v545: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 42 MiB/s wr, 668 op/s; 29982/185223 objects degraded (16.187%); 5351/185223 objects misplaced (2.889%); 10 MiB/s, 161 objects/s recovering 2024-09-13T14:48:58.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:58.370 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:48:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:58.371 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:58 smithi154 ceph-mon[107038]: pgmap v545: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 42 MiB/s wr, 668 op/s; 29982/185223 objects degraded (16.187%); 5351/185223 objects misplaced (2.889%); 10 MiB/s, 161 objects/s recovering 2024-09-13T14:48:58.371 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:58.372 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:48:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:58.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:48:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:58.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:48:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:58 smithi172 ceph-mon[100696]: pgmap v545: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 42 MiB/s wr, 668 op/s; 29982/185223 objects degraded (16.187%); 5351/185223 objects misplaced (2.889%); 10 MiB/s, 161 objects/s recovering 2024-09-13T14:48:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:48:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:48:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:48:58.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:58.590 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:58.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:58.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:58.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:48:58.864 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:48:58.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:48:58.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:58.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:48:59.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:59.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:59.220 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 62 16 66013 65997 66.516 60.4375 0.0212146 0.0150278 2024-09-13T14:48:59.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:48:59.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:48:59.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:48:59.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:48:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:48:59.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:59.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:59.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:48:59.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:48:59.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:48:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:48:59.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:48:59.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:48:59.875 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:48:59.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:00.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:00.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:00.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:00.229 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 63 16 66939 66923 66.3693 57.875 0.00751425 0.0150598 2024-09-13T14:49:00.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:00.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:00.303 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:00 smithi154 ceph-mon[102314]: pgmap v546: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s; 29982/191697 objects degraded (15.640%); 4969/191697 objects misplaced (2.592%); 12 MiB/s, 193 objects/s recovering 2024-09-13T14:49:00.303 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 29982/191697 objects degraded (15.640%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:00.303 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:00 smithi154 ceph-mon[107038]: pgmap v546: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s; 29982/191697 objects degraded (15.640%); 4969/191697 objects misplaced (2.592%); 12 MiB/s, 193 objects/s recovering 2024-09-13T14:49:00.304 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 29982/191697 objects degraded (15.640%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:00.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:00.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:00.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:00 smithi172 ceph-mon[100696]: pgmap v546: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s; 29982/191697 objects degraded (15.640%); 4969/191697 objects misplaced (2.592%); 12 MiB/s, 193 objects/s recovering 2024-09-13T14:49:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 29982/191697 objects degraded (15.640%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:00.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:00.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:00.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:00.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:00.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:00.861 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:00.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:00.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:01.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:01.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:01.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:01.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:01.229 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 64 16 67806 67790 66.179 54.1875 0.0160735 0.0151029 2024-09-13T14:49:01.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:01.259 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:01.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:01.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:01.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:01.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:01.549 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:01.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:01.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:01.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:01.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:01.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:01.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:01.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:02.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:02.106 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:49:02.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:02.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:02 smithi154 ceph-mon[102314]: pgmap v547: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 43 MiB/s wr, 684 op/s; 29982/196095 objects degraded (15.290%); 4969/196095 objects misplaced (2.534%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T14:49:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2626683186' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:02 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:49:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:49:01] "GET /metrics HTTP/1.1" 200 36585 "" "Prometheus/2.51.0" 2024-09-13T14:49:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:02 smithi154 ceph-mon[107038]: pgmap v547: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 43 MiB/s wr, 684 op/s; 29982/196095 objects degraded (15.290%); 4969/196095 objects misplaced (2.534%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T14:49:02.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2626683186' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:02 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 65 16 68646 68630 65.9685 52.5 0.0179843 0.0151523 2024-09-13T14:49:02.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:02.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:02.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:02 smithi172 ceph-mon[100696]: pgmap v547: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 43 MiB/s wr, 684 op/s; 29982/196095 objects degraded (15.290%); 4969/196095 objects misplaced (2.534%); 9.3 MiB/s, 148 objects/s recovering 2024-09-13T14:49:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2626683186' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:02 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:02.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:02.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:02.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:02.676 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:02.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:02.807 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:02.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:02.934 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:02.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:03.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:03.201 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:03.229 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 66 16 69467 69451 65.7464 51.3125 0.0113123 0.0152036 2024-09-13T14:49:03.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:03 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:03 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:03 smithi154 ceph-mon[102314]: osdmap e190: 8 total, 8 up, 7 in 2024-09-13T14:49:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:03 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:03 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:03 smithi154 ceph-mon[107038]: osdmap e190: 8 total, 8 up, 7 in 2024-09-13T14:49:03.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:03 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:03 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:03 smithi172 ceph-mon[100696]: osdmap e190: 8 total, 8 up, 7 in 2024-09-13T14:49:03.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:03.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:03.723 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:03.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:03.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:03.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:03.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:03.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:03.992 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:49:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:49:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:49:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:49:03.993 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:49:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.005 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:04.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:04.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:04.144 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:04.229 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 67 16 70265 70249 65.5095 49.875 0.0240162 0.015258 2024-09-13T14:49:04.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:04.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:04.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:04.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:04 smithi154 ceph-mon[102314]: pgmap v549: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 823 op/s; 29982/202767 objects degraded (14.786%); 4757/202767 objects misplaced (2.346%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T14:49:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:04 smithi154 ceph-mon[107038]: pgmap v549: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 823 op/s; 29982/202767 objects degraded (14.786%); 4757/202767 objects misplaced (2.346%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T14:49:04.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:04 smithi172 ceph-mon[100696]: pgmap v549: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 823 op/s; 29982/202767 objects degraded (14.786%); 4757/202767 objects misplaced (2.346%); 4.7 MiB/s, 75 objects/s recovering 2024-09-13T14:49:04.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:04.548 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:04.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:04.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:04.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:04.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:04.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:04.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:04.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:05.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:05.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:05.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:05.229 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 68 16 71155 71139 65.3641 55.625 0.0179627 0.0152928 2024-09-13T14:49:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:05.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:05.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:05.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:05.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 29982/202767 objects degraded (14.786%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:05.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 29982/202767 objects degraded (14.786%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:05.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:05.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 29982/202767 objects degraded (14.786%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:05.499 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:05.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:05.629 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:05.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:05.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:05.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:05.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:05.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:06.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:06.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:06.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:06.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:06.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 69 16 71970 71954 65.155 50.9375 0.00305239 0.0153329 2024-09-13T14:49:06.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:06.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:06.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:06.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:06 smithi154 ceph-mon[102314]: pgmap v550: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 809 op/s; 29982/206799 objects degraded (14.498%); 4757/206799 objects misplaced (2.300%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:49:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:06 smithi154 ceph-mon[107038]: pgmap v550: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 809 op/s; 29982/206799 objects degraded (14.498%); 4757/206799 objects misplaced (2.300%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:49:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.485 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:06.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:06.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:06 smithi172 ceph-mon[100696]: pgmap v550: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 809 op/s; 29982/206799 objects degraded (14.498%); 4757/206799 objects misplaced (2.300%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T14:49:06.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:06.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:06.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:06.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:06.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:06.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:06.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:07.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:07.046 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:07.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:07.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:07.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 70 16 72862 72846 65.0205 55.75 0.0143256 0.0153735 2024-09-13T14:49:07.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:07.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:07.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.485 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:07.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:07.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:07.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:07.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:07.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:07.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:07.908 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:07.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:07.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:08.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:08.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:49:08.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:08.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:08.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 71 16 73759 73743 64.8943 56.0625 0.00936896 0.0154029 2024-09-13T14:49:08.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:08.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:08.460 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:08 smithi172 ceph-mon[100696]: pgmap v551: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 50 MiB/s wr, 798 op/s; 29982/209184 objects degraded (14.333%); 4757/209184 objects misplaced (2.274%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:49:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:08 smithi154 ceph-mon[102314]: pgmap v551: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 50 MiB/s wr, 798 op/s; 29982/209184 objects degraded (14.333%); 4757/209184 objects misplaced (2.274%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:49:08.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:08 smithi154 ceph-mon[107038]: pgmap v551: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 50 MiB/s wr, 798 op/s; 29982/209184 objects degraded (14.333%); 4757/209184 objects misplaced (2.274%); 3.7 MiB/s, 59 objects/s recovering 2024-09-13T14:49:08.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:08.540 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:08.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:08.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:08.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:08.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:08.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:08.916 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:08.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:09.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:09.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:09.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:09.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:09.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 72 16 74580 74564 64.7057 51.3125 0.0249025 0.0154458 2024-09-13T14:49:09.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:49:09.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:09.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:09.469 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:09.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:09.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:09.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:09.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:09.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:09.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:09.894 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:09.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:09.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:10.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:10.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:10.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:10 smithi154 ceph-mon[102314]: pgmap v552: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 53 MiB/s wr, 847 op/s; 29982/217131 objects degraded (13.808%); 4560/217131 objects misplaced (2.100%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:49:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 29982/217131 objects degraded (13.808%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:10.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:10 smithi154 ceph-mon[107038]: pgmap v552: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 53 MiB/s wr, 847 op/s; 29982/217131 objects degraded (13.808%); 4560/217131 objects misplaced (2.100%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:49:10.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 29982/217131 objects degraded (13.808%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:10.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 73 16 75379 75363 64.5034 49.9375 0.00745542 0.0154961 2024-09-13T14:49:10.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:10.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:10.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:10.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:10 smithi172 ceph-mon[100696]: pgmap v552: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 53 MiB/s wr, 847 op/s; 29982/217131 objects degraded (13.808%); 4560/217131 objects misplaced (2.100%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:49:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 29982/217131 objects degraded (13.808%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:10.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:10.622 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:10.637 INFO:teuthology.orchestra.run.smithi154.stdout:moved 0 / 267 (0%) 2024-09-13T14:49:10.637 INFO:teuthology.orchestra.run.smithi154.stdout:avg 38.1429 2024-09-13T14:49:10.637 INFO:teuthology.orchestra.run.smithi154.stdout:stddev 5.91435 -> 5.91435 (expected baseline 5.71786) 2024-09-13T14:49:10.637 INFO:teuthology.orchestra.run.smithi154.stdout:min osd.2 with 27 -> 27 pgs (0.707865 -> 0.707865 * mean) 2024-09-13T14:49:10.638 INFO:teuthology.orchestra.run.smithi154.stdout:max osd.1 with 49 -> 49 pgs (1.28464 -> 1.28464 * mean) 2024-09-13T14:49:10.638 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:49:10.638 INFO:teuthology.orchestra.run.smithi154.stdout:oload 1000 2024-09-13T14:49:10.638 INFO:teuthology.orchestra.run.smithi154.stdout:max_change 1 2024-09-13T14:49:10.638 INFO:teuthology.orchestra.run.smithi154.stdout:max_change_osds 4 2024-09-13T14:49:10.638 INFO:teuthology.orchestra.run.smithi154.stdout:average_utilization 0.0238 2024-09-13T14:49:10.638 INFO:teuthology.orchestra.run.smithi154.stdout:overload_utilization 0.2377 2024-09-13T14:49:10.639 INFO:teuthology.orchestra.run.smithi154.stderr:no change 2024-09-13T14:49:10.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:10.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:10.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:10.888 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:10.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:10.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:11.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:11.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.157 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:11.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:11.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 74 16 76279 76263 64.3918 56.25 0.00839797 0.0155239 2024-09-13T14:49:11.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:11.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:11.366 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:11.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:11.414 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-13T14:49:11.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-09-13T14:49:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:11.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:11.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:11 smithi154 ceph-mon[102314]: from='client.44897 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:11.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:11 smithi154 ceph-mon[107038]: from='client.44897 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:11 smithi172 ceph-mon[100696]: from='client.44897 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:11.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:11.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:11.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:11.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:11.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:11.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:11.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:11.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:12.005 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:12.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.138 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:12.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:12.199 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:49:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:49:11] "GET /metrics HTTP/1.1" 200 36585 "" "Prometheus/2.51.0" 2024-09-13T14:49:12.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 75 16 77128 77112 64.2407 53.0625 0.0139673 0.0155603 2024-09-13T14:49:12.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:12.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:12.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:12.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:12.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:12 smithi154 ceph-mon[102314]: pgmap v553: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 54 MiB/s wr, 867 op/s; 29982/222120 objects degraded (13.498%); 4560/222120 objects misplaced (2.053%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:49:12.462 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:12 smithi154 ceph-mon[107038]: pgmap v553: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 54 MiB/s wr, 867 op/s; 29982/222120 objects degraded (13.498%); 4560/222120 objects misplaced (2.053%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:49:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:12 smithi172 ceph-mon[100696]: pgmap v553: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 54 MiB/s wr, 867 op/s; 29982/222120 objects degraded (13.498%); 4560/222120 objects misplaced (2.053%); 2.6 MiB/s, 40 objects/s recovering 2024-09-13T14:49:12.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.568 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:12.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:12.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:12.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:12.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:12.824 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:49:12.824 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:12.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:12.864 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:49:12.864 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:12.913 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:49:12.914 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:12.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:12.965 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:49:12.965 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:12.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:12.981 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:12.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:12.997 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:49:12.998 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:13.035 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:49:13.035 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:13.076 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:49:13.076 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:13.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:13.110 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:49:13.111 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:13.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:13.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 76 16 77953 77937 64.0738 51.5625 0.0187213 0.0156003 2024-09-13T14:49:13.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:13.266 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:13.362 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4073994624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:13.362 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:13.362 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4073994624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:13.362 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:13.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:13.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4073994624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:13.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:13.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:13.591 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:13.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:13.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:13.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:13.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:13.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:13.973 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:13.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:13.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:14.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:14.108 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:14.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:14.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 77 16 78865 78849 63.9819 57 0.0374789 0.0156229 2024-09-13T14:49:14.239 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:14.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:14.378 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:14.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:14 smithi154 ceph-mon[107038]: pgmap v554: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s; 29982/228435 objects degraded (13.125%); 4345/228435 objects misplaced (1.902%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:49:14.469 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:14 smithi154 ceph-mon[102314]: pgmap v554: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s; 29982/228435 objects degraded (13.125%); 4345/228435 objects misplaced (1.902%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:49:14.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:14.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:14 smithi172 ceph-mon[100696]: pgmap v554: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s; 29982/228435 objects degraded (13.125%); 4345/228435 objects misplaced (1.902%); 3.6 MiB/s, 57 objects/s recovering 2024-09-13T14:49:14.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:14.606 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:14.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:14.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:14.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:14.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:14.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:14.950 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:14.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:15.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.092 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:15.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:15.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:15.230 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 78 16 79751 79735 63.8715 55.375 0.00983258 0.0156497 2024-09-13T14:49:15.235 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:15.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:15.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:15.454 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 29982/228435 objects degraded (13.125%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:15.455 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 29982/228435 objects degraded (13.125%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:15.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:15.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 29982/228435 objects degraded (13.125%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:15.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:15.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:15.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:15.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:15.713 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:15.713 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:15.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.791 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:15.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:15.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:15.936 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:15.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:16.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:49:16.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:16.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:16.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:16.231 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 79 16 80604 80588 63.7379 53.3125 0.0197927 0.0156824 2024-09-13T14:49:16.255 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:16.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:16.384 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:16.428 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:16 smithi154 ceph-mon[102314]: pgmap v555: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 52 MiB/s wr, 825 op/s; 29982/232509 objects degraded (12.895%); 4345/232509 objects misplaced (1.869%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:16.428 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:16 smithi154 ceph-mon[107038]: pgmap v555: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 52 MiB/s wr, 825 op/s; 29982/232509 objects degraded (12.895%); 4345/232509 objects misplaced (1.869%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:16.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:49:16.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:16.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:16 smithi172 ceph-mon[100696]: pgmap v555: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 52 MiB/s wr, 825 op/s; 29982/232509 objects degraded (12.895%); 4345/232509 objects misplaced (1.869%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:16.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:16.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:16.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:16.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:16.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:16.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:16.922 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:16.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.024 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:17.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:17.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:17.195 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:17.231 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:49:17.230666+0000 min lat: 0.001048 max lat: 6.22442 avg lat: 0.01571 2024-09-13T14:49:17.231 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:49:17.231 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 80 16 81484 81468 63.6286 55 0.0165488 0.01571 2024-09-13T14:49:17.297 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:17.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:17.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:17.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:17.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:17.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:17.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:17.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:17.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:17.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:17.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:18.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.045 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:18.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:18.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:18.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:18.231 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 81 16 82315 82299 63.4842 51.9375 0.0135703 0.0157448 2024-09-13T14:49:18.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:18.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:18.324 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:18.335 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:18.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:18.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:18.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:18 smithi154 ceph-mon[102314]: pgmap v556: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 49 MiB/s wr, 782 op/s; 29982/234981 objects degraded (12.759%); 4345/234981 objects misplaced (1.849%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:18.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:18 smithi154 ceph-mon[107038]: pgmap v556: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 49 MiB/s wr, 782 op/s; 29982/234981 objects degraded (12.759%); 4345/234981 objects misplaced (1.849%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:18 smithi172 ceph-mon[100696]: pgmap v556: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 49 MiB/s wr, 782 op/s; 29982/234981 objects degraded (12.759%); 4345/234981 objects misplaced (1.849%); 2.1 MiB/s, 34 objects/s recovering 2024-09-13T14:49:18.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:18.577 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:18.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:18.721 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:18.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:18.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:18.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:18.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:18.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:19.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.232 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 82 16 83098 83082 63.3056 48.9375 0.0163734 0.0157879 2024-09-13T14:49:19.248 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:49:19.257 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.388 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1873948521' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:19.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1873948521' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:19.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1873948521' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:49:19.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:19.523 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:19.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:19.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:19.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.809 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:19.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:19.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:19.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:19.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:20.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:20.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:20.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:20.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:20.232 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 83 16 83908 83892 63.1528 50.625 0.0364496 0.015828 2024-09-13T14:49:20.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:20.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:20.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:20.484 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:20.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:20 smithi172 ceph-mon[100696]: pgmap v557: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 58 MiB/s wr, 932 op/s; 29982/242781 objects degraded (12.349%); 4121/242781 objects misplaced (1.697%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:49:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:20 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1873948521' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:20 smithi172 ceph-mon[100696]: osdmap e191: 8 total, 8 up, 7 in 2024-09-13T14:49:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 29982/242781 objects degraded (12.349%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:20.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:20.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[102314]: pgmap v557: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 58 MiB/s wr, 932 op/s; 29982/242781 objects degraded (12.349%); 4121/242781 objects misplaced (1.697%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1873948521' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[102314]: osdmap e191: 8 total, 8 up, 7 in 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 29982/242781 objects degraded (12.349%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[107038]: pgmap v557: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 58 MiB/s wr, 932 op/s; 29982/242781 objects degraded (12.349%); 4121/242781 objects misplaced (1.697%); 3.3 MiB/s, 52 objects/s recovering 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1873948521' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:49:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[107038]: osdmap e191: 8 total, 8 up, 7 in 2024-09-13T14:49:20.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 29982/242781 objects degraded (12.349%), 10 pgs degraded (PG_DEGRADED) 2024-09-13T14:49:20.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:20.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:20.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:20.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:20.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:21.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:21.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 84 16 84796 84780 63.0616 55.5 0.0318499 0.0158505 2024-09-13T14:49:21.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:21.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:21.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:21.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:21.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:21.618 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:21.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:21.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:21.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:21.890 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:21.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:21.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:22.028 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:22.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:22.162 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:22.199 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:49:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:49:21] "GET /metrics HTTP/1.1" 200 36580 "" "Prometheus/2.51.0" 2024-09-13T14:49:22.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 85 16 85618 85602 62.9241 51.375 0.0378165 0.0158846 2024-09-13T14:49:22.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:22.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:22.429 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:22.451 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:22.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:22 smithi154 ceph-mon[102314]: pgmap v559: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s; 29982/247734 objects degraded (12.102%); 4121/247734 objects misplaced (1.663%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:49:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'59322','2':'38719','3':'29701','6':'43806'}"} : dispatch 2024-09-13T14:49:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:22 smithi154 ceph-mon[107038]: pgmap v559: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s; 29982/247734 objects degraded (12.102%); 4121/247734 objects misplaced (1.663%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:49:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'59322','2':'38719','3':'29701','6':'43806'}"} : dispatch 2024-09-13T14:49:22.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:22.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:22.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:22.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:22.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:22 smithi172 ceph-mon[100696]: pgmap v559: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s; 29982/247734 objects degraded (12.102%); 4121/247734 objects misplaced (1.663%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:49:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'59322','2':'38719','3':'29701','6':'43806'}"} : dispatch 2024-09-13T14:49:22.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:22.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:22.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:22.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:22.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:22.951 INFO:teuthology.orchestra.run.smithi154.stdout:moved 33 / 267 (12.3596%) 2024-09-13T14:49:22.951 INFO:teuthology.orchestra.run.smithi154.stdout:avg 38.1429 2024-09-13T14:49:22.951 INFO:teuthology.orchestra.run.smithi154.stdout:stddev 5.91435 -> 9.49328 (expected baseline 5.71786) 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:min osd.2 with 27 -> 26 pgs (0.707865 -> 0.681648 * mean) 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:max osd.1 with 49 -> 48 pgs (1.28464 -> 1.25843 * mean) 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:oload 110 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:max_change 3 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:max_change_osds 4 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:average_utilization 0.0269 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:overload_utilization 0.0296 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3 weight 0.6552 -> 0.4532 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6 weight 0.8046 -> 0.6684 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2 weight 0.6541 -> 0.5908 2024-09-13T14:49:22.952 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1 weight 1.0000 -> 0.9052 2024-09-13T14:49:23.048 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:23.069 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:23.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:23.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:23.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:23.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 86 16 85699 85683 62.2514 5.0625 0.00916269 0.0158915 2024-09-13T14:49:23.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:23.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:23.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:23.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:23.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:23.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[102314]: from='client.34989 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59322','2':'38719','3':'29701','6':'43806'}"}]': finished 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[102314]: osdmap e192: 8 total, 8 up, 7 in 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[102314]: osdmap e193: 8 total, 8 up, 7 in 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[107038]: from='client.34989 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59322','2':'38719','3':'29701','6':'43806'}"}]': finished 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[107038]: osdmap e192: 8 total, 8 up, 7 in 2024-09-13T14:49:23.623 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:23 smithi154 ceph-mon[107038]: osdmap e193: 8 total, 8 up, 7 in 2024-09-13T14:49:23.654 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-13T14:49:23.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-13T14:49:23.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:23.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:23 smithi172 ceph-mon[100696]: from='client.34989 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:23 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59322','2':'38719','3':'29701','6':'43806'}"}]': finished 2024-09-13T14:49:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:23 smithi172 ceph-mon[100696]: osdmap e192: 8 total, 8 up, 7 in 2024-09-13T14:49:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:23 smithi172 ceph-mon[100696]: osdmap e193: 8 total, 8 up, 7 in 2024-09-13T14:49:23.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:23.842 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:23.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:23.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:23.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:24.067 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.091 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:24.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:24.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:24.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:24.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 87 16 85702 85686 61.5382 0.1875 0.003388 0.0159046 2024-09-13T14:49:24.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:24.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:24.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:24.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:24 smithi154 ceph-mon[102314]: pgmap v561: 89 pgs: 1 active+recovering+undersized+remapped, 6 peering, 3 remapped+peering, 1 unknown, 1 active+recovery_wait+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 878 op/s; 21564/253611 objects degraded (8.503%); 4345/253611 objects misplaced (1.713%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:24 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:49:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:24 smithi154 ceph-mon[102314]: osdmap e194: 8 total, 8 up, 7 in 2024-09-13T14:49:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:24 smithi154 ceph-mon[107038]: pgmap v561: 89 pgs: 1 active+recovering+undersized+remapped, 6 peering, 3 remapped+peering, 1 unknown, 1 active+recovery_wait+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 878 op/s; 21564/253611 objects degraded (8.503%); 4345/253611 objects misplaced (1.713%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:24 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:49:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:24 smithi154 ceph-mon[107038]: osdmap e194: 8 total, 8 up, 7 in 2024-09-13T14:49:24.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:24.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:24.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:24.712 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:24 smithi172 ceph-mon[100696]: pgmap v561: 89 pgs: 1 active+recovering+undersized+remapped, 6 peering, 3 remapped+peering, 1 unknown, 1 active+recovery_wait+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 878 op/s; 21564/253611 objects degraded (8.503%); 4345/253611 objects misplaced (1.713%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:49:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:24 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:49:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:24 smithi172 ceph-mon[100696]: osdmap e194: 8 total, 8 up, 7 in 2024-09-13T14:49:24.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:49:24.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:24.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:24.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:24.972 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:24.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:25.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:25.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:25.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:25.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:25.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 88 16 85773 85757 60.8894 4.4375 0.00335105 0.0161103 2024-09-13T14:49:25.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:25.343 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:25.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:25.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:25.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:25.593 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:25.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:25.698 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 21564/253611 objects degraded (8.503%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:25.698 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 21564/253611 objects degraded (8.503%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:25.717 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:25.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 21564/253611 objects degraded (8.503%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:25.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:25.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:25.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:25.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:25.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:26.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:26.091 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:26.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:26.207 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:26.208 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:26.215 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:26.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 89 16 85773 85757 60.2054 0 - 0.0161103 2024-09-13T14:49:26.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:26.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:26.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:26.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:26 smithi154 ceph-mon[102314]: pgmap v564: 89 pgs: 1 active+recovering+undersized+remapped, 6 peering, 4 remapped+peering, 1 unknown, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 41 MiB/s wr, 655 op/s; 21372/256515 objects degraded (8.332%); 4345/256515 objects misplaced (1.694%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:49:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/648388248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:26.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:26 smithi154 ceph-mon[107038]: pgmap v564: 89 pgs: 1 active+recovering+undersized+remapped, 6 peering, 4 remapped+peering, 1 unknown, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 41 MiB/s wr, 655 op/s; 21372/256515 objects degraded (8.332%); 4345/256515 objects misplaced (1.694%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:49:26.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/648388248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:26.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:26.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:26.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:26.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:26.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:26 smithi172 ceph-mon[100696]: pgmap v564: 89 pgs: 1 active+recovering+undersized+remapped, 6 peering, 4 remapped+peering, 1 unknown, 1 active+recovery_wait+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 41 MiB/s wr, 655 op/s; 21372/256515 objects degraded (8.332%); 4345/256515 objects misplaced (1.694%); 3.3 MiB/s, 53 objects/s recovering 2024-09-13T14:49:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/648388248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:26.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:26.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:26.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:26.964 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:26.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:27.047 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:27.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:27.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:27.192 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:27.215 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:27.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 90 16 85773 85757 59.5366 0 - 0.0161103 2024-09-13T14:49:27.284 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:27.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:27.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:27.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:27.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:27.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:27.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:27.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:27.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:27.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:27.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:27.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:27.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:27.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:28.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:28.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:28.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:28.213 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:28.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 91 16 85773 85757 58.8825 0 - 0.0161103 2024-09-13T14:49:28.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.338 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.463 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:28.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:28.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:28.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:28 smithi154 ceph-mon[102314]: pgmap v565: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 6 peering, 4 remapped+peering, 1 unknown, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 523 op/s; 21372/257163 objects degraded (8.311%); 4345/257163 objects misplaced (1.690%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:49:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:28 smithi154 ceph-mon[107038]: pgmap v565: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 6 peering, 4 remapped+peering, 1 unknown, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 523 op/s; 21372/257163 objects degraded (8.311%); 4345/257163 objects misplaced (1.690%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:49:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:28 smithi172 ceph-mon[100696]: pgmap v565: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 6 peering, 4 remapped+peering, 1 unknown, 6 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 33 MiB/s wr, 523 op/s; 21372/257163 objects degraded (8.311%); 4345/257163 objects misplaced (1.690%); 3.2 MiB/s, 50 objects/s recovering 2024-09-13T14:49:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:28.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:28.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:28.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:28.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:28.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:29.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:29.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:29.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:29.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:29.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:29.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 92 16 85773 85757 58.2426 0 - 0.0161103 2024-09-13T14:49:29.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:29.335 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:29.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:29.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:29.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:29.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:29.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:29.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:29.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:29 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:49:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:29 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:49:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:29 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T14:49:29.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:29.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:29.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:29.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:29.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:30.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:30.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:30.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:30.211 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:30.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:30.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 93 16 85773 85757 57.6164 0 - 0.0161103 2024-09-13T14:49:30.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:30.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:30.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:30.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:30.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:30.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:30.621 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:49:30.621 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:49:30.621 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T14:49:30.621 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:49:30.622 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:30 smithi154 ceph-mon[102314]: pgmap v566: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 4 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s; 21400/257268 objects degraded (8.318%); 3536/257268 objects misplaced (1.374%); 10 MiB/s, 166 objects/s recovering 2024-09-13T14:49:30.622 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 21400/257268 objects degraded (8.318%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:30.622 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:30 smithi154 ceph-mon[107038]: pgmap v566: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 4 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s; 21400/257268 objects degraded (8.318%); 3536/257268 objects misplaced (1.374%); 10 MiB/s, 166 objects/s recovering 2024-09-13T14:49:30.622 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 21400/257268 objects degraded (8.318%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:30.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:30.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:30 smithi172 ceph-mon[100696]: pgmap v566: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 4 remapped+peering, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 MiB/s wr, 475 op/s; 21400/257268 objects degraded (8.318%); 3536/257268 objects misplaced (1.374%); 10 MiB/s, 166 objects/s recovering 2024-09-13T14:49:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 21400/257268 objects degraded (8.318%), 7 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:30.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:30.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:30.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:30.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:30.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:31.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:31.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:31.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:31.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 94 16 85773 85757 57.0036 0 - 0.0161103 2024-09-13T14:49:31.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:31.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:31.453 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:31.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:31.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:31.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:31.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:31.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:31.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:31.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:31.951 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:31.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:32.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:32.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:32.142 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:49:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:49:31] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.51.0" 2024-09-13T14:49:32.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:32.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:32.233 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 95 16 85773 85757 56.4037 0 - 0.0161103 2024-09-13T14:49:32.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:32.329 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:32.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:32.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:32 smithi154 ceph-mon[102314]: pgmap v567: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 9 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 9.7 MiB/s wr, 155 op/s; 36928/257352 objects degraded (14.349%); 3536/257352 objects misplaced (1.374%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:49:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:32.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:32 smithi154 ceph-mon[107038]: pgmap v567: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 9 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 9.7 MiB/s wr, 155 op/s; 36928/257352 objects degraded (14.349%); 3536/257352 objects misplaced (1.374%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:49:32.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:32.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:32 smithi172 ceph-mon[100696]: pgmap v567: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 9 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 9.7 MiB/s wr, 155 op/s; 36928/257352 objects degraded (14.349%); 3536/257352 objects misplaced (1.374%); 6.3 MiB/s, 100 objects/s recovering 2024-09-13T14:49:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:49:32.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:32.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:32.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:49:32.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:32.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:32.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:32.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:32.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:32.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:32.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:33.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:33.115 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:33.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:33.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:33.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 96 16 85773 85757 55.8163 0 - 0.0161103 2024-09-13T14:49:33.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:33.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:33.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:33.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:33.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:33.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:33.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:33.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:33.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:33.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:33.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:33.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:33.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:34.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:34.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:34.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:34.232 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:34.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 97 16 85773 85757 55.2409 0 - 0.0161103 2024-09-13T14:49:34.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:34.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:34.413 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:34 smithi154 ceph-mon[102314]: pgmap v568: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 remapped+peering, 9 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 8.0 MiB/s wr, 128 op/s; 36928/257352 objects degraded (14.349%); 2461/257352 objects misplaced (0.956%); 12 MiB/s, 194 objects/s recovering 2024-09-13T14:49:34.413 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:34 smithi154 ceph-mon[107038]: pgmap v568: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 remapped+peering, 9 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 8.0 MiB/s wr, 128 op/s; 36928/257352 objects degraded (14.349%); 2461/257352 objects misplaced (0.956%); 12 MiB/s, 194 objects/s recovering 2024-09-13T14:49:34.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:34.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:34.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:34.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:34.652 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:34 smithi172 ceph-mon[100696]: pgmap v568: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 remapped+peering, 9 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 8.0 MiB/s wr, 128 op/s; 36928/257352 objects degraded (14.349%); 2461/257352 objects misplaced (0.956%); 12 MiB/s, 194 objects/s recovering 2024-09-13T14:49:34.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:34.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:34.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:34.857 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:34.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:34.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:34.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:35.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:35.109 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:35.140 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:35.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:35.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 98 16 85773 85757 54.6774 0 - 0.0161103 2024-09-13T14:49:35.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:35.261 INFO:teuthology.orchestra.run.smithi154.stdout:moved 0 / 267 (0%) 2024-09-13T14:49:35.261 INFO:teuthology.orchestra.run.smithi154.stdout:avg 38.1429 2024-09-13T14:49:35.261 INFO:teuthology.orchestra.run.smithi154.stdout:stddev 9.49328 -> 9.49328 (expected baseline 5.71786) 2024-09-13T14:49:35.261 INFO:teuthology.orchestra.run.smithi154.stdout:min osd.3 with 23 -> 23 pgs (0.602996 -> 0.602996 * mean) 2024-09-13T14:49:35.261 INFO:teuthology.orchestra.run.smithi154.stdout:max osd.1 with 48 -> 48 pgs (1.25843 -> 1.25843 * mean) 2024-09-13T14:49:35.261 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:49:35.261 INFO:teuthology.orchestra.run.smithi154.stdout:oload 1000 2024-09-13T14:49:35.262 INFO:teuthology.orchestra.run.smithi154.stdout:max_change 0.05 2024-09-13T14:49:35.262 INFO:teuthology.orchestra.run.smithi154.stdout:max_change_osds 4 2024-09-13T14:49:35.262 INFO:teuthology.orchestra.run.smithi154.stdout:average_utilization 0.0281 2024-09-13T14:49:35.262 INFO:teuthology.orchestra.run.smithi154.stdout:overload_utilization 0.2807 2024-09-13T14:49:35.262 INFO:teuthology.orchestra.run.smithi154.stderr:no change 2024-09-13T14:49:35.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:35.368 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:35.470 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:35.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:35.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:35.604 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:35 smithi172 ceph-mon[100696]: osdmap e195: 8 total, 8 up, 7 in 2024-09-13T14:49:35.604 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 36928/257352 objects degraded (14.349%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.620 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:35.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:35.709 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:35 smithi154 ceph-mon[102314]: osdmap e195: 8 total, 8 up, 7 in 2024-09-13T14:49:35.709 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 36928/257352 objects degraded (14.349%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:35.709 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:35 smithi154 ceph-mon[107038]: osdmap e195: 8 total, 8 up, 7 in 2024-09-13T14:49:35.709 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 36928/257352 objects degraded (14.349%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:49:35.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:35.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:35.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:35.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:35.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:35.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:35.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:36.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:36.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:36.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:36.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 99 16 85773 85757 54.1251 0 - 0.0161103 2024-09-13T14:49:36.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:36.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:36.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:36.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[102314]: pgmap v570: 89 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 MiB/s wr, 27 op/s; 33743/257352 objects degraded (13.112%); 5639/257352 objects misplaced (2.191%); 12 MiB/s, 189 objects/s recovering 2024-09-13T14:49:36.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[102314]: from='client.35001 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:36.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[102314]: osdmap e196: 8 total, 8 up, 7 in 2024-09-13T14:49:36.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:36.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:36.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[107038]: pgmap v570: 89 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 MiB/s wr, 27 op/s; 33743/257352 objects degraded (13.112%); 5639/257352 objects misplaced (2.191%); 12 MiB/s, 189 objects/s recovering 2024-09-13T14:49:36.432 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[107038]: from='client.35001 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:36.432 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[107038]: osdmap e196: 8 total, 8 up, 7 in 2024-09-13T14:49:36.432 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:36.432 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:36.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:36.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:36.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:36.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:36.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:36.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:36.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:36 smithi172 ceph-mon[100696]: pgmap v570: 89 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 MiB/s wr, 27 op/s; 33743/257352 objects degraded (13.112%); 5639/257352 objects misplaced (2.191%); 12 MiB/s, 189 objects/s recovering 2024-09-13T14:49:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:36 smithi172 ceph-mon[100696]: from='client.35001 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T14:49:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:36 smithi172 ceph-mon[100696]: osdmap e196: 8 total, 8 up, 7 in 2024-09-13T14:49:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:36.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:36.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:36.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:36.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:36.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:37.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:37.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:37.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:37.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:49:37.234068+0000 min lat: 0.001048 max lat: 6.22442 avg lat: 0.0161103 2024-09-13T14:49:37.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:49:37.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 100 16 85773 85757 53.584 0 - 0.0161103 2024-09-13T14:49:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.237 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:37.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:37.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:37.361 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:37.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:37.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:37.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:37.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:37.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:37.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:37.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:37.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:37.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:37.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:37.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:38.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:38.122 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:38.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:38.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 101 16 85773 85757 53.0535 0 - 0.0161103 2024-09-13T14:49:38.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:38.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:38.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:38.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:38.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:38.592 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:38 smithi154 ceph-mon[102314]: pgmap v572: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 224 KiB/s wr, 3 op/s; 33743/257352 objects degraded (13.112%); 5639/257352 objects misplaced (2.191%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T14:49:38.593 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:38 smithi154 ceph-mon[107038]: pgmap v572: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 224 KiB/s wr, 3 op/s; 33743/257352 objects degraded (13.112%); 5639/257352 objects misplaced (2.191%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T14:49:38.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:38.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:38.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:38.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:38 smithi172 ceph-mon[100696]: pgmap v572: 89 pgs: 1 active+recovery_wait+remapped, 2 active+recovering+undersized+remapped, 2 remapped+peering, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 224 KiB/s wr, 3 op/s; 33743/257352 objects degraded (13.112%); 5639/257352 objects misplaced (2.191%); 8.5 MiB/s, 135 objects/s recovering 2024-09-13T14:49:38.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:38.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:38.880 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:38.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:38.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:39.008 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:39.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:39.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:39.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:39.235 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 102 16 85773 85757 52.5335 0 - 0.0161103 2024-09-13T14:49:39.259 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:39.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:39.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:39.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:39.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:39.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:39.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:39.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:39.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:39.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:39.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:39.883 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:39.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:39.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:40.009 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:40.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:40.137 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:40.147 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:49:40.147 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:49:40.147 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:49:40.148 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:40.148 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:40.148 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:40.150 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:49:40.150 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:40.168 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:49:40.168 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:40.185 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:49:40.185 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:40.202 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:49:40.203 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:40.219 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:49:40.219 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:40.234 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 103 16 85773 85757 52.0235 0 - 0.0161103 2024-09-13T14:49:40.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:40.264 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:40.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:40.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:40.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:40.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:40.611 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:40 smithi154 ceph-mon[102314]: pgmap v573: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 5347/257358 objects misplaced (2.078%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:49:40.611 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 53003/257358 objects degraded (20.595%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:40.611 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:40 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1823386502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:40.611 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:40 smithi154 ceph-mon[107038]: pgmap v573: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 5347/257358 objects misplaced (2.078%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:49:40.611 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 53003/257358 objects degraded (20.595%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:40.611 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:40 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1823386502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:40.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:40.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:40.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:40 smithi172 ceph-mon[100696]: pgmap v573: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 5347/257358 objects misplaced (2.078%); 11 MiB/s, 171 objects/s recovering 2024-09-13T14:49:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 53003/257358 objects degraded (20.595%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:40 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1823386502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:40.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:40.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:40.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:40.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:40.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.010 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:41.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:41.147 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:49:41.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:49:41.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:41.235 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 104 16 85773 85757 51.5234 0 - 0.0161103 2024-09-13T14:49:41.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:41.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:41.384 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:41.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:41.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:41.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:41.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:41.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:41.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:41.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:41.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:42.006 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:42.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:42.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:49:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:49:41] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.51.0" 2024-09-13T14:49:42.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:42.235 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 105 16 85773 85757 51.0325 0 - 0.0161103 2024-09-13T14:49:42.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:42.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:42.366 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:49:42.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:42.440 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:42.440 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:42 smithi154 ceph-mon[102314]: pgmap v574: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4877/257358 objects misplaced (1.895%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:49:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3889063683' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:42.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:42 smithi154 ceph-mon[107038]: pgmap v574: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4877/257358 objects misplaced (1.895%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:49:42.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3889063683' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:42.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:42.508 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:42.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:42.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:42.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:42.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:42 smithi172 ceph-mon[100696]: pgmap v574: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4877/257358 objects misplaced (1.895%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:49:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3889063683' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:49:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:42.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:42.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:42.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:42.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:42.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:43.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:43.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:43.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:49:43.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:43.135 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:43.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 106 16 85773 85757 50.551 0 - 0.0161103 2024-09-13T14:49:43.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:43.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:43.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:43.384 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:43.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:43.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:43.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:43.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:43.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:49:43.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3889063683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:43.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[107038]: osdmap e197: 8 total, 8 up, 7 in 2024-09-13T14:49:43.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:43.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:43.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:49:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3889063683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[102314]: osdmap e197: 8 total, 8 up, 7 in 2024-09-13T14:49:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:43.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:43.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:43 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:49:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3889063683' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:49:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:43 smithi172 ceph-mon[100696]: osdmap e197: 8 total, 8 up, 7 in 2024-09-13T14:49:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:43.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:43.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:43.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:43.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:43.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:49:43.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:44.007 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:44.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:44.131 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:44.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:44.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 107 16 85773 85757 50.0787 0 - 0.0161103 2024-09-13T14:49:44.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:44.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:44.379 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:44.411 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:44.448 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:44.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:44.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:44.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:44.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:44.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:44 smithi154 ceph-mon[102314]: pgmap v576: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4877/257358 objects misplaced (1.895%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:49:44.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:44 smithi154 ceph-mon[107038]: pgmap v576: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4877/257358 objects misplaced (1.895%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:49:44.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:44.740 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:44 smithi172 ceph-mon[100696]: pgmap v576: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 activating+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 16 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4877/257358 objects misplaced (1.895%); 6.0 MiB/s, 95 objects/s recovering 2024-09-13T14:49:44.752 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:44.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:44.878 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:44.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:44.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.005 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:45.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:45.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:45.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:45.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total time run: 107.87 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total writes made: 85773 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Write size: 65536 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object size: 65536 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Bandwidth (MB/sec): 49.6969 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Bandwidth: 44.7754 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max bandwidth (MB/sec): 219.5 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min bandwidth (MB/sec): 0 2024-09-13T14:49:45.236 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average IOPS: 795 2024-09-13T14:49:45.237 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev IOPS: 716.411 2024-09-13T14:49:45.237 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max IOPS: 3512 2024-09-13T14:49:45.237 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min IOPS: 0 2024-09-13T14:49:45.237 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average Latency(s): 0.0200614 2024-09-13T14:49:45.237 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Latency(s): 0.301651 2024-09-13T14:49:45.237 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max latency(s): 22.3974 2024-09-13T14:49:45.237 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min latency(s): 0.001048 2024-09-13T14:49:45.240 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:49:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:45.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:45.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:45.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:45.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:45.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:45.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:45.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:45.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:49:45.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:45.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:45.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:45.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:45.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:46.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:46.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:46.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:46.168 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:46.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:46.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:46.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:46.416 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:46 smithi154 ceph-mon[102314]: pgmap v577: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 1 active+recovery_wait+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 13 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4302/257358 objects misplaced (1.672%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T14:49:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:46 smithi154 ceph-mon[107038]: pgmap v577: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 1 active+recovery_wait+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 13 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4302/257358 objects misplaced (1.672%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T14:49:46.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:46.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:46.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:46.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:46.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:46 smithi172 ceph-mon[100696]: pgmap v577: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 1 active+recovery_wait+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 13 KiB/s wr, 0 op/s; 53003/257358 objects degraded (20.595%); 4302/257358 objects misplaced (1.672%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T14:49:46.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:46.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:46.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:46.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:46.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:47.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:47.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:47.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:47.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:47.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:47.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:47.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:47.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:47.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:47.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:47.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:47.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:47.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:47 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 53003/257334 objects degraded (20.597%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:47.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:47 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 53003/257334 objects degraded (20.597%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:47.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:47 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 53003/257334 objects degraded (20.597%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:47.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:47.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:47.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:47.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:47.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:48.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:48.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:48.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:48.164 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:48.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:48.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:48.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:48.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:48 smithi154 ceph-mon[102314]: pgmap v578: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 13 KiB/s wr, 0 op/s; 53003/257334 objects degraded (20.597%); 4302/257334 objects misplaced (1.672%); 8.4 MiB/s, 133 objects/s recovering 2024-09-13T14:49:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:48 smithi154 ceph-mon[107038]: pgmap v578: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 13 KiB/s wr, 0 op/s; 53003/257334 objects degraded (20.597%); 4302/257334 objects misplaced (1.672%); 8.4 MiB/s, 133 objects/s recovering 2024-09-13T14:49:48.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:48.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:48.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:48.669 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:48 smithi172 ceph-mon[100696]: pgmap v578: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 13 KiB/s wr, 0 op/s; 53003/257334 objects degraded (20.597%); 4302/257334 objects misplaced (1.672%); 8.4 MiB/s, 133 objects/s recovering 2024-09-13T14:49:48.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:48.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:48.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:48.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:48.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:49.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:49.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:49.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:49.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:49.168 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:49.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:49.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:49.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:49.419 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:49.438 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:49:49.520 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:49.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:49.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:49.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:49.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:49.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:49 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3972227419' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:49:49.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:49 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:49:49.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:49 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3972227419' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:49:49.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:49 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:49:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:49 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3972227419' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:49:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:49 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:49:49.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:49.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:49.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:49.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:49.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:50.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:50.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:50.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:50.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:50.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:50.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:50.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:50.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:50.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:50.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:50.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:50.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:50.681 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[102314]: pgmap v579: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 8 op/s; 53003/257094 objects degraded (20.616%); 4302/257094 objects misplaced (1.673%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:49:50.681 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[102314]: osdmap e198: 8 total, 8 up, 7 in 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[107038]: pgmap v579: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 8 op/s; 53003/257094 objects degraded (20.616%); 4302/257094 objects misplaced (1.673%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[107038]: osdmap e198: 8 total, 8 up, 7 in 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:50.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:50 smithi172 ceph-mon[100696]: pgmap v579: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+laggy, 9 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 8 op/s; 53003/257094 objects degraded (20.616%); 4302/257094 objects misplaced (1.673%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:49:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:50 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:49:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:50 smithi172 ceph-mon[100696]: osdmap e198: 8 total, 8 up, 7 in 2024-09-13T14:49:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:50.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.793 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:50.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:50.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:50.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:50.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:51.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:51.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:51.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:51.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:51.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:51.286 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:51.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:51.335 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:49:51.335 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:49:51.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:51.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:51.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:51.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:51.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:51.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:51.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:51.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:51.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:51.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:51.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:52.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:52.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:52.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:52.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:52.174 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:49:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:49:51] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-09-13T14:49:52.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:49:52.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:52.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:52.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:52.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:52.417 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:49:52.417 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:49:52.437 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:49:52.438 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:49:52.454 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:49:52.454 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:49:52.470 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:49:52.470 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:49:52.489 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:49:52.489 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:49:52.506 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:49:52.506 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:49:52.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:52.523 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:49:52.523 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:49:52.541 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:49:52.541 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:49:52.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:52.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:52.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[102314]: pgmap v581: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 119 B/s rd, 112 KiB/s wr, 42 op/s; 52861/256365 objects degraded (20.619%); 3627/256365 objects misplaced (1.415%); 9.6 MiB/s, 153 objects/s recovering 2024-09-13T14:49:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2590944935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[107038]: pgmap v581: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 119 B/s rd, 112 KiB/s wr, 42 op/s; 52861/256365 objects degraded (20.619%); 3627/256365 objects misplaced (1.415%); 9.6 MiB/s, 153 objects/s recovering 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:52 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2590944935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:52 smithi172 ceph-mon[100696]: pgmap v581: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 119 B/s rd, 112 KiB/s wr, 42 op/s; 52861/256365 objects degraded (20.619%); 3627/256365 objects misplaced (1.415%); 9.6 MiB/s, 153 objects/s recovering 2024-09-13T14:49:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:49:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:49:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:52 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2590944935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:49:52.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:52.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:52.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:49:52.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:52.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:52.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:53.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:53.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:53.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:53.186 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:53.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:53.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:53.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.439 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:53.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:53.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:53.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:53.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:53.671 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:53 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 52861/256089 objects degraded (20.642%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:53.672 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:53 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 52861/256089 objects degraded (20.642%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:53.688 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:53 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 52861/256089 objects degraded (20.642%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:49:53.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:53.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:53.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:53.937 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:53.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:54.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:54.062 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:54.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:54.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:54.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:54.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:54.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:54.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:54.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:54.503 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:54.503 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:54.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:54.561 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:54.574 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:54 smithi154 ceph-mon[107038]: pgmap v582: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 102 B/s rd, 96 KiB/s wr, 45 op/s; 52861/256089 objects degraded (20.642%); 3627/256089 objects misplaced (1.416%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:49:54.575 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:54 smithi154 ceph-mon[102314]: pgmap v582: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 102 B/s rd, 96 KiB/s wr, 45 op/s; 52861/256089 objects degraded (20.642%); 3627/256089 objects misplaced (1.416%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:49:54.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:54.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:54.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:54.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:54.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:54.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:54.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:54.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:54 smithi172 ceph-mon[100696]: pgmap v582: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 102 B/s rd, 96 KiB/s wr, 45 op/s; 52861/256089 objects degraded (20.642%); 3627/256089 objects misplaced (1.416%); 8.2 MiB/s, 131 objects/s recovering 2024-09-13T14:49:55.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:55.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:55.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.194 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:55.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:55.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:55.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:55.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:55.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:55.497 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:49:55.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.576 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:55.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:55.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:55.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:55.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2685046244' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:55.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2685046244' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:55.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:55.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:55.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:55.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:55.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:55.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2685046244' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:49:56.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:49:56.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:56.078 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:56.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:56.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:56.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:56.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:56.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:56.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.578 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:56.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:56.616 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[102314]: pgmap v583: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 102 B/s rd, 96 KiB/s wr, 67 op/s; 52715/255417 objects degraded (20.639%); 2715/255417 objects misplaced (1.063%); 11 MiB/s, 172 objects/s recovering 2024-09-13T14:49:56.616 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:56.616 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2685046244' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:56.616 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[102314]: osdmap e199: 8 total, 8 up, 7 in 2024-09-13T14:49:56.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[107038]: pgmap v583: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 102 B/s rd, 96 KiB/s wr, 67 op/s; 52715/255417 objects degraded (20.639%); 2715/255417 objects misplaced (1.063%); 11 MiB/s, 172 objects/s recovering 2024-09-13T14:49:56.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:56.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2685046244' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:56.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:56 smithi154 ceph-mon[107038]: osdmap e199: 8 total, 8 up, 7 in 2024-09-13T14:49:56.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:56.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:56.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:56.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:56.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:56.954 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:56.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:56 smithi172 ceph-mon[100696]: pgmap v583: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 102 B/s rd, 96 KiB/s wr, 67 op/s; 52715/255417 objects degraded (20.639%); 2715/255417 objects misplaced (1.063%); 11 MiB/s, 172 objects/s recovering 2024-09-13T14:49:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:56 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:49:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:56 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2685046244' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:49:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:56 smithi172 ceph-mon[100696]: osdmap e199: 8 total, 8 up, 7 in 2024-09-13T14:49:57.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:57.078 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:57.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:57.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:57.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:57.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:57.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:57.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:57.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:57.508 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:49:57.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:57.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:57.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:57.656 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:57 smithi154 ceph-mon[102314]: osdmap e200: 8 total, 8 up, 7 in 2024-09-13T14:49:57.656 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1305800297' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:49:57.656 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:57 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:49:57.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:57 smithi154 ceph-mon[107038]: osdmap e200: 8 total, 8 up, 7 in 2024-09-13T14:49:57.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1305800297' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:49:57.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:57 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:49:57.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:57.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:57.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:57.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:57.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:57.929 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:49:57.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:49:57.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:57.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:49:57.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:57 smithi172 ceph-mon[100696]: osdmap e200: 8 total, 8 up, 7 in 2024-09-13T14:49:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1305800297' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:49:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:57 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:49:58.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:58.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:58.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:49:58.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:49:58.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:58.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:58.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:58.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:58.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:58.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:58.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:58.701 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:58.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[102314]: pgmap v586: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 136 B/s rd, 128 KiB/s wr, 67 op/s; 52715/255270 objects degraded (20.651%); 2715/255270 objects misplaced (1.064%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[102314]: osdmap e201: 8 total, 8 up, 7 in 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[107038]: pgmap v586: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 136 B/s rd, 128 KiB/s wr, 67 op/s; 52715/255270 objects degraded (20.651%); 2715/255270 objects misplaced (1.064%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[107038]: osdmap e201: 8 total, 8 up, 7 in 2024-09-13T14:49:58.818 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:58.819 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:58.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:58.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:49:58.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:58.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:58.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:58 smithi172 ceph-mon[100696]: pgmap v586: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 136 B/s rd, 128 KiB/s wr, 67 op/s; 52715/255270 objects degraded (20.651%); 2715/255270 objects misplaced (1.064%); 8.5 MiB/s, 136 objects/s recovering 2024-09-13T14:49:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:49:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:58 smithi172 ceph-mon[100696]: osdmap e201: 8 total, 8 up, 7 in 2024-09-13T14:49:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:49:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:49:59.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:59.076 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:59.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:49:59.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:49:59.199 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:49:59.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:49:59.325 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:49:59.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:59.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:59.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:49:59.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:49:59.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:49:59.696 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:49:59.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:49:59.820 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:49:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:49:59.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:49:59.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:49:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:49:59.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:49:59.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:49:59 smithi154 ceph-mon[102314]: osdmap e202: 8 total, 8 up, 7 in 2024-09-13T14:49:59.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:49:59 smithi154 ceph-mon[107038]: osdmap e202: 8 total, 8 up, 7 in 2024-09-13T14:49:59.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:49:59 smithi172 ceph-mon[100696]: osdmap e202: 8 total, 8 up, 7 in 2024-09-13T14:50:00.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:00.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:00.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:00.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:00.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:00.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:00.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:00.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:00.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:00.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:00.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:00.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:00.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:00.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:00.841 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pgmap v589: 89 pgs: 2 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 43 op/s; 52715/254892 objects degraded (20.681%); 2715/254892 objects misplaced (1.065%) 2024-09-13T14:50:00.841 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: osdmap e203: 8 total, 8 up, 7 in 2024-09-13T14:50:00.841 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:00.841 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized 2024-09-13T14:50:00.841 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: [WRN] PG_DEGRADED: Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 2.1 is active+recovery_wait+undersized+degraded+remapped, acting [0,1] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.0 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.1 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.2 is active+recovery_wait+undersized+degraded+remapped, acting [4,2] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.4 is active+recovery_wait+undersized+degraded+remapped, acting [5,6] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.5 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [3,2] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.6 is active+recovery_wait+undersized+degraded+remapped, acting [4,5] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.9 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.c is active+recovery_wait+undersized+degraded+remapped, acting [5,1] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.e is stuck undersized for 80s, current state active+recovering+undersized+remapped, last acting [4,1] 2024-09-13T14:50:00.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[102314]: pg 6.f is active+recovery_wait+undersized+degraded+remapped, acting [4,6] 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pgmap v589: 89 pgs: 2 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 43 op/s; 52715/254892 objects degraded (20.681%); 2715/254892 objects misplaced (1.065%) 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: osdmap e203: 8 total, 8 up, 7 in 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: [WRN] PG_DEGRADED: Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 2.1 is active+recovery_wait+undersized+degraded+remapped, acting [0,1] 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.0 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.1 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-13T14:50:00.843 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.2 is active+recovery_wait+undersized+degraded+remapped, acting [4,2] 2024-09-13T14:50:00.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.4 is active+recovery_wait+undersized+degraded+remapped, acting [5,6] 2024-09-13T14:50:00.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.5 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [3,2] 2024-09-13T14:50:00.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.6 is active+recovery_wait+undersized+degraded+remapped, acting [4,5] 2024-09-13T14:50:00.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.9 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:50:00.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.c is active+recovery_wait+undersized+degraded+remapped, acting [5,1] 2024-09-13T14:50:00.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.e is stuck undersized for 80s, current state active+recovering+undersized+remapped, last acting [4,1] 2024-09-13T14:50:00.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:00 smithi154 ceph-mon[107038]: pg 6.f is active+recovery_wait+undersized+degraded+remapped, acting [4,6] 2024-09-13T14:50:00.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:00.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:00.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pgmap v589: 89 pgs: 2 peering, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 43 op/s; 52715/254892 objects degraded (20.681%); 2715/254892 objects misplaced (1.065%) 2024-09-13T14:50:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: osdmap e203: 8 total, 8 up, 7 in 2024-09-13T14:50:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:00.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: [WRN] PG_DEGRADED: Degraded data redundancy: 52715/254892 objects degraded (20.681%), 10 pgs degraded, 5 pgs undersized 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 2.1 is active+recovery_wait+undersized+degraded+remapped, acting [0,1] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.0 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.1 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [1,6] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.2 is active+recovery_wait+undersized+degraded+remapped, acting [4,2] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.4 is active+recovery_wait+undersized+degraded+remapped, acting [5,6] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.5 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [3,2] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.6 is active+recovery_wait+undersized+degraded+remapped, acting [4,5] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.9 is stuck undersized for 80s, current state active+recovery_wait+undersized+degraded+remapped, last acting [0,3] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.c is active+recovery_wait+undersized+degraded+remapped, acting [5,1] 2024-09-13T14:50:00.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.e is stuck undersized for 80s, current state active+recovering+undersized+remapped, last acting [4,1] 2024-09-13T14:50:00.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:00 smithi172 ceph-mon[100696]: pg 6.f is active+recovery_wait+undersized+degraded+remapped, acting [4,6] 2024-09-13T14:50:01.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:01.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T14:50:01.080 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:50:01.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:01.184 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:01.282 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:50:01.283 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:01.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:01.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:01.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:01.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:01.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:01.560 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:01.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:01.683 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:01.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.807 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:01.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:01.875 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3456139895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:01.875 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3456139895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:01.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:01.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:01.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:01.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3456139895' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:02.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:02.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:02.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:02.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:50:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:50:01] "GET /metrics HTTP/1.1" 200 36584 "" "Prometheus/2.51.0" 2024-09-13T14:50:02.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:50:02.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:02.313 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:02.327 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:02.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:02.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:02.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:02.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:02.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:02.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:02.705 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:02.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:02.831 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:02.841 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:02 smithi154 ceph-mon[107038]: pgmap v591: 89 pgs: 2 peering, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 67 op/s; 52688/254367 objects degraded (20.713%); 2169/254367 objects misplaced (0.853%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T14:50:02.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:02 smithi154 ceph-mon[102314]: pgmap v591: 89 pgs: 2 peering, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 67 op/s; 52688/254367 objects degraded (20.713%); 2169/254367 objects misplaced (0.853%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T14:50:02.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:02.955 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:02.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:02.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:02 smithi172 ceph-mon[100696]: pgmap v591: 89 pgs: 2 peering, 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 67 op/s; 52688/254367 objects degraded (20.713%); 2169/254367 objects misplaced (0.853%); 8.4 MiB/s, 134 objects/s recovering 2024-09-13T14:50:03.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:03.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:03.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:03.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:03.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:03.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:03.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:03.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:03.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:03.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:03.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:03.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:03.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:03.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:03.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:03.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:03.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:04.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:04.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:04.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.199 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:04.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:04.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:04.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:04.451 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:04.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:04.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:04.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:04.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:04 smithi154 ceph-mon[102314]: pgmap v592: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 53 op/s; 52688/254298 objects degraded (20.719%); 2169/254298 objects misplaced (0.853%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:50:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:04 smithi154 ceph-mon[107038]: pgmap v592: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 53 op/s; 52688/254298 objects degraded (20.719%); 2169/254298 objects misplaced (0.853%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:50:04.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:04.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:04.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:04.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:04.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:04.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:04 smithi172 ceph-mon[100696]: pgmap v592: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 53 op/s; 52688/254298 objects degraded (20.719%); 2169/254298 objects misplaced (0.853%); 6.2 MiB/s, 99 objects/s recovering 2024-09-13T14:50:05.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:05.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:05.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:05.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:05.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:05.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:05.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:05.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:05.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:05.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:05.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:05.690 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 52688/254298 objects degraded (20.719%), 10 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:05.691 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 52688/254298 objects degraded (20.719%), 10 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:05.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:05.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:05.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:05.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:05.955 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:05.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:05.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 52688/254298 objects degraded (20.719%), 10 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:06.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:06.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:06.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:06.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:06.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:06.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:06.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:06.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:06.541 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:50:06.544 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:50:06.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:06.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:06.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:06.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:06.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:06.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:06.851 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[102314]: pgmap v593: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 45 op/s; 52683/254265 objects degraded (20.720%); 2169/254265 objects misplaced (0.853%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:50:06.851 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/629872938' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1047789454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[107038]: pgmap v593: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 45 op/s; 52683/254265 objects degraded (20.720%); 2169/254265 objects misplaced (0.853%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/629872938' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1047789454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:50:06.852 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:06 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:50:06.936 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:06.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:06.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:06.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:06 smithi172 ceph-mon[100696]: pgmap v593: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 45 op/s; 52683/254265 objects degraded (20.720%); 2169/254265 objects misplaced (0.853%); 5.0 MiB/s, 80 objects/s recovering 2024-09-13T14:50:06.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/629872938' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:06.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:06 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:06.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1047789454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:50:06.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:06 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:50:07.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:07.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:07.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:07.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:07.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:07.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:07.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:07.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:07.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:07.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:07.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:07.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:07.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:07.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:07.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:07.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[102314]: osdmap e204: 8 total, 8 up, 7 in 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[102314]: osdmap e205: 8 total, 8 up, 7 in 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[107038]: osdmap e204: 8 total, 8 up, 7 in 2024-09-13T14:50:07.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:07 smithi154 ceph-mon[107038]: osdmap e205: 8 total, 8 up, 7 in 2024-09-13T14:50:07.975 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:07.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:07 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:07 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:07 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:50:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:07 smithi172 ceph-mon[100696]: osdmap e204: 8 total, 8 up, 7 in 2024-09-13T14:50:07.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:07 smithi172 ceph-mon[100696]: osdmap e205: 8 total, 8 up, 7 in 2024-09-13T14:50:08.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:08.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:08.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:08.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:08.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:08.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:08.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:08.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:08.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:08.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:08.596 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:08.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:08.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:08.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:08.845 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:08.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:08 smithi172 ceph-mon[100696]: pgmap v595: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 26 op/s; 52683/254265 objects degraded (20.720%); 2169/254265 objects misplaced (0.853%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:50:08.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:08 smithi172 ceph-mon[100696]: osdmap e206: 8 total, 8 up, 7 in 2024-09-13T14:50:08.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:08 smithi154 ceph-mon[102314]: pgmap v595: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 26 op/s; 52683/254265 objects degraded (20.720%); 2169/254265 objects misplaced (0.853%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:50:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:08 smithi154 ceph-mon[102314]: osdmap e206: 8 total, 8 up, 7 in 2024-09-13T14:50:08.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:08 smithi154 ceph-mon[107038]: pgmap v595: 89 pgs: 1 active+recovery_wait+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 26 op/s; 52683/254265 objects degraded (20.720%); 2169/254265 objects misplaced (0.853%); 4.7 MiB/s, 74 objects/s recovering 2024-09-13T14:50:08.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:08 smithi154 ceph-mon[107038]: osdmap e206: 8 total, 8 up, 7 in 2024-09-13T14:50:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:08.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:08.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:09.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:09.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:09.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:09.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:09.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:09.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:09.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:09.468 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:09.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:09.592 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:09.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:09.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:09.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:09.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:09.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:09.965 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:09.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:10.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:10.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:10.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:10.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:10.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:10.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:10.560 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:50:10.561 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:10.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:10.582 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:50:10.582 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:10.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:10.598 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:50:10.599 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:10.615 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:50:10.616 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:10.632 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:50:10.632 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:10.649 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:50:10.649 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:10.665 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:50:10.665 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:10.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:10.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:10.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:10.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:10.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:10 smithi154 ceph-mon[102314]: pgmap v598: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1 op/s; 47434/254265 objects degraded (18.655%); 5138/254265 objects misplaced (2.021%); 2.9 MiB/s, 47 objects/s recovering 2024-09-13T14:50:10.842 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 47434/254265 objects degraded (18.655%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:10.842 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:10 smithi154 ceph-mon[107038]: pgmap v598: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1 op/s; 47434/254265 objects degraded (18.655%); 5138/254265 objects misplaced (2.021%); 2.9 MiB/s, 47 objects/s recovering 2024-09-13T14:50:10.842 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 47434/254265 objects degraded (18.655%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:10.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:10.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:10.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:10 smithi172 ceph-mon[100696]: pgmap v598: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1 op/s; 47434/254265 objects degraded (18.655%); 5138/254265 objects misplaced (2.021%); 2.9 MiB/s, 47 objects/s recovering 2024-09-13T14:50:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 47434/254265 objects degraded (18.655%), 10 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:11.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:11.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:11.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:11.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:11.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:11.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:50:11.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:11.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:11.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:11.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:11.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:11.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:11.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:11.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:11.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:11.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:11.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:11.928 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:11.950 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:11.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:11.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:12.004 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:50:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:50:11] "GET /metrics HTTP/1.1" 200 36584 "" "Prometheus/2.51.0" 2024-09-13T14:50:12.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:12.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:12.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:12.196 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:12.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:12.320 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:12.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:12.444 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:12.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:12.568 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:12.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:12.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:12.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:12.816 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:12.863 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:12 smithi154 ceph-mon[102314]: pgmap v599: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 peering, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 8 op/s; 29978/254115 objects degraded (11.797%); 16290/254115 objects misplaced (6.410%); 26 MiB/s, 430 objects/s recovering 2024-09-13T14:50:12.863 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:12 smithi154 ceph-mon[107038]: pgmap v599: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 peering, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 8 op/s; 29978/254115 objects degraded (11.797%); 16290/254115 objects misplaced (6.410%); 26 MiB/s, 430 objects/s recovering 2024-09-13T14:50:12.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:12.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:12.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:12 smithi172 ceph-mon[100696]: pgmap v599: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 peering, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 8 op/s; 29978/254115 objects degraded (11.797%); 16290/254115 objects misplaced (6.410%); 26 MiB/s, 430 objects/s recovering 2024-09-13T14:50:13.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:13.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:13.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:13.186 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:13.233 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:13.233 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:13.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:13.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:13.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:13.435 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:13.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:13.560 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:13.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:13.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:13.686 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:13.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.809 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:13.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:13.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:50:13.879 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:13.879 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:13.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:13.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:13.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:14.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:14.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:14.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:14.184 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:14.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:14.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:14.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:50:14.402 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:14.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.432 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:14.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:14.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:14.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:14.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:14.678 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:14.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:14.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:14.802 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:14.902 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:14 smithi154 ceph-mon[102314]: pgmap v600: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 7 op/s; 32943/254115 objects degraded (12.964%); 13321/254115 objects misplaced (5.242%); 25 MiB/s, 403 objects/s recovering 2024-09-13T14:50:14.903 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:14 smithi154 ceph-mon[107038]: pgmap v600: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 7 op/s; 32943/254115 objects degraded (12.964%); 13321/254115 objects misplaced (5.242%); 25 MiB/s, 403 objects/s recovering 2024-09-13T14:50:14.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:14.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:14.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:14 smithi172 ceph-mon[100696]: pgmap v600: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 peering, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 7 op/s; 32943/254115 objects degraded (12.964%); 13321/254115 objects misplaced (5.242%); 25 MiB/s, 403 objects/s recovering 2024-09-13T14:50:15.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:15.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:15.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:15.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:15.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:15.297 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:15.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:15.420 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:15.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:15.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:15.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:15.668 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:15.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:15.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:15.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:15.916 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:15.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 32943/254115 objects degraded (12.964%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:15.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 32943/254115 objects degraded (12.964%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:15.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 32943/254115 objects degraded (12.964%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:16.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:16.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:16.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:16.162 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:16.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:16.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:16.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:16.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:16.474 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:16.500 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:16.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:16.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:16.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:16.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:16 smithi154 ceph-mon[102314]: pgmap v601: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 6 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 23 MiB/s, 376 objects/s recovering 2024-09-13T14:50:16.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:16 smithi154 ceph-mon[107038]: pgmap v601: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 6 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 23 MiB/s, 376 objects/s recovering 2024-09-13T14:50:16.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:16.783 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:16.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:16.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:16.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:16 smithi172 ceph-mon[100696]: pgmap v601: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 6 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 23 MiB/s, 376 objects/s recovering 2024-09-13T14:50:17.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:17.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:17.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:17.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:17.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:17.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:17.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:17.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:17.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:17.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:17.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:17.647 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:17.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:17.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:17.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:17.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:17.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:18.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:18.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:18.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:18.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:18.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:18.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:18.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:18.393 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:18.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:18.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:18.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:18.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:18 smithi154 ceph-mon[102314]: pgmap v602: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 5 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 18 MiB/s, 299 objects/s recovering 2024-09-13T14:50:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:18 smithi154 ceph-mon[107038]: pgmap v602: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 5 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 18 MiB/s, 299 objects/s recovering 2024-09-13T14:50:18.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:18.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:18.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:18.891 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:18.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:18 smithi172 ceph-mon[100696]: pgmap v602: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 5 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 18 MiB/s, 299 objects/s recovering 2024-09-13T14:50:18.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:19.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:19.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:19.138 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:19.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:19.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:19.356 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:50:19.356 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:19.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:19.388 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:19.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:19.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:19.590 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:50:19.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:19.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/648993391' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:50:19.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:19 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:50:19.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2319722097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:19.618 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/648993391' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:50:19.618 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:19 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:50:19.618 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2319722097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:19.636 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:19.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:19.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:19.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:19.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:19.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:19.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/648993391' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:50:19.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:19 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:50:19.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2319722097' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:20.027 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:20.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:20.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:20.171 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:20.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:20.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:20.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:20.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:20.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:20.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.545 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:20.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:20.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:50:20.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:20.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:20 smithi172 ceph-mon[100696]: pgmap v603: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 16 MiB/s, 262 objects/s recovering 2024-09-13T14:50:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:50:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:20 smithi172 ceph-mon[100696]: osdmap e207: 8 total, 8 up, 7 in 2024-09-13T14:50:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:20 smithi172 ceph-mon[100696]: osdmap e208: 8 total, 8 up, 7 in 2024-09-13T14:50:20.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:20.807 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:20.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[102314]: pgmap v603: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 16 MiB/s, 262 objects/s recovering 2024-09-13T14:50:20.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:50:20.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[102314]: osdmap e207: 8 total, 8 up, 7 in 2024-09-13T14:50:20.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[102314]: osdmap e208: 8 total, 8 up, 7 in 2024-09-13T14:50:20.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[107038]: pgmap v603: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 32943/254115 objects degraded (12.964%); 12895/254115 objects misplaced (5.074%); 16 MiB/s, 262 objects/s recovering 2024-09-13T14:50:20.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:50:20.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[107038]: osdmap e207: 8 total, 8 up, 7 in 2024-09-13T14:50:20.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:20 smithi154 ceph-mon[107038]: osdmap e208: 8 total, 8 up, 7 in 2024-09-13T14:50:20.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:20.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:20.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:21.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.056 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:21.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:21.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:21.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:21.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:21.302 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:21.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:21.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:21.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:21.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:21.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:21.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:21.700 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:21.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:21.789 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:50:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:21.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:21.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:21.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:21.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:21.960 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:21 smithi154 ceph-mon[102314]: osdmap e209: 8 total, 8 up, 7 in 2024-09-13T14:50:21.960 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:21 smithi154 ceph-mon[102314]: 4.1d continuing backfill to osd.3 from (0'0,0'0] MIN to 0'0 2024-09-13T14:50:21.960 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2580598386' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:21 smithi154 ceph-mon[107038]: osdmap e209: 8 total, 8 up, 7 in 2024-09-13T14:50:21.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:21 smithi154 ceph-mon[107038]: 4.1d continuing backfill to osd.3 from (0'0,0'0] MIN to 0'0 2024-09-13T14:50:21.961 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2580598386' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:21.961 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:50:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:50:21] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-13T14:50:22.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:22.057 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:50:22.057 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:22.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:22.080 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:50:22.080 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:22.112 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:50:22.112 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:22.140 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:50:22.141 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:22.157 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:50:22.157 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:22.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:22.193 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:50:22.193 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:22.196 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:22.215 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:50:22.215 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:21 smithi172 ceph-mon[100696]: osdmap e209: 8 total, 8 up, 7 in 2024-09-13T14:50:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:21 smithi172 ceph-mon[100696]: 4.1d continuing backfill to osd.3 from (0'0,0'0] MIN to 0'0 2024-09-13T14:50:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2580598386' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:50:22.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:22.321 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:22.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:22.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:22.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:22.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:22.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:22.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:22.692 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:22.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:22.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:22.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:22.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:22.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:23.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:23.070 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[102314]: pgmap v607: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 32943/254115 objects degraded (12.964%); 12419/254115 objects misplaced (4.887%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:50:23.070 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:23.070 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2580598386' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:23.070 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[102314]: osdmap e210: 8 total, 8 up, 7 in 2024-09-13T14:50:23.070 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[107038]: pgmap v607: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 32943/254115 objects degraded (12.964%); 12419/254115 objects misplaced (4.887%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:50:23.071 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:23.071 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2580598386' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:23.071 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:22 smithi154 ceph-mon[107038]: osdmap e210: 8 total, 8 up, 7 in 2024-09-13T14:50:23.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:23.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:23.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:22 smithi172 ceph-mon[100696]: pgmap v607: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 32943/254115 objects degraded (12.964%); 12419/254115 objects misplaced (4.887%); 5.0 MiB/s, 79 objects/s recovering 2024-09-13T14:50:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:22 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:22 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2580598386' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:50:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:22 smithi172 ceph-mon[100696]: osdmap e210: 8 total, 8 up, 7 in 2024-09-13T14:50:23.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:23.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:23.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.451 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:23.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:23.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:23.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:23.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:23.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:23.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:23.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:23.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:23.971 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:23.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:23.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:23 smithi154 ceph-mon[102314]: osdmap e211: 8 total, 8 up, 7 in 2024-09-13T14:50:23.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:23 smithi154 ceph-mon[102314]: osdmap e212: 8 total, 8 up, 7 in 2024-09-13T14:50:23.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:23 smithi154 ceph-mon[107038]: osdmap e211: 8 total, 8 up, 7 in 2024-09-13T14:50:23.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:23 smithi154 ceph-mon[107038]: osdmap e212: 8 total, 8 up, 7 in 2024-09-13T14:50:24.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:24.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27212 5.a 5:58ee1d02:::benchmark_data_smithi154_144140_object27211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:50:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.79164193000003, 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17096689200000001, 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27212 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664198+0000", 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9800000000000003e-07 2024-09-13T14:50:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664220+0000", 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1591999999999999e-05 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664227+0000", 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.712e-06 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664233+0000", 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8289999999999996e-06 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669218+0000", 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049845829999999999 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669320+0000", 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102304 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669434+0000", 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113581 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674379+0000", 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00494525 2024-09-13T14:50:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674700+0000", 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032070699999999999 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.834993+0000", 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160292992 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835121+0000", 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012801800000000001 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835164+0000", 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3726000000000001e-05 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27213 5.a 5:571e06b1:::benchmark_data_smithi154_144140_object27212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.79159789800002, 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17104000799999999, 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27213 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.182 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664242+0000", 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2e-07 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664265+0000", 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3476e-05 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664273+0000", 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2230000000000001e-06 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664278+0000", 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1599999999999997e-06 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.670614+0000", 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063359059999999997 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671024+0000", 2024-09-13T14:50:24.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041025499999999999 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671138+0000", 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114418 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674735+0000", 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035970080000000001 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674771+0000", 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5506999999999997e-05 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835177+0000", 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16040633400000001 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835217+0000", 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9641000000000002e-05 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835281+0000", 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4659999999999994e-05 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27221 5.a 5:51807044:::benchmark_data_smithi154_144140_object27220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:50:24.188 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78744433700001, 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16984976800000001, 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27221 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668396+0000", 2024-09-13T14:50:24.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9999999999999997e-07 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668416+0000", 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0516000000000001e-05 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668424+0000", 2024-09-13T14:50:24.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6389999999999992e-06 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668430+0000", 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9939999999999997e-06 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672751+0000", 2024-09-13T14:50:24.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043213569999999996 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673077+0000", 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032614299999999998 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673198+0000", 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120492 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833603+0000", 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160404986 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835418+0000", 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018152349999999999 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838152+0000", 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002733989 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838206+0000", 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4165000000000001e-05 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838245+0000", 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8652000000000002e-05 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.195 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27226 5.a 5:58f709a9:::benchmark_data_smithi154_144140_object27225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.786683627, 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.169199034, 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27226 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1900000000000001e-07 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669187+0000", 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0571999999999998e-05 2024-09-13T14:50:24.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669198+0000", 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0920999999999999e-05 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669205+0000", 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6479999999999997e-06 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674834+0000", 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056293519999999998 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674951+0000", 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011713 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675069+0000", 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117726 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835380+0000", 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160310762 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835455+0000", 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5813000000000001e-05 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838286+0000", 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028307559999999998 2024-09-13T14:50:24.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838325+0000", 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8574e-05 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838355+0000", 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9961e-05 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27229 5.2 e117/111 5:47a3bf38:::benchmark_data_smithi154_144140_object27228:head v 117'1668, mlcod=117'1665)", 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78460764900001, 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16704787199999999, 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:50:24.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5599999999999995e-07 2024-09-13T14:50:24.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671265+0000", 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3231000000000001e-05 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671274+0000", 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9479999999999997e-06 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671280+0000", 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672510+0000", 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012303030000000001 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672591+0000", 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1227e-05 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838248+0000", 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16565628499999999 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838280+0000", 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1881999999999997e-05 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27212 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78450767999999, 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16395237200000001, 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8700000000000001e-07 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671335+0000", 2024-09-13T14:50:24.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.858e-06 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671342+0000", 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4189999999999996e-06 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671347+0000", 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9089999999999999e-06 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674368+0000", 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003020371 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674374+0000", 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2829999999999998e-06 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835284+0000", 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160910145 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27231 5.d 5:bd1c1b1b:::benchmark_data_smithi154_144140_object27230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78418868099999, 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16226318100000001, 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27231 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:50:24.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671652+0000", 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0100000000000003e-07 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671671+0000", 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9686000000000001e-05 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671680+0000", 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6989999999999992e-06 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671685+0000", 2024-09-13T14:50:24.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2950000000000001e-06 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671954+0000", 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026847900000000002 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672066+0000", 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011244 2024-09-13T14:50:24.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672249+0000", 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000182573 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674493+0000", 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022442439999999998 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674564+0000", 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1538000000000005e-05 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675871+0000", 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013069449999999999 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833838+0000", 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15796623000000001 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833914+0000", 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6150999999999996e-05 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:50:24.219 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78076399299999, 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158535863, 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675076+0000", 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5400000000000002e-07 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675079+0000", 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.365e-06 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675087+0000", 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6249999999999998e-06 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675093+0000", 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8300000000000001e-06 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833585+0000", 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158492884 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833596+0000", 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0645999999999999e-05 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833611+0000", 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5058999999999999e-05 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.223 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27237 5.b 5:d9e8c993:::benchmark_data_smithi154_144140_object27236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78075494799998, 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16640950199999999, 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.224 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27237 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675086+0000", 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.274e-06 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675108+0000", 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2412000000000002e-05 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675120+0000", 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1729000000000001e-05 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675127+0000", 2024-09-13T14:50:24.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7680000000000003e-06 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675594+0000", 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000466992 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675717+0000", 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012363700000000001 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833850+0000", 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158133311 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836021+0000", 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021703880000000001 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836035+0000", 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3657e-05 2024-09-13T14:50:24.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841412+0000", 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053777620000000003 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841458+0000", 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5235000000000003e-05 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841494+0000", 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6337000000000001e-05 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27239 5.2 e117/111 5:4246bfbf:::benchmark_data_smithi154_144140_object27238:head v 117'1669, mlcod=117'1665)", 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78028906999998, 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16283244499999999, 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675551+0000", 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4000000000000001e-07 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675624+0000", 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3522000000000005e-05 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675633+0000", 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0499999999999997e-06 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675639+0000", 2024-09-13T14:50:24.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8839999999999999e-06 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833644+0000", 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158004476 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833663+0000", 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9238000000000001e-05 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838359+0000", 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046954900000000001 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838383+0000", 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4145e-05 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.235 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27231 5.d e117/111 ondisk, result = 0)", 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78014878900001, 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15823179200000001, 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8500000000000001e-07 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675695+0000", 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9099999999999998e-06 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675705+0000", 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6670000000000008e-06 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675710+0000", 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7810000000000002e-06 2024-09-13T14:50:24.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675848+0000", 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137677 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675855+0000", 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1790000000000002e-06 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833922+0000", 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15806699299999999 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27241 5.6 5:6b00c174:::benchmark_data_smithi154_144140_object27240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78010125600002, 2024-09-13T14:50:24.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16225515099999999, 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27241 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675739+0000", 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6499999999999999e-07 2024-09-13T14:50:24.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675758+0000", 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8799000000000001e-05 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675766+0000", 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4649999999999994e-06 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675772+0000", 2024-09-13T14:50:24.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6470000000000003e-06 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675823+0000", 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1010000000000001e-05 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675943+0000", 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120389 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833416+0000", 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15747296699999999 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835725+0000", 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023092400000000002 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835762+0000", 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6693000000000003e-05 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837162+0000", 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001400152 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837949+0000", 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000787417 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837993+0000", 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3707e-05 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27240 5.e e117/111 5:77f483c1:::benchmark_data_smithi154_144140_object27239:head v 117'1698, mlcod=117'1696)", 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.78008972800001, 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16227988300000001, 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.247 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675755+0000", 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5079999999999996e-06 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675810+0000", 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.452e-05 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675818+0000", 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8659999999999995e-06 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675824+0000", 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7930000000000003e-06 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675915+0000", 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0556000000000005e-05 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675932+0000", 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6979000000000001e-05 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837997+0000", 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16206567699999999 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838029+0000", 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1983999999999997e-05 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27226 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.77895009500003, 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15849907399999999, 2024-09-13T14:50:24.252 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676890+0000", 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7599999999999997e-07 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676894+0000", 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1590000000000003e-06 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833403+0000", 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.156509065 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833418+0000", 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5142e-05 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835366+0000", 2024-09-13T14:50:24.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019481570000000001 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835374+0000", 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5639999999999999e-06 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835388+0000", 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4411e-05 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.256 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 414.77887889200002, 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15846376300000001, 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676961+0000", 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4199999999999995e-07 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676965+0000", 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1130000000000003e-06 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833821+0000", 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15685534100000001 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833830+0000", 2024-09-13T14:50:24.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1130000000000007e-06 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835407+0000", 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015777499999999999 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835413+0000", 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835424+0000", 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0829000000000001e-05 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:117172 5.a 5:5df8026a:::benchmark_data_smithi154_144140_object6819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 325.29392576999999, 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15741727699999999, 2024-09-13T14:50:24.261 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 117172 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:50:24.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161915+0000", 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.314e-06 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161919+0000", 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6569999999999996e-06 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161933+0000", 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3055e-05 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161940+0000", 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.593e-06 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164296+0000", 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023560709999999999 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164406+0000", 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011003899999999999 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164510+0000", 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010334400000000001 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.175396+0000", 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010886682 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179579+0000", 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041824269999999998 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319249+0000", 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13966985800000001 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319294+0000", 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5141e-05 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319331+0000", 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7095999999999999e-05 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:117176 5.a 5:574be98d:::benchmark_data_smithi154_144140_object6823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:50:24.268 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 325.29372926500002, 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15802838399999999, 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 117176 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162111+0000", 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5499999999999997e-07 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162115+0000", 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6679999999999999e-06 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162124+0000", 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3700000000000001e-06 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162130+0000", 2024-09-13T14:50:24.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.789e-06 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171194+0000", 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090644290000000006 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171302+0000", 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108194 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171403+0000", 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100468 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179555+0000", 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008152365 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179585+0000", 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0063e-05 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320052+0000", 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14046650699999999 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320103+0000", 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0924000000000002e-05 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320138+0000", 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5751999999999999e-05 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:117185 5.f e132/130 5:f308a48e:::benchmark_data_smithi154_144140_object6832:head v 132'7432, mlcod=125'5196)", 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 325.28102689899998, 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.151216923, 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.275 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174813+0000", 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7100000000000001e-07 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174820+0000", 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8739999999999997e-06 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174828+0000", 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4130000000000006e-06 2024-09-13T14:50:24.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174834+0000", 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8370000000000002e-06 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319258+0000", 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14442416499999999 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319282+0000", 2024-09-13T14:50:24.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3923000000000001e-05 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.325925+0000", 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066427910000000003 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.326029+0000", 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104249 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.279 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:117851 5.e e133/111 5:700ea90e:::benchmark_data_smithi154_144140_object7498:head v 133'7344, mlcod=133'7342)", 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 324.35920886899999, 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.26475042900000001, 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096632+0000", 2024-09-13T14:50:24.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.158e-06 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096640+0000", 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4800000000000001e-06 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219022+0000", 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.122381897 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219039+0000", 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6616000000000002e-05 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219381+0000", 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000342443 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219401+0000", 2024-09-13T14:50:24.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9698e-05 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.361334+0000", 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.141933647 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.361381+0000", 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6489999999999997e-05 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:197886 5.d 5:ba09dd65:::benchmark_data_smithi154_144140_object87533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 222.44295183, 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15809542200000001, 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 197886 2024-09-13T14:50:24.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:24.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012889+0000", 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2810000000000001e-06 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012893+0000", 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.771e-06 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012906+0000", 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.329e-05 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012913+0000", 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3089999999999998e-06 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012969+0000", 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.013085+0000", 2024-09-13T14:50:24.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115708 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.013194+0000", 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010938300000000001 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.018482+0000", 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052877269999999999 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.125895+0000", 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.107413177 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170889+0000", 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.044993742000000003 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170944+0000", 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4966999999999997e-05 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:24.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170983+0000", 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8781e-05 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:24.291 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:50:24.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:24.292 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:23 smithi172 ceph-mon[100696]: osdmap e211: 8 total, 8 up, 7 in 2024-09-13T14:50:24.292 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:23 smithi172 ceph-mon[100696]: osdmap e212: 8 total, 8 up, 7 in 2024-09-13T14:50:24.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:24.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:24.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:24.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:24.560 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:24.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:24.684 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:24.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:24.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:24.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:50:24.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:24.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:24.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:25.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:25.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:25.069 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:24 smithi154 ceph-mon[102314]: pgmap v610: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 2 activating, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 27527/254115 objects degraded (10.832%); 12419/254115 objects misplaced (4.887%) 2024-09-13T14:50:25.069 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:24 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:50:25.069 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:24 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 27527/254115 objects degraded (10.832%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:25.069 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:24 smithi154 ceph-mon[107038]: pgmap v610: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 2 activating, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 27527/254115 objects degraded (10.832%); 12419/254115 objects misplaced (4.887%) 2024-09-13T14:50:25.069 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:24 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:50:25.069 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:24 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 27527/254115 objects degraded (10.832%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:25.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:25.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:24 smithi172 ceph-mon[100696]: pgmap v610: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 2 activating, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovery_wait+undersized+degraded+remapped, 74 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 27527/254115 objects degraded (10.832%); 12419/254115 objects misplaced (4.887%) 2024-09-13T14:50:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:24 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:50:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:24 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 27527/254115 objects degraded (10.832%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:25.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:25.307 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:25.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:25.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:25.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:25.557 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:25.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.681 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:25.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:25.782 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:25.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:25.906 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:25.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:25.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:25.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:25 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-13T14:50:25.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:25 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-13T14:50:26.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:26.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:26.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:26.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:25 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-13T14:50:26.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:26.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:26.377 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:26.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:26.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:26.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:26.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:26.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:26.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:26.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:26.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:26.860 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:50:26.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:26.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:26.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:26.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:26 smithi154 ceph-mon[102314]: pgmap v612: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 1 activating, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 26628/254115 objects degraded (10.479%); 8241/254115 objects misplaced (3.243%); 721 KiB/s, 11 objects/s recovering 2024-09-13T14:50:26.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2750703907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:50:26.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:26 smithi154 ceph-mon[107038]: pgmap v612: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 1 activating, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 26628/254115 objects degraded (10.479%); 8241/254115 objects misplaced (3.243%); 721 KiB/s, 11 objects/s recovering 2024-09-13T14:50:26.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2750703907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:50:27.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:27.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:26 smithi172 ceph-mon[100696]: pgmap v612: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 1 activating, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 26628/254115 objects degraded (10.479%); 8241/254115 objects misplaced (3.243%); 721 KiB/s, 11 objects/s recovering 2024-09-13T14:50:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2750703907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:50:27.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:27.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:27.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:27.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:27.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:50:27.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.680 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:27.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:27.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:27.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:27.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:27.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:50:27.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:27.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:27.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:28.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:28.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:50:28.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:28.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2750703907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:50:28.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:27 smithi154 ceph-mon[102314]: osdmap e213: 8 total, 8 up, 7 in 2024-09-13T14:50:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2750703907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:50:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:27 smithi154 ceph-mon[107038]: osdmap e213: 8 total, 8 up, 7 in 2024-09-13T14:50:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2750703907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:50:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:27 smithi172 ceph-mon[100696]: osdmap e213: 8 total, 8 up, 7 in 2024-09-13T14:50:28.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:28.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:28.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:28.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:28.461 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:28.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:28.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:28.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:28.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:28.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:28.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:28.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:28.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:28.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:29.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:29.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:29.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:29.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:28 smithi154 ceph-mon[102314]: pgmap v614: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 1 activating, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 26628/254115 objects degraded (10.479%); 8241/254115 objects misplaced (3.243%); 583 KiB/s, 9 objects/s recovering 2024-09-13T14:50:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:28 smithi154 ceph-mon[107038]: pgmap v614: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 1 activating, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 26628/254115 objects degraded (10.479%); 8241/254115 objects misplaced (3.243%); 583 KiB/s, 9 objects/s recovering 2024-09-13T14:50:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:28 smithi172 ceph-mon[100696]: pgmap v614: 89 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 1 peering, 1 activating, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 26628/254115 objects degraded (10.479%); 8241/254115 objects misplaced (3.243%); 583 KiB/s, 9 objects/s recovering 2024-09-13T14:50:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:29.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:29.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:29.361 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:29.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:29.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:29.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:29.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:29.637 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:29.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:29.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:29.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:29.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:29.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:29.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:29.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:29.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:29 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 26628/254115 objects degraded (10.479%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:29.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:29 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 26628/254115 objects degraded (10.479%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:30.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:30.070 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:30.112 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:50:30.112 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:50:30.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.112 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86887 6.c 6.9f67740c (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T14:50:30.112 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:50:30.112 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.3213824770000002, 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3214081450000004, 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for peered", 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86887 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:30.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789254+0000", 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6649999999999999e-06 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789262+0000", 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.542e-06 2024-09-13T14:50:30.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:30.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789278+0000", 2024-09-13T14:50:30.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6351e-05 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789288+0000", 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0358e-05 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789368+0000", 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9726000000000002e-05 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T14:50:30.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789377+0000", 2024-09-13T14:50:30.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1609999999999992e-06 2024-09-13T14:50:30.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.796583+0000", 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072051260000000001 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.796594+0000", 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1331e-05 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:30.119 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:50:30.120 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 1 2024-09-13T14:50:30.120 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:50:30.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.194 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:30.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:29 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 26628/254115 objects degraded (10.479%), 8 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:50:30.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:30.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:30.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:30.442 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:30.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:30.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.567 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:30.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:30.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:30.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:30.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:30.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:30.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:30.937 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:30.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:30 smithi154 ceph-mon[102314]: pgmap v615: 89 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 1 peering, 1 activating, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 21419/254115 objects degraded (8.429%); 2073/254115 objects misplaced (0.816%); 9.4 MiB/s, 150 objects/s recovering 2024-09-13T14:50:30.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:30 smithi154 ceph-mon[107038]: pgmap v615: 89 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 1 peering, 1 activating, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 21419/254115 objects degraded (8.429%); 2073/254115 objects misplaced (0.816%); 9.4 MiB/s, 150 objects/s recovering 2024-09-13T14:50:30.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:50:30 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T14:50:30.685+0000 7f48519c1640 -1 osd.1 213 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.44843.0:86673 6.4 6.968b0b24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e202) 2024-09-13T14:50:31.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:31.060 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:31.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:31.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:30 smithi172 ceph-mon[100696]: pgmap v615: 89 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 1 peering, 1 activating, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 21419/254115 objects degraded (8.429%); 2073/254115 objects misplaced (0.816%); 9.4 MiB/s, 150 objects/s recovering 2024-09-13T14:50:31.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:31.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:31.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:31.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:31.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:31.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:31.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:31.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:31.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:31.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:31.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:31.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:31.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:31.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:31 smithi154 ceph-mon[102314]: 15 slow requests (by type [ 'delayed' : 15 ] most affected pool [ 'unique_pool_1' : 15 ]) 2024-09-13T14:50:31.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:50:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:50:31] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:50:31.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:31 smithi154 ceph-mon[107038]: 15 slow requests (by type [ 'delayed' : 15 ] most affected pool [ 'unique_pool_1' : 15 ]) 2024-09-13T14:50:31.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:50:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T14:50:31.691+0000 7f48519c1640 -1 osd.1 213 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.44843.0:86673 6.4 6.968b0b24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e202) 2024-09-13T14:50:32.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:32.052 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:32.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:32.176 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:31 smithi172 ceph-mon[100696]: 15 slow requests (by type [ 'delayed' : 15 ] most affected pool [ 'unique_pool_1' : 15 ]) 2024-09-13T14:50:32.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:32.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:32.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:50:32.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:32.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:32.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:32.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:32.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:32.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:32.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.799 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:32.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:32.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:32.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:32.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:32 smithi154 ceph-mon[102314]: pgmap v616: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 23762/254115 objects degraded (9.351%); 2073/254115 objects misplaced (0.816%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:50:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:32 smithi154 ceph-mon[102314]: 15 slow requests (by type [ 'delayed' : 15 ] most affected pool [ 'unique_pool_1' : 15 ]) 2024-09-13T14:50:32.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:32 smithi154 ceph-mon[107038]: pgmap v616: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 23762/254115 objects degraded (9.351%); 2073/254115 objects misplaced (0.816%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:50:32.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:32 smithi154 ceph-mon[107038]: 15 slow requests (by type [ 'delayed' : 15 ] most affected pool [ 'unique_pool_1' : 15 ]) 2024-09-13T14:50:32.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 14:50:32 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T14:50:32.642+0000 7f48519c1640 -1 osd.1 213 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.44843.0:86716 6.4 6:26c9dec4:::benchmark_data_smithi154_175640_object940:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202) 2024-09-13T14:50:33.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:33.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:33.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:33.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:32 smithi172 ceph-mon[100696]: pgmap v616: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 23762/254115 objects degraded (9.351%); 2073/254115 objects misplaced (0.816%); 7.2 MiB/s, 115 objects/s recovering 2024-09-13T14:50:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:32 smithi172 ceph-mon[100696]: 15 slow requests (by type [ 'delayed' : 15 ] most affected pool [ 'unique_pool_1' : 15 ]) 2024-09-13T14:50:33.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:33.310 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:33.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:33.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:33.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:33.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:33.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:33.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:33.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:33.851 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:33.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:33.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:33.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:34.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:34.101 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:33 smithi154 ceph-mon[102314]: 4 slow requests (by type [ 'delayed' : 2 'waiting for sub ops' : 2 ] most affected pool [ 'unique_pool_1' : 4 ]) 2024-09-13T14:50:34.101 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:33 smithi154 ceph-mon[107038]: 4 slow requests (by type [ 'delayed' : 2 'waiting for sub ops' : 2 ] most affected pool [ 'unique_pool_1' : 4 ]) 2024-09-13T14:50:34.146 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:34.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.222 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:34.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:33 smithi172 ceph-mon[100696]: 4 slow requests (by type [ 'delayed' : 2 'waiting for sub ops' : 2 ] most affected pool [ 'unique_pool_1' : 4 ]) 2024-09-13T14:50:34.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:34.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:34.419 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:50:34.419 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:34.447 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:50:34.447 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:34.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:34.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:34.477 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:50:34.477 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:34.505 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:50:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:34.525 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:50:34.526 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:34.551 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:50:34.551 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:34.571 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:50:34.571 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:34.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:34.596 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:50:34.596 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:34.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:34.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:34.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:34.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:34.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:34.913 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:50:34.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:34.986 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:34.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:35.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:35.110 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:35.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:35.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[102314]: pgmap v617: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 0 op/s; 23762/254088 objects degraded (9.352%); 1107/254088 objects misplaced (0.436%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:50:35.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3868948741' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:35.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:35.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3165671300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:35.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 23762/254088 objects degraded (9.352%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:35.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[107038]: pgmap v617: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 0 op/s; 23762/254088 objects degraded (9.352%); 1107/254088 objects misplaced (0.436%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:50:35.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3868948741' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:35.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:35.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3165671300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:35.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:34 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 23762/254088 objects degraded (9.352%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:35.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:34 smithi172 ceph-mon[100696]: pgmap v617: 89 pgs: 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 6 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 0 op/s; 23762/254088 objects degraded (9.352%); 1107/254088 objects misplaced (0.436%); 13 MiB/s, 206 objects/s recovering 2024-09-13T14:50:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:34 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3868948741' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:34 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:50:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:34 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3165671300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:34 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 23762/254088 objects degraded (9.352%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:35.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:35.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:35.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:35.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:35.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:35.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:35.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:35.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:35.746 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:35.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:35.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:35.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:35.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:36.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:36.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:36.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:36.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[102314]: osdmap e214: 8 total, 8 up, 7 in 2024-09-13T14:50:36.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[102314]: pgmap v619: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 43 op/s; 19196/253047 objects degraded (7.586%); 5346/253047 objects misplaced (2.113%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:50:36.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:36.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:36.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[107038]: osdmap e214: 8 total, 8 up, 7 in 2024-09-13T14:50:36.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:35 smithi154 ceph-mon[107038]: pgmap v619: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 43 op/s; 19196/253047 objects degraded (7.586%); 5346/253047 objects misplaced (2.113%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:50:36.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:36.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:35 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:50:36.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:35 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:50:36.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:35 smithi172 ceph-mon[100696]: osdmap e214: 8 total, 8 up, 7 in 2024-09-13T14:50:36.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:35 smithi172 ceph-mon[100696]: pgmap v619: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 43 op/s; 19196/253047 objects degraded (7.586%); 5346/253047 objects misplaced (2.113%); 15 MiB/s, 246 objects/s recovering 2024-09-13T14:50:36.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:36.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:36.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:36.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:36.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:36.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:36.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:36.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:36.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:36.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:36.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:36.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:36.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:50:36.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:37.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:37.040 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3218106483' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:50:37.040 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:36 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:50:37.040 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3218106483' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:50:37.040 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:36 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:50:37.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:37.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3218106483' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:50:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:36 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:50:37.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:37.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:37.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.408 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:37.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:37.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.534 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:37.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:37.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:37.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:37.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.784 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:37.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:37.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:37.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:37.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:38.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:38.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:38.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:38.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:37 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:50:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:37 smithi154 ceph-mon[102314]: osdmap e215: 8 total, 8 up, 7 in 2024-09-13T14:50:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:37 smithi154 ceph-mon[102314]: pgmap v621: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 44 op/s; 19196/253047 objects degraded (7.586%); 5346/253047 objects misplaced (2.113%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T14:50:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:37 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:50:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:37 smithi154 ceph-mon[107038]: osdmap e215: 8 total, 8 up, 7 in 2024-09-13T14:50:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:37 smithi154 ceph-mon[107038]: pgmap v621: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 44 op/s; 19196/253047 objects degraded (7.586%); 5346/253047 objects misplaced (2.113%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T14:50:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:37 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:50:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:37 smithi172 ceph-mon[100696]: osdmap e215: 8 total, 8 up, 7 in 2024-09-13T14:50:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:37 smithi172 ceph-mon[100696]: pgmap v621: 89 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 44 op/s; 19196/253047 objects degraded (7.586%); 5346/253047 objects misplaced (2.113%); 8.7 MiB/s, 139 objects/s recovering 2024-09-13T14:50:38.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:38.291 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:38.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:38.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:38.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:38.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:38.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:38.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:38.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:50:38.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:38.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:50:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:38.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:38.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:38.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:38.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:39.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:39.074 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:39.076 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:38 smithi154 ceph-mon[107038]: osdmap e216: 8 total, 8 up, 7 in 2024-09-13T14:50:39.077 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:38 smithi154 ceph-mon[102314]: osdmap e216: 8 total, 8 up, 7 in 2024-09-13T14:50:39.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:39.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:38 smithi172 ceph-mon[100696]: osdmap e216: 8 total, 8 up, 7 in 2024-09-13T14:50:39.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:39.327 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:50:39.327 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:50:39.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:39.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:39.456 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:39.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:39.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:39.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:39.695 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:39.704 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:39.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:39.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:39.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:39.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:50:39.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:39.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:39.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:39 smithi154 ceph-mon[102314]: pgmap v624: 89 pgs: 2 remapped+peering, 3 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 151 op/s; 14289/251211 objects degraded (5.688%); 4239/251211 objects misplaced (1.687%); 348 KiB/s, 5 objects/s recovering 2024-09-13T14:50:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:39 smithi154 ceph-mon[102314]: osdmap e217: 8 total, 8 up, 7 in 2024-09-13T14:50:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:39 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19196/253047 objects degraded (7.586%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:39.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:39 smithi154 ceph-mon[107038]: pgmap v624: 89 pgs: 2 remapped+peering, 3 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 151 op/s; 14289/251211 objects degraded (5.688%); 4239/251211 objects misplaced (1.687%); 348 KiB/s, 5 objects/s recovering 2024-09-13T14:50:39.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:39 smithi154 ceph-mon[107038]: osdmap e217: 8 total, 8 up, 7 in 2024-09-13T14:50:39.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:39 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19196/253047 objects degraded (7.586%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:40.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:40.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:40.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:40.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:39 smithi172 ceph-mon[100696]: pgmap v624: 89 pgs: 2 remapped+peering, 3 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 151 op/s; 14289/251211 objects degraded (5.688%); 4239/251211 objects misplaced (1.687%); 348 KiB/s, 5 objects/s recovering 2024-09-13T14:50:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:39 smithi172 ceph-mon[100696]: osdmap e217: 8 total, 8 up, 7 in 2024-09-13T14:50:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:39 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19196/253047 objects degraded (7.586%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:50:40.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:40.343 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:40.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:40.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:40.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:40.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:40.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:40.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:40.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:40.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:40.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:40.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:40.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:40.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:41.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:41.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:41.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:41.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:41.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:41.351 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:41.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:41.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:41.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:41.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:41.733 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:41.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:41.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:41.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:41.989 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:41.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:42.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:42.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:42.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:41 smithi154 ceph-mon[102314]: pgmap v625: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 14338/249840 objects degraded (5.739%); 3628/249840 objects misplaced (1.452%); 6.0 MiB/s, 100 objects/s recovering 2024-09-13T14:50:42.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:41 smithi154 ceph-mon[107038]: pgmap v625: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 14338/249840 objects degraded (5.739%); 3628/249840 objects misplaced (1.452%); 6.0 MiB/s, 100 objects/s recovering 2024-09-13T14:50:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:50:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:50:41] "GET /metrics HTTP/1.1" 200 36550 "" "Prometheus/2.51.0" 2024-09-13T14:50:42.240 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:41 smithi172 ceph-mon[100696]: pgmap v625: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 14338/249840 objects degraded (5.739%); 3628/249840 objects misplaced (1.452%); 6.0 MiB/s, 100 objects/s recovering 2024-09-13T14:50:42.244 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:42.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:42.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:42.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:42.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:42.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:42.622 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:42.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:42.746 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:42.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:42.873 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:42.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:42.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:43.127 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:43.229 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:43.252 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:43.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:43.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:43.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:43.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:43.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:43.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:43.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:43.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:43.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:43.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:43.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:43.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:44.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:44.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:44.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:43 smithi154 ceph-mon[102314]: pgmap v626: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 177 op/s; 14338/249837 objects degraded (5.739%); 3628/249837 objects misplaced (1.452%); 6.0 MiB/s, 100 objects/s recovering 2024-09-13T14:50:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:43 smithi154 ceph-mon[107038]: pgmap v626: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 177 op/s; 14338/249837 objects degraded (5.739%); 3628/249837 objects misplaced (1.452%); 6.0 MiB/s, 100 objects/s recovering 2024-09-13T14:50:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:44.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:43 smithi172 ceph-mon[100696]: pgmap v626: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 177 op/s; 14338/249837 objects degraded (5.739%); 3628/249837 objects misplaced (1.452%); 6.0 MiB/s, 100 objects/s recovering 2024-09-13T14:50:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:44.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:44.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:44.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:44.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:44.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:44.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:44.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:44.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:44.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:44.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:44.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:44.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:45.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:45.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:45.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:45.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 14338/249837 objects degraded (5.739%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:45.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 14338/249837 objects degraded (5.739%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 14338/249837 objects degraded (5.739%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:45.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:45.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:45.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:45.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:45.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:45.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:45.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:50:45.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:45.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:45.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:45.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:45.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:45.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:45.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:45.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:46.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:46.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:46.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:46.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:46.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:45 smithi154 ceph-mon[102314]: pgmap v627: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 134 op/s; 14338/249828 objects degraded (5.739%); 2868/249828 objects misplaced (1.148%); 10 MiB/s, 170 objects/s recovering 2024-09-13T14:50:46.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:45 smithi154 ceph-mon[107038]: pgmap v627: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 134 op/s; 14338/249828 objects degraded (5.739%); 2868/249828 objects misplaced (1.148%); 10 MiB/s, 170 objects/s recovering 2024-09-13T14:50:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:45 smithi172 ceph-mon[100696]: pgmap v627: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 134 op/s; 14338/249828 objects degraded (5.739%); 2868/249828 objects misplaced (1.148%); 10 MiB/s, 170 objects/s recovering 2024-09-13T14:50:46.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:46.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:46.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:46.444 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:46.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:46.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:46.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:46.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:46.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:46.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:46.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:46.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:46.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:47.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:47.078 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:47.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:47.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:47.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:47.333 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:47.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:47.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:47.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:47.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:47.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:47.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:47.712 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:47.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:47.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:47.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:47.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:47.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:48.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:48.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:48.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:47 smithi154 ceph-mon[102314]: pgmap v628: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 77 op/s; 14338/249828 objects degraded (5.739%); 2868/249828 objects misplaced (1.148%); 9.1 MiB/s, 148 objects/s recovering 2024-09-13T14:50:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:47 smithi154 ceph-mon[107038]: pgmap v628: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 77 op/s; 14338/249828 objects degraded (5.739%); 2868/249828 objects misplaced (1.148%); 9.1 MiB/s, 148 objects/s recovering 2024-09-13T14:50:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:48.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:47 smithi172 ceph-mon[100696]: pgmap v628: 89 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 77 op/s; 14338/249828 objects degraded (5.739%); 2868/249828 objects misplaced (1.148%); 9.1 MiB/s, 148 objects/s recovering 2024-09-13T14:50:48.300 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:48.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:48.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:48.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:48.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:48.599 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:48.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:48.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:48.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:48.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:48.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:48.975 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:48.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:49.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:49.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:49.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:49.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:49.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:49.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:48 smithi172 ceph-mon[100696]: osdmap e218: 8 total, 8 up, 7 in 2024-09-13T14:50:49.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.354 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:49.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:49.443 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:48 smithi154 ceph-mon[102314]: osdmap e218: 8 total, 8 up, 7 in 2024-09-13T14:50:49.444 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:48 smithi154 ceph-mon[107038]: osdmap e218: 8 total, 8 up, 7 in 2024-09-13T14:50:49.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:49.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:49.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:49.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:49.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:49.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:49.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:49.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:49.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:49.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:49.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:49.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:50.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:50.103 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:50.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:49 smithi154 ceph-mon[102314]: pgmap v630: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 47 op/s; 22601/249792 objects degraded (9.048%); 2868/249792 objects misplaced (1.148%); 8.2 MiB/s, 134 objects/s recovering 2024-09-13T14:50:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:49 smithi154 ceph-mon[102314]: osdmap e219: 8 total, 8 up, 7 in 2024-09-13T14:50:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:49 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 22601/249792 objects degraded (9.048%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:49 smithi154 ceph-mon[107038]: pgmap v630: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 47 op/s; 22601/249792 objects degraded (9.048%); 2868/249792 objects misplaced (1.148%); 8.2 MiB/s, 134 objects/s recovering 2024-09-13T14:50:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:49 smithi154 ceph-mon[107038]: osdmap e219: 8 total, 8 up, 7 in 2024-09-13T14:50:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:49 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 22601/249792 objects degraded (9.048%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:50.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:49 smithi172 ceph-mon[100696]: pgmap v630: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 47 op/s; 22601/249792 objects degraded (9.048%); 2868/249792 objects misplaced (1.148%); 8.2 MiB/s, 134 objects/s recovering 2024-09-13T14:50:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:49 smithi172 ceph-mon[100696]: osdmap e219: 8 total, 8 up, 7 in 2024-09-13T14:50:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:49 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 22601/249792 objects degraded (9.048%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:50.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.355 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.484 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:50.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:50.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:50.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:50.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:50.881 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:50.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:50.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:51.006 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:51.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.131 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:51.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:51.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:51.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:51.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:51.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:51.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:51.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:51.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:51.636 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:51.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:51.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:51.776 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:51.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:51.900 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:51.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:52.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:52.007 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:50:52.026 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:52.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:52.128 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:52.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:52.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:51 smithi154 ceph-mon[102314]: pgmap v632: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 22605/249732 objects degraded (9.052%); 2124/249732 objects misplaced (0.851%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:50:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/153595990' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:50:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:51 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:50:52.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:50:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:50:51] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:50:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:51 smithi154 ceph-mon[107038]: pgmap v632: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 22605/249732 objects degraded (9.052%); 2124/249732 objects misplaced (0.851%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:50:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/153595990' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:50:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:51 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:50:52.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:52.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:52.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:52.403 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:51 smithi172 ceph-mon[100696]: pgmap v632: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 4 op/s; 22605/249732 objects degraded (9.052%); 2124/249732 objects misplaced (0.851%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:50:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/153595990' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:50:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:51 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:50:52.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.530 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:52.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:52.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:52.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:52.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:52.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:52.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:52.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:52.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:53.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:53.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:53.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:53.162 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:53.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:52 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:50:53.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:52 smithi154 ceph-mon[102314]: osdmap e220: 8 total, 8 up, 7 in 2024-09-13T14:50:53.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:53.169 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:52 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:50:53.170 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:52 smithi154 ceph-mon[107038]: osdmap e220: 8 total, 8 up, 7 in 2024-09-13T14:50:53.170 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:53.253 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:52 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:50:53.253 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:52 smithi172 ceph-mon[100696]: osdmap e220: 8 total, 8 up, 7 in 2024-09-13T14:50:53.253 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:50:53.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:53.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:53.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:53.423 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:53.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:53.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:53.654 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.677 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:53.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:53.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:53.802 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:50:53.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:50:53.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:50:53.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:53.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:53.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:54.020 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:50:54.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:54.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:54.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:54.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:54.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[107038]: pgmap v634: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 18 op/s; 22605/249489 objects degraded (9.061%); 2124/249489 objects misplaced (0.851%); 7.7 MiB/s, 133 objects/s recovering 2024-09-13T14:50:54.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3213539259' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/998511592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:54.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[102314]: pgmap v634: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 18 op/s; 22605/249489 objects degraded (9.061%); 2124/249489 objects misplaced (0.851%); 7.7 MiB/s, 133 objects/s recovering 2024-09-13T14:50:54.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3213539259' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/998511592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:54 smithi172 ceph-mon[100696]: pgmap v634: 89 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 18 op/s; 22605/249489 objects degraded (9.061%); 2124/249489 objects misplaced (0.851%); 7.7 MiB/s, 133 objects/s recovering 2024-09-13T14:50:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3213539259' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:54 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:50:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/998511592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:50:54.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:54.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:54.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:54.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:54.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:54.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:54.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.697 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:54.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:54.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:54.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:54.865 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:50:54.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:50:54.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:54.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:54.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:55.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:55.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:55.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:55.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:55.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:55.329 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:55.375 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:55 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:55.375 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:55 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:55.375 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:55 smithi172 ceph-mon[100696]: osdmap e221: 8 total, 8 up, 7 in 2024-09-13T14:50:55.376 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 22605/249489 objects degraded (9.061%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:55.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:55.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:55.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[102314]: osdmap e221: 8 total, 8 up, 7 in 2024-09-13T14:50:55.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 22605/249489 objects degraded (9.061%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:55.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:50:55.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:50:55.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[107038]: osdmap e221: 8 total, 8 up, 7 in 2024-09-13T14:50:55.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 22605/249489 objects degraded (9.061%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:50:55.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.456 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:55.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:55.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:55.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:55.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:55.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:55.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:55.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:55.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:55.965 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:55.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:56.067 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:56.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:56.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:56 smithi154 ceph-mon[102314]: pgmap v636: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 144 op/s; 22665/247188 objects degraded (9.169%); 1995/247188 objects misplaced (0.807%); 8.4 MiB/s, 149 objects/s recovering 2024-09-13T14:50:56.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:56 smithi154 ceph-mon[107038]: pgmap v636: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 144 op/s; 22665/247188 objects degraded (9.169%); 1995/247188 objects misplaced (0.807%); 8.4 MiB/s, 149 objects/s recovering 2024-09-13T14:50:56.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:56.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:56.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:56.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:56.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:56 smithi172 ceph-mon[100696]: pgmap v636: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 144 op/s; 22665/247188 objects degraded (9.169%); 1995/247188 objects misplaced (0.807%); 8.4 MiB/s, 149 objects/s recovering 2024-09-13T14:50:56.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:56.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:56.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:56.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:56.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:56.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:56.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:56.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:56.979 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23631 6.b 6:def0569e:::benchmark_data_smithi154_175640_object23630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.797951+0000", 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.17620122299999, 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96814932600000003, 2024-09-13T14:50:56.980 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23631 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.797951+0000", 2024-09-13T14:50:56.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.797951+0000", 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.797952+0000", 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0999999999999997e-07 2024-09-13T14:50:56.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.797975+0000", 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2486000000000001e-05 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.797984+0000", 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9379999999999998e-06 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.797990+0000", 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4710000000000004e-06 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.798044+0000", 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4106e-05 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.798052+0000", 2024-09-13T14:50:56.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2689999999999999e-06 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.763915+0000", 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96586282199999995 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.764031+0000", 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115438 2024-09-13T14:50:56.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.764136+0000", 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105791 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.765994+0000", 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018578620000000001 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:56.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766004+0000", 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7569999999999999e-06 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766031+0000", 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6537e-05 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766062+0000", 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1032999999999999e-05 2024-09-13T14:50:56.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766101+0000", 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8906000000000003e-05 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23632 6.b 6:d50c4219:::benchmark_data_smithi154_175640_object23631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.799223+0000", 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.17492975499999, 2024-09-13T14:50:56.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96877568199999997, 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23632 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799223+0000", 2024-09-13T14:50:56.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799223+0000", 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799224+0000", 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.114e-06 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:56.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799245+0000", 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1645e-05 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799257+0000", 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1459e-05 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799264+0000", 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6370000000000002e-06 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799387+0000", 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012366300000000001 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799393+0000", 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0700000000000003e-06 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.765346+0000", 2024-09-13T14:50:56.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96595293400000004 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.765425+0000", 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8867000000000002e-05 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.765506+0000", 2024-09-13T14:50:56.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1290000000000003e-05 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.767853+0000", 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00234621 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.767880+0000", 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7063000000000002e-05 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:56.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.767914+0000", 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4128999999999998e-05 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.767965+0000", 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1511999999999997e-05 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.767998+0000", 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3089000000000001e-05 2024-09-13T14:50:56.995 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23633 6.b 6:df253204:::benchmark_data_smithi154_175640_object23632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.799283+0000", 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.17486927300001, 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96900027499999997, 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:56.996 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23633 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799283+0000", 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799283+0000", 2024-09-13T14:50:56.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799284+0000", 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3300000000000002e-07 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799301+0000", 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6983999999999999e-05 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799308+0000", 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.844e-06 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799312+0000", 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9060000000000001e-06 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799406+0000", 2024-09-13T14:50:56.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3632000000000001e-05 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.799410+0000", 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4450000000000001e-06 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766132+0000", 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96672286299999999 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766204+0000", 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1118000000000006e-05 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766321+0000", 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117166 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768007+0000", 2024-09-13T14:50:57.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016864250000000001 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768164+0000", 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156551 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768199+0000", 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5756000000000001e-05 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768240+0000", 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0292e-05 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768283+0000", 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3760000000000001e-05 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23651 6.b 6:dc027494:::benchmark_data_smithi154_175640_object23650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.003 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.880354+0000", 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.09379809500001, 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89002726799999998, 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23651 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.004 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880354+0000", 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880354+0000", 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880355+0000", 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.99e-07 2024-09-13T14:50:57.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880403+0000", 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7716999999999997e-05 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880412+0000", 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5710000000000004e-06 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880419+0000", 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7839999999999998e-06 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880516+0000", 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6744999999999996e-05 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.880524+0000", 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0299999999999994e-06 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.766938+0000", 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.88641392299999999 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.767055+0000", 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116865 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.767180+0000", 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012535899999999999 2024-09-13T14:50:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768172+0000", 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00099236500000000004 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770234+0000", 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020621670000000002 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770282+0000", 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.723e-05 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770335+0000", 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3706999999999999e-05 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770382+0000", 2024-09-13T14:50:57.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6205999999999998e-05 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23660 6.b 6:dbb18b7c:::benchmark_data_smithi154_175640_object23659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.883872+0000", 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.090280734, 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89452775600000001, 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.012 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23660 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.883872+0000", 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.883872+0000", 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.883873+0000", 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6759999999999999e-06 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.883929+0000", 2024-09-13T14:50:57.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5844000000000003e-05 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.883942+0000", 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.269e-05 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.883950+0000", 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6939999999999995e-06 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.884230+0000", 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000280745 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.884241+0000", 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0217999999999999e-05 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768317+0000", 2024-09-13T14:50:57.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.88407668699999997 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768424+0000", 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107064 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.768562+0000", 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013718700000000001 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770434+0000", 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018729269999999999 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.772838+0000", 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024035670000000001 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778295+0000", 2024-09-13T14:50:57.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054568389999999998 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778344+0000", 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8687999999999999e-05 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778399+0000", 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5930000000000002e-05 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.019 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23678 6.b 6:d8301805:::benchmark_data_smithi154_175640_object23677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.894626+0000", 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.079526276, 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.88392171500000005, 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.020 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23678 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.894626+0000", 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.894626+0000", 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.894628+0000", 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5629999999999999e-06 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.894656+0000", 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8371e-05 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.894668+0000", 2024-09-13T14:50:57.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.894675+0000", 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9600000000000003e-06 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.895455+0000", 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078075699999999998 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.895466+0000", 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0815999999999999e-05 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.769326+0000", 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.87385975699999996 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.769450+0000", 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012350100000000001 2024-09-13T14:50:57.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.769589+0000", 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013936499999999999 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.772881+0000", 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032920699999999998 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.772951+0000", 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0396999999999997e-05 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778410+0000", 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054584799999999999 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778511+0000", 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101599 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778548+0000", 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6417e-05 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.027 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23683 6.b 6:dab416aa:::benchmark_data_smithi154_175640_object23682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.899743+0000", 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.07440916600001, 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.87891958000000003, 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23683 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.028 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899743+0000", 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899743+0000", 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899745+0000", 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.756e-06 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899782+0000", 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6956000000000003e-05 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899797+0000", 2024-09-13T14:50:57.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5415000000000001e-05 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899807+0000", 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3810000000000005e-06 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899953+0000", 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014637299999999999 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.899962+0000", 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4610000000000008e-06 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770470+0000", 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.87050807100000005 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770602+0000", 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000132206 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.770772+0000", 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000170154 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.772919+0000", 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021464520000000001 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.772986+0000", 2024-09-13T14:50:57.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7080000000000001e-05 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778557+0000", 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055715779999999998 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778631+0000", 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3977e-05 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778663+0000", 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1720000000000001e-05 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23707 6.b 6:daf4f907:::benchmark_data_smithi154_175640_object23706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.915258+0000", 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.05889459400001, 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86349428699999997, 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.035 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23707 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915258+0000", 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915258+0000", 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915259+0000", 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.35e-06 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915281+0000", 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.156e-05 2024-09-13T14:50:57.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915292+0000", 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.152e-05 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915299+0000", 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.866e-06 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915368+0000", 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9220000000000005e-05 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.915378+0000", 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3589999999999997e-06 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.771798+0000", 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.85642004599999999 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.771906+0000", 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108503 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.772032+0000", 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012563599999999999 2024-09-13T14:50:57.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.041 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:50:57.041 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:57.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:57.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.775216+0000", 2024-09-13T14:50:57.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031838510000000001 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.775271+0000", 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5356999999999999e-05 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778670+0000", 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00339902 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.043 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778716+0000", 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5849000000000001e-05 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778752+0000", 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6149999999999998e-05 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23733 6.b 6:ddb7636f:::benchmark_data_smithi154_175640_object23732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.931846+0000", 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.04230657700001, 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.84702469000000002, 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23733 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.045 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931846+0000", 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931846+0000", 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931847+0000", 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6799999999999999e-07 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931884+0000", 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7036000000000003e-05 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931894+0000", 2024-09-13T14:50:57.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0817e-05 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931901+0000", 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6549999999999998e-06 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931966+0000", 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4652000000000004e-05 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.931974+0000", 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5599999999999994e-06 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.773014+0000", 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.841039286 2024-09-13T14:50:57.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.773134+0000", 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011992700000000001 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.773270+0000", 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013687500000000001 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.775311+0000", 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020404109999999998 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.775348+0000", 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7190999999999997e-05 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778791+0000", 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003442578 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778830+0000", 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9515999999999999e-05 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778871+0000", 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0417999999999998e-05 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23741 6.b 6:d9015a09:::benchmark_data_smithi154_175640_object23740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.936255+0000", 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.03789748400001, 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.84268980199999999, 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23741 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.053 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936255+0000", 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936255+0000", 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936256+0000", 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.186e-06 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936275+0000", 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9241000000000001e-05 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936284+0000", 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9949999999999994e-06 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936290+0000", 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6359999999999999e-06 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936353+0000", 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3064000000000005e-05 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.936363+0000", 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9839999999999996e-06 2024-09-13T14:50:57.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.774207+0000", 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83784362700000004 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.774355+0000", 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000148484 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.774483+0000", 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012801199999999999 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778180+0000", 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036971959999999998 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778282+0000", 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010173100000000001 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778879+0000", 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00059668899999999997 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778916+0000", 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7007e-05 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778945+0000", 2024-09-13T14:50:57.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8949999999999999e-05 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23758 6.b 6:d8b92afe:::benchmark_data_smithi154_175640_object23757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.947212+0000", 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.026940537, 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.835686237, 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.060 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23758 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947212+0000", 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947212+0000", 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947214+0000", 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.669e-06 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947261+0000", 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7718e-05 2024-09-13T14:50:57.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947275+0000", 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4161999999999999e-05 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947283+0000", 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3259999999999998e-06 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947362+0000", 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9229999999999999e-05 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.947370+0000", 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0499999999999992e-06 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.775379+0000", 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.82800843899999998 2024-09-13T14:50:57.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.775478+0000", 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9763e-05 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.775626+0000", 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014762299999999999 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.778220+0000", 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002593836 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.781429+0000", 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00320908 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.782568+0000", 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011395699999999999 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.782848+0000", 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027944800000000001 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.782898+0000", 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0322999999999999e-05 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23808 6.b 6:d5c9888d:::benchmark_data_smithi154_175640_object23807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.978762+0000", 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 166.99539057699999, 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.80426537499999995, 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23808 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978762+0000", 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978762+0000", 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978763+0000", 2024-09-13T14:50:57.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9800000000000002e-07 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978782+0000", 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8989999999999999e-05 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978794+0000", 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1992e-05 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978801+0000", 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4869999999999996e-06 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978878+0000", 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6594000000000006e-05 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.978891+0000", 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2884000000000001e-05 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.777095+0000", 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79820398599999998 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.777238+0000", 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014301099999999999 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.777375+0000", 2024-09-13T14:50:57.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013675699999999999 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.781434+0000", 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040596870000000002 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.781617+0000", 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000182895 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.782946+0000", 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001328463 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.782994+0000", 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8084999999999998e-05 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.783027+0000", 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3546e-05 2024-09-13T14:50:57.074 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:23813 6.b 6:dabfeb2c:::benchmark_data_smithi154_175640_object23812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:48:09.982897+0000", 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 166.99125535499999, 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.80685278900000001, 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.075 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 23813 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982897+0000", 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982897+0000", 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982898+0000", 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9099999999999991e-07 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982920+0000", 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.207e-05 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982931+0000", 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0606e-05 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982936+0000", 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3720000000000004e-06 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982985+0000", 2024-09-13T14:50:57.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8718999999999999e-05 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:09.982991+0000", 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2230000000000004e-06 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.779005+0000", 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79601351099999995 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.780200+0000", 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011952969999999999 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.780323+0000", 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012353200000000001 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.783037+0000", 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027137300000000001 2024-09-13T14:50:57.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.783090+0000", 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3318e-05 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.789654+0000", 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065633260000000004 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.789707+0000", 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3164000000000001e-05 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:48:10.789750+0000", 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2929999999999997e-05 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86887 6.c 6:302ee6f9:::benchmark_data_smithi154_175640_object1111:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T14:50:57.082 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 34.184899876000003, 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8528283699999992, 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86887 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789254+0000", 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6649999999999999e-06 2024-09-13T14:50:57.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789262+0000", 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.542e-06 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789278+0000", 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6351e-05 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789288+0000", 2024-09-13T14:50:57.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0358e-05 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789368+0000", 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9726000000000002e-05 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789377+0000", 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1609999999999992e-06 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.796583+0000", 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072051260000000001 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.796594+0000", 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1331e-05 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.513336+0000", 2024-09-13T14:50:57.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7167424520000001 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.519370+0000", 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060334999999999998 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.527233+0000", 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078633379999999992 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.538182+0000", 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010949257 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.547496+0000", 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093131220000000001 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.631104+0000", 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.083608314000000003 2024-09-13T14:50:57.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.633486+0000", 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002382463 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.642081+0000", 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085946640000000001 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:89265 6.c 6:3d31a432:::benchmark_data_smithi154_175640_object3489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:54.139163+0000", 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.8349890609999999, 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79334962399999998, 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 89265 2024-09-13T14:50:57.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.139163+0000", 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.139163+0000", 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.139165+0000", 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.26e-06 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.139170+0000", 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4419999999999997e-06 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.139182+0000", 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1751e-05 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.139188+0000", 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1489999999999999e-06 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.835250+0000", 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.69606155400000003 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.849305+0000", 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014055635 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.867320+0000", 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018014511 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.903137+0000", 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035817590000000003 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.905768+0000", 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026309150000000002 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.907312+0000", 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015441680000000001 2024-09-13T14:50:57.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.914307+0000", 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006994353 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.932513+0000", 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018206296 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:89294 6.c 6:3b102ae3:::benchmark_data_smithi154_175640_object3518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:54.289737+0000", 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.684415306, 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79153275099999998, 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 89294 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.098 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289737+0000", 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289737+0000", 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289739+0000", 2024-09-13T14:50:57.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.911e-06 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289747+0000", 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2470000000000008e-06 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289764+0000", 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6674999999999999e-05 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289775+0000", 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1167e-05 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.989759+0000", 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.69998365900000004 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.999680+0000", 2024-09-13T14:50:57.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099207059999999996 2024-09-13T14:50:57.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:50:57.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.014603+0000", 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014923815 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.052457+0000", 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037853914000000002 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.055365+0000", 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029081580000000001 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.060237+0000", 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048719899999999997 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.063926+0000", 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003688317 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.081270+0000", 2024-09-13T14:50:57.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017344192000000001 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:89295 6.c 6:34f37022:::benchmark_data_smithi154_175640_object3519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:54.289796+0000", 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.6843564280000001, 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86391756399999997, 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.105 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 89295 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289796+0000", 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289796+0000", 2024-09-13T14:50:57.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289797+0000", 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5199999999999997e-07 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289800+0000", 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2160000000000002e-06 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289811+0000", 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0953e-05 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.289817+0000", 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9309999999999996e-06 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.084187+0000", 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79436991899999998 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.093173+0000", 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089862399999999995 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.100666+0000", 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074929640000000004 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.132997+0000", 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032331533000000003 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.135222+0000", 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022251459999999999 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.136468+0000", 2024-09-13T14:50:57.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012450989999999999 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.139739+0000", 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003271585 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.153714+0000", 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013974426 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:89348 6.c 6:3eebc703:::benchmark_data_smithi154_175640_object3572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:54.515867+0000", 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.4582855559999999, 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.82757735899999996, 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.112 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 89348 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.515867+0000", 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.515867+0000", 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.515868+0000", 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8299999999999995e-07 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.515872+0000", 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0369999999999996e-06 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.515881+0000", 2024-09-13T14:50:57.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3479999999999993e-06 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.515888+0000", 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7950000000000002e-06 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.195073+0000", 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.67918446099999996 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.203343+0000", 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082706159999999997 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.217074+0000", 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013730398 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.255445+0000", 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038371608000000001 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.257684+0000", 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022387940000000001 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.326887+0000", 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069203309000000005 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.329885+0000", 2024-09-13T14:50:57.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002997946 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.343444+0000", 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013559063999999999 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:89422 6.c 6:31ca6280:::benchmark_data_smithi154_175640_object3646:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:54.806636+0000", 2024-09-13T14:50:57.118 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.1675161169999999, 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.81191542699999997, 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 89422 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.806636+0000", 2024-09-13T14:50:57.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.806636+0000", 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.806638+0000", 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.547e-06 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.806643+0000", 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.741e-06 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.806653+0000", 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0817e-05 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.806661+0000", 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1450000000000002e-06 2024-09-13T14:50:57.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.533276+0000", 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.72661526200000004 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.538627+0000", 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053513839999999998 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:50:57.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.548139+0000", 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095116009999999997 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.592465+0000", 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.044326363000000001 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.594499+0000", 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020339870000000001 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.600478+0000", 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059788660000000002 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.603691+0000", 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032134329999999999 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.618552+0000", 2024-09-13T14:50:57.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014860280999999999 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:89453 6.c 6:33dfe3d9:::benchmark_data_smithi154_175640_object3677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:54.919122+0000", 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.0550307459999999, 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79354211100000005, 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:50:57.125 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 89453 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.919122+0000", 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.919122+0000", 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.919123+0000", 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3099999999999996e-07 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:50:57.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.919127+0000", 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2089999999999999e-06 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.919138+0000", 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1593000000000001e-05 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:54.919147+0000", 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1079999999999993e-06 2024-09-13T14:50:57.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.638707+0000", 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.71956007799999999 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.646627+0000", 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079206729999999996 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:50:57.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.654084+0000", 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007456575 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.689703+0000", 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035619149000000003 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.691896+0000", 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021929660000000002 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.695641+0000", 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037452119999999999 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.698811+0000", 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031697499999999998 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:55.712664+0000", 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013852867 2024-09-13T14:50:57.131 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.132 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:50:57.132 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:50:57.132 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:50:57.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:57.176 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:57.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:50:57.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:50:57.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:50:57.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:50:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:57.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:50:57.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:57.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:57.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:57.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:57.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:57.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:57.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:57.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:57.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:57.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:57.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:58.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:58.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:58.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:58.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:58.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:58.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:58.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:50:58.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:58.453 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:58.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:58.576 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:50:58.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:50:58.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:50:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:58 smithi154 ceph-mon[102314]: pgmap v637: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 108 op/s; 22665/247188 objects degraded (9.169%); 1995/247188 objects misplaced (0.807%); 6.3 MiB/s, 112 objects/s recovering 2024-09-13T14:50:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:58.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:50:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:58.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:58 smithi154 ceph-mon[107038]: pgmap v637: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 108 op/s; 22665/247188 objects degraded (9.169%); 1995/247188 objects misplaced (0.807%); 6.3 MiB/s, 112 objects/s recovering 2024-09-13T14:50:58.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:58.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:50:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:58.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:58.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:58.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:58.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:58.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:58 smithi172 ceph-mon[100696]: pgmap v637: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 108 op/s; 22665/247188 objects degraded (9.169%); 1995/247188 objects misplaced (0.807%); 6.3 MiB/s, 112 objects/s recovering 2024-09-13T14:50:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:50:58.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:50:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:50:59.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:59.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:59.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:50:59.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:50:59.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:59.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:59.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:50:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:59.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:50:59.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:59.613 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:59.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:50:59.746 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:50:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:50:59.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:50:59.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:50:59.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:50:59.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:50:59.996 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:50:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:50:59.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:00.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:00.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:00.149 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:00.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:00.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:00.267 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:00.268 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:00.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:00.388 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:00.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:00.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:00.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:00.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:00.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:00.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:00.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:00.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:00.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:00.926 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:00 smithi154 ceph-mon[102314]: pgmap v638: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 22665/245457 objects degraded (9.234%); 1995/245457 objects misplaced (0.813%); 536 KiB/s, 11 objects/s recovering 2024-09-13T14:51:00.926 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 22665/245457 objects degraded (9.234%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:51:00.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:00 smithi154 ceph-mon[107038]: pgmap v638: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 22665/245457 objects degraded (9.234%); 1995/245457 objects misplaced (0.813%); 536 KiB/s, 11 objects/s recovering 2024-09-13T14:51:00.927 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 22665/245457 objects degraded (9.234%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:51:00.981 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:00 smithi172 ceph-mon[100696]: pgmap v638: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 22665/245457 objects degraded (9.234%); 1995/245457 objects misplaced (0.813%); 536 KiB/s, 11 objects/s recovering 2024-09-13T14:51:00.982 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 22665/245457 objects degraded (9.234%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:51:01.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:01.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:01.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:01.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:01.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:01.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:01.325 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:51:01.333 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:51:01.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:01.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:01.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:01.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:01.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:01.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:01.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:01.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.798 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:01.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:01.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:01.925 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:01.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:02.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:02.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:02.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:02.181 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:51:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:51:01] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:51:02.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:02.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:02.353 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:51:02.353 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:51:02.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.356 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:91462 6.f 6:fd5aa33a:::benchmark_data_smithi154_175640_object5686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-09-13T14:51:02.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:51:02.103124+0000", 2024-09-13T14:51:02.356 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.003194957, 2024-09-13T14:51:02.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003215018, 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 91462 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103124+0000", 2024-09-13T14:51:02.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103124+0000", 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103123+0000", 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:02.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103129+0000", 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.207e-06 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103148+0000", 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8678999999999999e-05 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103157+0000", 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5059999999999995e-06 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103201+0000", 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4316000000000001e-05 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103318+0000", 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011723200000000001 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.103457+0000", 2024-09-13T14:51:02.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013916500000000001 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:51:02.106063+0000", 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026056410000000001 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-13T14:51:02.361 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:51:02.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:02.428 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:02.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:02.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.561 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:02.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:02.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:02.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:02 smithi154 ceph-mon[102314]: pgmap v639: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 22665/243237 objects degraded (9.318%); 1445/243237 objects misplaced (0.594%); 4.3 MiB/s, 72 objects/s recovering 2024-09-13T14:51:02.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:02 smithi154 ceph-mon[102314]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:02 smithi154 ceph-mon[107038]: pgmap v639: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 22665/243237 objects degraded (9.318%); 1445/243237 objects misplaced (0.594%); 4.3 MiB/s, 72 objects/s recovering 2024-09-13T14:51:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:02 smithi154 ceph-mon[107038]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:02.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:02.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:02.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:02.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:02.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:02.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:02 smithi172 ceph-mon[100696]: pgmap v639: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 22665/243237 objects degraded (9.318%); 1445/243237 objects misplaced (0.594%); 4.3 MiB/s, 72 objects/s recovering 2024-09-13T14:51:02.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:02 smithi172 ceph-mon[100696]: from='client.54832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:02.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:51:03.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:03.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:03.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:03.191 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:03.212 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:03.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:03.304 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-13T14:51:00.960694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5316483069,"num_objects":81079,"num_object_clones":0,"num_object_copies":243237,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":22665,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":81079,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":91620,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23260,"num_bytes_recovered":1505447045,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90651,"ondisk_log_size":90651,"up":267,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":301,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18128756,"kb_used_data":16422076,"kb_used_omap":32,"kb_used_meta":1706591,"kb_avail":638087308,"statfs":{"total":671965249536,"available":653401403392,"internally_reserved":0,"allocated":16816205824,"data_stored":16807647618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1747549744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,3,1,3,2,0,2,3,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-141885440,"num_objects":-2165,"num_object_clones":0,"num_object_copies":-6495,"num_objects_missing_on_primary":-25,"num_objects_missing":-25,"num_objects_degraded":60,"num_objects_misplaced":-679,"num_objects_unfound":0,"num_objects_dirty":-2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2165,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":646,"num_bytes_recovered":40435712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.962058"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":322,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.083917+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:50:54.083917+0000","last_peered":"2024-09-13T14:50:54.083917+0000","last_clean":"2024-09-13T14:50:54.083917+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:50:54.083917+0000","last_undegraded":"2024-09-13T14:50:54.083917+0000","last_fullsized":"2024-09-13T14:50:54.083917+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":86,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008642+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:50:52.008642+0000","last_peered":"2024-09-13T14:50:52.008642+0000","last_clean":"2024-09-13T14:50:52.008642+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:50:52.008642+0000","last_undegraded":"2024-09-13T14:50:52.008642+0000","last_fullsized":"2024-09-13T14:50:52.008642+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":56,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008439+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:50:52.008439+0000","last_peered":"2024-09-13T14:50:52.008439+0000","last_clean":"2024-09-13T14:50:52.008439+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:50:52.008439+0000","last_undegraded":"2024-09-13T14:50:52.008439+0000","last_fullsized":"2024-09-13T14:50:52.008439+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":312,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033685+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:50:54.033685+0000","last_peered":"2024-09-13T14:50:54.033685+0000","last_clean":"2024-09-13T14:50:54.033685+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:50:54.033685+0000","last_undegraded":"2024-09-13T14:50:54.033685+0000","last_fullsized":"2024-09-13T14:50:54.033685+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":312,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033627+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:50:54.033627+0000","last_peered":"2024-09-13T14:50:54.033627+0000","last_clean":"2024-09-13T14:50:54.033627+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:50:54.033627+0000","last_undegraded":"2024-09-13T14:50:54.033627+0000","last_fullsized":"2024-09-13T14:50:54.033627+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":156,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008118+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:50:52.008118+0000","last_peered":"2024-09-13T14:50:52.008118+0000","last_clean":"2024-09-13T14:50:52.008118+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:50:52.008118+0000","last_undegraded":"2024-09-13T14:50:52.008118+0000","last_fullsized":"2024-09-13T14:50:52.008118+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":38,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.014549+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:50:54.014549+0000","last_peered":"2024-09-13T14:50:54.014549+0000","last_clean":"2024-09-13T14:50:54.014549+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:50:54.014549+0000","last_undegraded":"2024-09-13T14:50:54.014549+0000","last_fullsized":"2024-09-13T14:50:54.014549+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":132,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.011237+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:50:52.011237+0000","last_peered":"2024-09-13T14:50:52.011237+0000","last_clean":"2024-09-13T14:50:52.011237+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:50:52.011237+0000","last_undegraded":"2024-09-13T14:50:52.011237+0000","last_fullsized":"2024-09-13T14:50:52.011237+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"221'5770","reported_seq":6584,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.551971+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:50:59.551971+0000","last_peered":"2024-09-13T14:50:59.551971+0000","last_clean":"2024-09-13T14:50:59.551971+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:50:59.551971+0000","last_undegraded":"2024-09-13T14:50:59.551971+0000","last_fullsized":"2024-09-13T14:50:59.551971+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335937536,"num_objects":5126,"num_object_clones":0,"num_object_copies":15378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5770,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":247,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.038239+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:50:54.038239+0000","last_peered":"2024-09-13T14:50:54.038239+0000","last_clean":"2024-09-13T14:50:54.038239+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:50:54.038239+0000","last_undegraded":"2024-09-13T14:50:54.038239+0000","last_fullsized":"2024-09-13T14:50:54.038239+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":83,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.007928+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:50:52.007928+0000","last_peered":"2024-09-13T14:50:52.007928+0000","last_clean":"2024-09-13T14:50:52.007928+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:50:52.007928+0000","last_undegraded":"2024-09-13T14:50:52.007928+0000","last_fullsized":"2024-09-13T14:50:52.007928+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"221'5566","reported_seq":7136,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:57.835680+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:50:57.835680+0000","last_peered":"2024-09-13T14:50:57.835680+0000","last_clean":"2024-09-13T14:50:57.835680+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:50:57.835680+0000","last_undegraded":"2024-09-13T14:50:57.835680+0000","last_fullsized":"2024-09-13T14:50:57.835680+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5566,"log_dups_size":0,"ondisk_log_size":5566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328466432,"num_objects":5012,"num_object_clones":0,"num_object_copies":15036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5566,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":84,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.009509+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:50:52.009509+0000","last_peered":"2024-09-13T14:50:52.009509+0000","last_clean":"2024-09-13T14:50:52.009509+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:50:52.009509+0000","last_undegraded":"2024-09-13T14:50:52.009509+0000","last_fullsized":"2024-09-13T14:50:52.009509+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":236,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.009133+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:50:52.009133+0000","last_peered":"2024-09-13T14:50:52.009133+0000","last_clean":"2024-09-13T14:50:52.009133+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:50:52.009133+0000","last_undegraded":"2024-09-13T14:50:52.009133+0000","last_fullsized":"2024-09-13T14:50:52.009133+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"221'5723","reported_seq":7386,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.213540+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:50:59.213540+0000","last_peered":"2024-09-13T14:50:59.213540+0000","last_clean":"2024-09-13T14:50:59.213540+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:50:59.213540+0000","last_undegraded":"2024-09-13T14:50:59.213540+0000","last_fullsized":"2024-09-13T14:50:59.213540+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5723,"log_dups_size":0,"ondisk_log_size":5723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335609856,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5723,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":60,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008630+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:50:52.008630+0000","last_peered":"2024-09-13T14:50:52.008630+0000","last_clean":"2024-09-13T14:50:52.008630+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:50:52.008630+0000","last_undegraded":"2024-09-13T14:50:52.008630+0000","last_fullsized":"2024-09-13T14:50:52.008630+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.011175+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:50:52.011175+0000","last_peered":"2024-09-13T14:50:52.011175+0000","last_clean":"2024-09-13T14:50:52.011175+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:50:52.011175+0000","last_undegraded":"2024-09-13T14:50:52.011175+0000","last_fullsized":"2024-09-13T14:50:52.011175+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"221'5533","reported_seq":7256,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:57.877768+0000","last_change":"2024-09-13T14:50:07.839326+0000","last_active":"2024-09-13T14:50:57.877768+0000","last_peered":"2024-09-13T14:50:57.877768+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:50:57.877768+0000","last_undegraded":"2024-09-13T14:50:07.570096+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5533,"log_dups_size":0,"ondisk_log_size":5533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326041600,"num_objects":4975,"num_object_clones":0,"num_object_copies":14925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5533,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":51445760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":2874},{"shards":"0,3,6","objects":2101}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":72,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008352+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:50:52.008352+0000","last_peered":"2024-09-13T14:50:52.008352+0000","last_clean":"2024-09-13T14:50:52.008352+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:50:52.008352+0000","last_undegraded":"2024-09-13T14:50:52.008352+0000","last_fullsized":"2024-09-13T14:50:52.008352+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":291,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033190+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:50:54.033190+0000","last_peered":"2024-09-13T14:50:54.033190+0000","last_clean":"2024-09-13T14:50:54.033190+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:50:54.033190+0000","last_undegraded":"2024-09-13T14:50:54.033190+0000","last_fullsized":"2024-09-13T14:50:54.033190+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"221'5651","reported_seq":12118,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:58.999189+0000","last_change":"2024-09-13T14:50:08.716863+0000","last_active":"2024-09-13T14:50:58.999189+0000","last_peered":"2024-09-13T14:50:58.999189+0000","last_clean":"2024-09-13T14:49:59.725669+0000","last_became_active":"2024-09-13T14:50:08.707516+0000","last_became_peered":"2024-09-13T14:50:08.707516+0000","last_unstale":"2024-09-13T14:50:58.999189+0000","last_undegraded":"2024-09-13T14:50:08.570515+0000","last_fullsized":"2024-09-13T14:50:08.544112+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:20.218839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334430208,"num_objects":5103,"num_object_clones":0,"num_object_copies":15309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3185,"num_bytes_recovered":208732160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-219},{"shards":"1,4,5","objects":5103}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":300,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013600+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:50:54.013600+0000","last_peered":"2024-09-13T14:50:54.013600+0000","last_clean":"2024-09-13T14:50:54.013600+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:50:54.013600+0000","last_undegraded":"2024-09-13T14:50:54.013600+0000","last_fullsized":"2024-09-13T14:50:54.013600+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":203,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008836+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:50:52.008836+0000","last_peered":"2024-09-13T14:50:52.008836+0000","last_clean":"2024-09-13T14:50:52.008836+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:50:52.008836+0000","last_undegraded":"2024-09-13T14:50:52.008836+0000","last_fullsized":"2024-09-13T14:50:52.008836+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"221'5628","reported_seq":5992,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:59.050143+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:50:59.050143+0000","last_peered":"2024-09-13T14:50:59.050143+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:50:59.050143+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328466432,"num_objects":5012,"num_object_clones":0,"num_object_copies":15036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":2832},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":103,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.011360+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:50:52.011360+0000","last_peered":"2024-09-13T14:50:52.011360+0000","last_clean":"2024-09-13T14:50:52.011360+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:50:52.011360+0000","last_undegraded":"2024-09-13T14:50:52.011360+0000","last_fullsized":"2024-09-13T14:50:52.011360+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":200,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008921+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:50:52.008921+0000","last_peered":"2024-09-13T14:50:52.008921+0000","last_clean":"2024-09-13T14:50:52.008921+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:50:52.008921+0000","last_undegraded":"2024-09-13T14:50:52.008921+0000","last_fullsized":"2024-09-13T14:50:52.008921+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"221'5735","reported_seq":7845,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:58.219958+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:50:58.219958+0000","last_peered":"2024-09-13T14:50:58.219958+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:50:58.219958+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333381632,"num_objects":5087,"num_object_clones":0,"num_object_copies":15261,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":3649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5735,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73203712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":1248},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":1693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":195,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.009081+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:50:52.009081+0000","last_peered":"2024-09-13T14:50:52.009081+0000","last_clean":"2024-09-13T14:50:52.009081+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:50:52.009081+0000","last_undegraded":"2024-09-13T14:50:52.009081+0000","last_fullsized":"2024-09-13T14:50:52.009081+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":230,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.009578+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:50:52.009578+0000","last_peered":"2024-09-13T14:50:52.009578+0000","last_clean":"2024-09-13T14:50:52.009578+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:50:52.009578+0000","last_undegraded":"2024-09-13T14:50:52.009578+0000","last_fullsized":"2024-09-13T14:50:52.009578+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"221'5552","reported_seq":9959,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.564688+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:50:59.564688+0000","last_peered":"2024-09-13T14:50:59.564688+0000","last_clean":"2024-09-13T14:50:59.564688+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:50:59.564688+0000","last_undegraded":"2024-09-13T14:50:59.564688+0000","last_fullsized":"2024-09-13T14:50:59.564688+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5552,"log_dups_size":0,"ondisk_log_size":5552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324599832,"num_objects":4954,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5552,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013513+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:50:54.013513+0000","last_peered":"2024-09-13T14:50:54.013513+0000","last_clean":"2024-09-13T14:50:54.013513+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:50:54.013513+0000","last_undegraded":"2024-09-13T14:50:54.013513+0000","last_fullsized":"2024-09-13T14:50:54.013513+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":147,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.038335+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:50:54.038335+0000","last_peered":"2024-09-13T14:50:54.038335+0000","last_clean":"2024-09-13T14:50:54.038335+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:50:54.038335+0000","last_undegraded":"2024-09-13T14:50:54.038335+0000","last_fullsized":"2024-09-13T14:50:54.038335+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"221'5640","reported_seq":6869,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.528764+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:50:59.528764+0000","last_peered":"2024-09-13T14:50:59.528764+0000","last_clean":"2024-09-13T14:50:59.528764+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:50:59.528764+0000","last_undegraded":"2024-09-13T14:50:59.528764+0000","last_fullsized":"2024-09-13T14:50:59.528764+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334495744,"num_objects":5104,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5640,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":494,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.035705+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:50:54.035705+0000","last_peered":"2024-09-13T14:50:54.035705+0000","last_clean":"2024-09-13T14:50:54.035705+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:50:54.035705+0000","last_undegraded":"2024-09-13T14:50:54.035705+0000","last_fullsized":"2024-09-13T14:50:54.035705+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":787,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024279+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:50:54.024279+0000","last_peered":"2024-09-13T14:50:54.024279+0000","last_clean":"2024-09-13T14:50:54.024279+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:50:54.024279+0000","last_undegraded":"2024-09-13T14:50:54.024279+0000","last_fullsized":"2024-09-13T14:50:54.024279+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":258,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.014434+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:50:54.014434+0000","last_peered":"2024-09-13T14:50:54.014434+0000","last_clean":"2024-09-13T14:50:54.014434+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:50:54.014434+0000","last_undegraded":"2024-09-13T14:50:54.014434+0000","last_fullsized":"2024-09-13T14:50:54.014434+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.011628+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:50:52.011628+0000","last_peered":"2024-09-13T14:50:52.011628+0000","last_clean":"2024-09-13T14:50:52.011628+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:50:52.011628+0000","last_undegraded":"2024-09-13T14:50:52.011628+0000","last_fullsized":"2024-09-13T14:50:52.011628+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"221'5741","reported_seq":7217,"reported_epoch":221,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:59.688470+0000","last_change":"2024-09-13T14:50:32.902196+0000","last_active":"2024-09-13T14:50:59.688470+0000","last_peered":"2024-09-13T14:50:59.688470+0000","last_clean":"2024-09-13T14:48:19.187296+0000","last_became_active":"2024-09-13T14:50:32.492668+0000","last_became_peered":"2024-09-13T14:50:32.492668+0000","last_unstale":"2024-09-13T14:50:59.688470+0000","last_undegraded":"2024-09-13T14:50:32.850704+0000","last_fullsized":"2024-09-13T14:50:23.793735+0000","mapping_epoch":211,"log_start":"164'418","ondisk_log_start":"164'418","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334692352,"num_objects":5107,"num_object_clones":0,"num_object_copies":15321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":155,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":738,"num_bytes_recovered":44630016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":3507},{"shards":"1,3,6","objects":1445}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":339,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.012869+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:50:54.012869+0000","last_peered":"2024-09-13T14:50:54.012869+0000","last_clean":"2024-09-13T14:50:54.012869+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:50:54.012869+0000","last_undegraded":"2024-09-13T14:50:54.012869+0000","last_fullsized":"2024-09-13T14:50:54.012869+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":306,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033669+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:50:54.033669+0000","last_peered":"2024-09-13T14:50:54.033669+0000","last_clean":"2024-09-13T14:50:54.033669+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:50:54.033669+0000","last_undegraded":"2024-09-13T14:50:54.033669+0000","last_fullsized":"2024-09-13T14:50:54.033669+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":317,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033851+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:50:54.033851+0000","last_peered":"2024-09-13T14:50:54.033851+0000","last_clean":"2024-09-13T14:50:54.033851+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:50:54.033851+0000","last_undegraded":"2024-09-13T14:50:54.033851+0000","last_fullsized":"2024-09-13T14:50:54.033851+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"221'5589","reported_seq":11421,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:58.258882+0000","last_change":"2024-09-13T14:50:38.024593+0000","last_active":"2024-09-13T14:50:58.258882+0000","last_peered":"2024-09-13T14:50:58.258882+0000","last_clean":"2024-09-13T14:48:36.189224+0000","last_became_active":"2024-09-13T14:50:38.015505+0000","last_became_peered":"2024-09-13T14:50:38.015505+0000","last_unstale":"2024-09-13T14:50:58.258882+0000","last_undegraded":"2024-09-13T14:50:37.965648+0000","last_fullsized":"2024-09-13T14:50:37.942220+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328138752,"num_objects":5007,"num_object_clones":0,"num_object_copies":15021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5589,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2845,"num_bytes_recovered":186449920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":2946},{"shards":"2,3,5","objects":2061}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":842,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.038973+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:50:54.038973+0000","last_peered":"2024-09-13T14:50:54.038973+0000","last_clean":"2024-09-13T14:50:54.038973+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:50:54.038973+0000","last_undegraded":"2024-09-13T14:50:54.038973+0000","last_fullsized":"2024-09-13T14:50:54.038973+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":297,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013137+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:50:54.013137+0000","last_peered":"2024-09-13T14:50:54.013137+0000","last_clean":"2024-09-13T14:50:54.013137+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:50:54.013137+0000","last_undegraded":"2024-09-13T14:50:54.013137+0000","last_fullsized":"2024-09-13T14:50:54.013137+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":50,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.015455+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:50:54.015455+0000","last_peered":"2024-09-13T14:50:54.015455+0000","last_clean":"2024-09-13T14:50:54.015455+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:50:54.015455+0000","last_undegraded":"2024-09-13T14:50:54.015455+0000","last_fullsized":"2024-09-13T14:50:54.015455+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"221'5676","reported_seq":10250,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:58.211366+0000","last_change":"2024-09-13T14:50:42.308313+0000","last_active":"2024-09-13T14:50:58.211366+0000","last_peered":"2024-09-13T14:50:58.211366+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:50:58.211366+0000","last_undegraded":"2024-09-13T14:50:40.576119+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330563584,"num_objects":5044,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":89,"num_objects_missing":89,"num_objects_degraded":2392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":148504576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"1,4,5","objects":9},{"shards":"3,4","objects":2927},{"shards":"3,4,5","objects":2108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":74,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008443+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:50:52.008443+0000","last_peered":"2024-09-13T14:50:52.008443+0000","last_clean":"2024-09-13T14:50:52.008443+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:50:52.008443+0000","last_undegraded":"2024-09-13T14:50:52.008443+0000","last_fullsized":"2024-09-13T14:50:52.008443+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":204,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008835+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:50:52.008835+0000","last_peered":"2024-09-13T14:50:52.008835+0000","last_clean":"2024-09-13T14:50:52.008835+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:50:52.008835+0000","last_undegraded":"2024-09-13T14:50:52.008835+0000","last_fullsized":"2024-09-13T14:50:52.008835+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":300,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.019425+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:50:54.019425+0000","last_peered":"2024-09-13T14:50:54.019425+0000","last_clean":"2024-09-13T14:50:54.019425+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:50:54.019425+0000","last_undegraded":"2024-09-13T14:50:54.019425+0000","last_fullsized":"2024-09-13T14:50:54.019425+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"221'5717","reported_seq":5860,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:57.842210+0000","last_change":"2024-09-13T14:50:07.836955+0000","last_active":"2024-09-13T14:50:57.842210+0000","last_peered":"2024-09-13T14:50:57.842210+0000","last_clean":"2024-09-13T14:48:36.202432+0000","last_became_active":"2024-09-13T14:50:07.829887+0000","last_became_peered":"2024-09-13T14:50:07.829887+0000","last_unstale":"2024-09-13T14:50:57.842210+0000","last_undegraded":"2024-09-13T14:50:07.565518+0000","last_fullsized":"2024-09-13T14:50:07.543585+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5717,"log_dups_size":0,"ondisk_log_size":5717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334036992,"num_objects":5097,"num_object_clones":0,"num_object_copies":15291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5717,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":2982},{"shards":"0,3,5","objects":2115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":74,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008369+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:50:52.008369+0000","last_peered":"2024-09-13T14:50:52.008369+0000","last_clean":"2024-09-13T14:50:52.008369+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:50:52.008369+0000","last_undegraded":"2024-09-13T14:50:52.008369+0000","last_fullsized":"2024-09-13T14:50:52.008369+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":852,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.022995+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:50:54.022995+0000","last_peered":"2024-09-13T14:50:54.022995+0000","last_clean":"2024-09-13T14:50:54.022995+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:50:54.022995+0000","last_undegraded":"2024-09-13T14:50:54.022995+0000","last_fullsized":"2024-09-13T14:50:54.022995+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":277,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.014088+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:50:54.014088+0000","last_peered":"2024-09-13T14:50:54.014088+0000","last_clean":"2024-09-13T14:50:54.014088+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:50:54.014088+0000","last_undegraded":"2024-09-13T14:50:54.014088+0000","last_fullsized":"2024-09-13T14:50:54.014088+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"221'5740","reported_seq":10416,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.665422+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:50:59.665422+0000","last_peered":"2024-09-13T14:50:59.665422+0000","last_clean":"2024-09-13T14:50:59.665422+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:50:59.665422+0000","last_undegraded":"2024-09-13T14:50:59.665422+0000","last_fullsized":"2024-09-13T14:50:59.665422+0000","mapping_epoch":218,"log_start":"164'445","ondisk_log_start":"164'445","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338690048,"num_objects":5168,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":304,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.022942+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:50:54.022942+0000","last_peered":"2024-09-13T14:50:54.022942+0000","last_clean":"2024-09-13T14:50:54.022942+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:50:54.022942+0000","last_undegraded":"2024-09-13T14:50:54.022942+0000","last_fullsized":"2024-09-13T14:50:54.022942+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":525,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.034049+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:50:54.034049+0000","last_peered":"2024-09-13T14:50:54.034049+0000","last_clean":"2024-09-13T14:50:54.034049+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:50:54.034049+0000","last_undegraded":"2024-09-13T14:50:54.034049+0000","last_fullsized":"2024-09-13T14:50:54.034049+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":302,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033109+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:50:54.033109+0000","last_peered":"2024-09-13T14:50:54.033109+0000","last_clean":"2024-09-13T14:50:54.033109+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:50:54.033109+0000","last_undegraded":"2024-09-13T14:50:54.033109+0000","last_fullsized":"2024-09-13T14:50:54.033109+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"221'5547","reported_seq":7732,"reported_epoch":221,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:50:59.064846+0000","last_change":"2024-09-13T14:50:19.881564+0000","last_active":"2024-09-13T14:50:59.064846+0000","last_peered":"2024-09-13T14:50:59.064846+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:50:59.064846+0000","last_undegraded":"2024-09-13T14:50:19.795499+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325255168,"num_objects":4963,"num_object_clones":0,"num_object_copies":14889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2872},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2090}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":322,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.015184+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:50:54.015184+0000","last_peered":"2024-09-13T14:50:54.015184+0000","last_clean":"2024-09-13T14:50:54.015184+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:50:54.015184+0000","last_undegraded":"2024-09-13T14:50:54.015184+0000","last_fullsized":"2024-09-13T14:50:54.015184+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":753,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.015519+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:50:54.015519+0000","last_peered":"2024-09-13T14:50:54.015519+0000","last_clean":"2024-09-13T14:50:54.015519+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:50:54.015519+0000","last_undegraded":"2024-09-13T14:50:54.015519+0000","last_fullsized":"2024-09-13T14:50:54.015519+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":38,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013417+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:50:54.013417+0000","last_peered":"2024-09-13T14:50:54.013417+0000","last_clean":"2024-09-13T14:50:54.013417+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:50:54.013417+0000","last_undegraded":"2024-09-13T14:50:54.013417+0000","last_fullsized":"2024-09-13T14:50:54.013417+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"221'5689","reported_seq":7243,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:58.752707+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:50:58.752707+0000","last_peered":"2024-09-13T14:50:58.752707+0000","last_clean":"2024-09-13T14:50:58.752707+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:50:58.752707+0000","last_undegraded":"2024-09-13T14:50:58.752707+0000","last_fullsized":"2024-09-13T14:50:58.752707+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338886656,"num_objects":5171,"num_object_clones":0,"num_object_copies":15513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5689,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":297,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033533+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:50:54.033533+0000","last_peered":"2024-09-13T14:50:54.033533+0000","last_clean":"2024-09-13T14:50:54.033533+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:50:54.033533+0000","last_undegraded":"2024-09-13T14:50:54.033533+0000","last_fullsized":"2024-09-13T14:50:54.033533+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":772,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.011282+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:50:52.011282+0000","last_peered":"2024-09-13T14:50:52.011282+0000","last_clean":"2024-09-13T14:50:52.011282+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:50:52.011282+0000","last_undegraded":"2024-09-13T14:50:52.011282+0000","last_fullsized":"2024-09-13T14:50:52.011282+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":206,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008168+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:50:52.008168+0000","last_peered":"2024-09-13T14:50:52.008168+0000","last_clean":"2024-09-13T14:50:52.008168+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:50:52.008168+0000","last_undegraded":"2024-09-13T14:50:52.008168+0000","last_fullsized":"2024-09-13T14:50:52.008168+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":202,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008758+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:50:52.008758+0000","last_peered":"2024-09-13T14:50:52.008758+0000","last_clean":"2024-09-13T14:50:52.008758+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:50:52.008758+0000","last_undegraded":"2024-09-13T14:50:52.008758+0000","last_fullsized":"2024-09-13T14:50:52.008758+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":250,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008942+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:50:52.008942+0000","last_peered":"2024-09-13T14:50:52.008942+0000","last_clean":"2024-09-13T14:50:52.008942+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:50:52.008942+0000","last_undegraded":"2024-09-13T14:50:52.008942+0000","last_fullsized":"2024-09-13T14:50:52.008942+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":301,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013913+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:50:54.013913+0000","last_peered":"2024-09-13T14:50:54.013913+0000","last_clean":"2024-09-13T14:50:54.013913+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:50:54.013913+0000","last_undegraded":"2024-09-13T14:50:54.013913+0000","last_fullsized":"2024-09-13T14:50:54.013913+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":142,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.015720+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:50:54.015720+0000","last_peered":"2024-09-13T14:50:54.015720+0000","last_clean":"2024-09-13T14:50:54.015720+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:50:54.015720+0000","last_undegraded":"2024-09-13T14:50:54.015720+0000","last_fullsized":"2024-09-13T14:50:54.015720+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":83,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008396+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:50:52.008396+0000","last_peered":"2024-09-13T14:50:52.008396+0000","last_clean":"2024-09-13T14:50:52.008396+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:50:52.008396+0000","last_undegraded":"2024-09-13T14:50:52.008396+0000","last_fullsized":"2024-09-13T14:50:52.008396+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":278,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.014014+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:50:54.014014+0000","last_peered":"2024-09-13T14:50:54.014014+0000","last_clean":"2024-09-13T14:50:54.014014+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:50:54.014014+0000","last_undegraded":"2024-09-13T14:50:54.014014+0000","last_fullsized":"2024-09-13T14:50:54.014014+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":333,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033995+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:50:54.033995+0000","last_peered":"2024-09-13T14:50:54.033995+0000","last_clean":"2024-09-13T14:50:54.033995+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:50:54.033995+0000","last_undegraded":"2024-09-13T14:50:54.033995+0000","last_fullsized":"2024-09-13T14:50:54.033995+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":329,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013748+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:50:54.013748+0000","last_peered":"2024-09-13T14:50:54.013748+0000","last_clean":"2024-09-13T14:50:54.013748+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:50:54.013748+0000","last_undegraded":"2024-09-13T14:50:54.013748+0000","last_fullsized":"2024-09-13T14:50:54.013748+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":112,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.011201+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:50:52.011201+0000","last_peered":"2024-09-13T14:50:52.011201+0000","last_clean":"2024-09-13T14:50:52.011201+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:50:52.011201+0000","last_undegraded":"2024-09-13T14:50:52.011201+0000","last_fullsized":"2024-09-13T14:50:52.011201+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":296,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013908+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:50:54.013908+0000","last_peered":"2024-09-13T14:50:54.013908+0000","last_clean":"2024-09-13T14:50:54.013908+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:50:54.013908+0000","last_undegraded":"2024-09-13T14:50:54.013908+0000","last_fullsized":"2024-09-13T14:50:54.013908+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":313,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033767+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:50:54.033767+0000","last_peered":"2024-09-13T14:50:54.033767+0000","last_clean":"2024-09-13T14:50:54.033767+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:50:54.033767+0000","last_undegraded":"2024-09-13T14:50:54.033767+0000","last_fullsized":"2024-09-13T14:50:54.033767+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":282,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.013482+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:50:54.013482+0000","last_peered":"2024-09-13T14:50:54.013482+0000","last_clean":"2024-09-13T14:50:54.013482+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:50:54.013482+0000","last_undegraded":"2024-09-13T14:50:54.013482+0000","last_fullsized":"2024-09-13T14:50:54.013482+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":135,"reported_epoch":221,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:50:54.014553+0000","last_change":"2024-09-13T14:50:20.795913+0000","last_active":"2024-09-13T14:50:54.014553+0000","last_peered":"2024-09-13T14:50:54.014553+0000","last_clean":"2024-09-13T14:50:08.544047+0000","last_became_active":"2024-09-13T14:50:20.795534+0000","last_became_peered":"2024-09-13T14:50:20.795534+0000","last_unstale":"2024-09-13T14:50:54.014553+0000","last_undegraded":"2024-09-13T14:50:54.014553+0000","last_fullsized":"2024-09-13T14:50:54.014553+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[1,4,5],"avail_no_missing":["1","3","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":201,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.009273+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:50:52.009273+0000","last_peered":"2024-09-13T14:50:52.009273+0000","last_clean":"2024-09-13T14:50:52.009273+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:50:52.009273+0000","last_undegraded":"2024-09-13T14:50:52.009273+0000","last_fullsized":"2024-09-13T14:50:52.009273+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":41,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.037560+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:50:54.037560+0000","last_peered":"2024-09-13T14:50:54.037560+0000","last_clean":"2024-09-13T14:50:54.037560+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:50:54.037560+0000","last_undegraded":"2024-09-13T14:50:54.037560+0000","last_fullsized":"2024-09-13T14:50:54.037560+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":306,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033008+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:50:54.033008+0000","last_peered":"2024-09-13T14:50:54.033008+0000","last_clean":"2024-09-13T14:50:54.033008+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:50:54.033008+0000","last_undegraded":"2024-09-13T14:50:54.033008+0000","last_fullsized":"2024-09-13T14:50:54.033008+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":202,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008898+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:50:52.008898+0000","last_peered":"2024-09-13T14:50:52.008898+0000","last_clean":"2024-09-13T14:50:52.008898+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:50:52.008898+0000","last_undegraded":"2024-09-13T14:50:52.008898+0000","last_fullsized":"2024-09-13T14:50:52.008898+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":58,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024280+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:50:54.024280+0000","last_peered":"2024-09-13T14:50:54.024280+0000","last_clean":"2024-09-13T14:50:54.024280+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:50:54.024280+0000","last_undegraded":"2024-09-13T14:50:54.024280+0000","last_fullsized":"2024-09-13T14:50:54.024280+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":199,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.009202+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:50:52.009202+0000","last_peered":"2024-09-13T14:50:52.009202+0000","last_clean":"2024-09-13T14:50:52.009202+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:50:52.009202+0000","last_undegraded":"2024-09-13T14:50:52.009202+0000","last_fullsized":"2024-09-13T14:50:52.009202+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":60,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024501+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:50:54.024501+0000","last_peered":"2024-09-13T14:50:54.024501+0000","last_clean":"2024-09-13T14:50:54.024501+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:50:54.024501+0000","last_undegraded":"2024-09-13T14:50:54.024501+0000","last_fullsized":"2024-09-13T14:50:54.024501+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":299,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.014370+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:50:54.014370+0000","last_peered":"2024-09-13T14:50:54.014370+0000","last_clean":"2024-09-13T14:50:54.014370+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:50:54.014370+0000","last_undegraded":"2024-09-13T14:50:54.014370+0000","last_fullsized":"2024-09-13T14:50:54.014370+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":316,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.014985+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:50:54.014985+0000","last_peered":"2024-09-13T14:50:54.014985+0000","last_clean":"2024-09-13T14:50:54.014985+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:50:54.014985+0000","last_undegraded":"2024-09-13T14:50:54.014985+0000","last_fullsized":"2024-09-13T14:50:54.014985+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":229,"reported_epoch":220,"state":"active+clean","last_fresh":"2024-09-13T14:50:52.008027+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:50:52.008027+0000","last_peered":"2024-09-13T14:50:52.008027+0000","last_clean":"2024-09-13T14:50:52.008027+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:50:52.008027+0000","last_undegraded":"2024-09-13T14:50:52.008027+0000","last_fullsized":"2024-09-13T14:50:52.008027+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":369,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.033959+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:50:54.033959+0000","last_peered":"2024-09-13T14:50:54.033959+0000","last_clean":"2024-09-13T14:50:54.033959+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:50:54.033959+0000","last_undegraded":"2024-09-13T14:50:54.033959+0000","last_fullsized":"2024-09-13T14:50:54.033959+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5311692824,"num_objects":81051,"num_object_clones":0,"num_object_copies":243153,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":22665,"num_objects_misplaced":1445,"num_objects_unfound":0,"num_objects_dirty":81051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90497,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23104,"num_bytes_recovered":1503068160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16878546944,"data_stored":16878534728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89469,"ondisk_log_size":89469,"up":48,"acting":39,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234283,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495919,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3031800,"kb_used_data":2798064,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":90713352,"statfs":{"total":95995035648,"available":92890472448,"internally_reserved":0,"allocated":2865217536,"data_stored":2863994315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":239335879},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757554,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2655600,"kb_used_data":2476008,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":91089552,"statfs":{"total":95995035648,"available":93275701248,"internally_reserved":0,"allocated":2535432192,"data_stored":2534212766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":183892423},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019194,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1906872,"kb_used_data":1618992,"kb_used_omap":1,"kb_used_meta":287870,"kb_avail":91838280,"statfs":{"total":95995035648,"available":94042398720,"internally_reserved":0,"allocated":1657847808,"data_stored":1656624648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294779334},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468079,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3652140,"kb_used_data":3449764,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":90093012,"statfs":{"total":95995035648,"available":92255244288,"internally_reserved":0,"allocated":3532558336,"data_stored":3531342756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":207219876},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,0,3,2,0,2,3,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631608,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2513980,"kb_used_data":2203252,"kb_used_omap":8,"kb_used_meta":310711,"kb_avail":91231172,"statfs":{"total":95995035648,"available":93420720128,"internally_reserved":0,"allocated":2256130048,"data_stored":2254903318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":318169055},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795131,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2655156,"kb_used_data":2429100,"kb_used_omap":7,"kb_used_meta":226040,"kb_avail":91089996,"statfs":{"total":95995035648,"available":93276155904,"internally_reserved":0,"allocated":2487398400,"data_stored":2486175077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":231465586},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958661,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713208,"kb_used_data":1446896,"kb_used_omap":7,"kb_used_meta":266296,"kb_avail":92031944,"statfs":{"total":95995035648,"available":94240710656,"internally_reserved":0,"allocated":1481621504,"data_stored":1480394738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":272687591},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1468141568,"data_stored":1468137496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2481852416,"data_stored":2481848344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2247360512,"data_stored":2247360512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3524788224,"data_stored":3524788224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1654587392,"data_stored":1654587392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2532511744,"data_stored":2532507672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2858352640,"data_stored":2858352640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":110952448,"data_stored":110952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:03.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.319 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:03.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:03.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:03.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:03.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:03.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:03.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:03.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:03.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:03.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:03.824 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:03.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:03.948 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:03.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:04.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:04.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:04.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:04.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:04.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:04.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:04.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:04.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:04.685 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:04 smithi154 ceph-mon[102314]: pgmap v640: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 226 op/s; 22665/242679 objects degraded (9.339%); 1445/242679 objects misplaced (0.595%); 3.9 MiB/s, 64 objects/s recovering 2024-09-13T14:51:04.685 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:04 smithi154 ceph-mon[107038]: pgmap v640: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 226 op/s; 22665/242679 objects degraded (9.339%); 1445/242679 objects misplaced (0.595%); 3.9 MiB/s, 64 objects/s recovering 2024-09-13T14:51:04.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:04.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:04.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.857 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:04.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:04.900 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:04 smithi172 ceph-mon[100696]: pgmap v640: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 226 op/s; 22665/242679 objects degraded (9.339%); 1445/242679 objects misplaced (0.595%); 3.9 MiB/s, 64 objects/s recovering 2024-09-13T14:51:04.937 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:51:04.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:04.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:04.960 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:51:04.960 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:04.980 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:51:04.980 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:04.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:04.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:05.003 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:51:05.003 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:05.023 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:51:05.023 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:05.051 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:51:05.051 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:05.075 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:51:05.075 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:05.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.099 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:51:05.100 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:05.109 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.176 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:05.176 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:05.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:05.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:05.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:05.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:05.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:05.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.837 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 22665/242679 objects degraded (9.339%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:51:05.838 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:05 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2680836366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:05.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:05.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:51:05.868 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:05.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 22665/242679 objects degraded (9.339%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:51:05.889 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:05 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2680836366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:05.890 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 22665/242679 objects degraded (9.339%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:51:05.890 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:05 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2680836366' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:05.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:05.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:05.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:06.095 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:06.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:06.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:06.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:06.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:06.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:06.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:06.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:06.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:06.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:06.640 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:06.663 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:51:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:06 smithi172 ceph-mon[100696]: pgmap v641: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 302 op/s; 22665/238800 objects degraded (9.491%); 747/238800 objects misplaced (0.313%); 7.5 MiB/s, 122 objects/s recovering 2024-09-13T14:51:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1424985079' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:06 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:06.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:06.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:06.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:06.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:06.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:06.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:06 smithi154 ceph-mon[102314]: pgmap v641: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 302 op/s; 22665/238800 objects degraded (9.491%); 747/238800 objects misplaced (0.313%); 7.5 MiB/s, 122 objects/s recovering 2024-09-13T14:51:06.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1424985079' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:06.974 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:06 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:06.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:06 smithi154 ceph-mon[107038]: pgmap v641: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 302 op/s; 22665/238800 objects degraded (9.491%); 747/238800 objects misplaced (0.313%); 7.5 MiB/s, 122 objects/s recovering 2024-09-13T14:51:06.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1424985079' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:06.975 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:06 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:06.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:07.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:07.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:07.152 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:07.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:07.278 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:07.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:07.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:07.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:07.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:07.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:07.657 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:07.698 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:07.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:07.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:07.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:07.902 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:07 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:07.902 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:07 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:07.902 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:07 smithi172 ceph-mon[100696]: osdmap e222: 8 total, 8 up, 7 in 2024-09-13T14:51:07.908 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:07.909 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:07 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:07.909 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:07 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:07.909 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:07 smithi154 ceph-mon[102314]: osdmap e222: 8 total, 8 up, 7 in 2024-09-13T14:51:07.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:07 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:07.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:07 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:07.910 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:07 smithi154 ceph-mon[107038]: osdmap e222: 8 total, 8 up, 7 in 2024-09-13T14:51:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:07.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:08.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:08.012 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:08.012 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:08.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:08.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:08.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:08.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:08.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:08.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:08.421 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:08.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:08.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:08.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:08.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:08.677 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:08.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.803 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:08.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:08.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:08.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:08.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:08.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:08 smithi172 ceph-mon[100696]: pgmap v643: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 279 op/s; 22665/238800 objects degraded (9.491%); 747/238800 objects misplaced (0.313%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:51:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:08 smithi154 ceph-mon[102314]: pgmap v643: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 279 op/s; 22665/238800 objects degraded (9.491%); 747/238800 objects misplaced (0.313%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:51:08.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:08 smithi154 ceph-mon[107038]: pgmap v643: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 279 op/s; 22665/238800 objects degraded (9.491%); 747/238800 objects misplaced (0.313%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:51:08.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:51:09.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:09.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:09.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:09.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:09.212 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:09.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:09.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:09.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:09.455 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:09.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:09.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:09.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:51:09.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:09.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:09.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:09.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:09.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:09.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:09.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:10.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:10.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:10.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:10.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:10.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:10.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:10.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:10.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:10.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.629 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:10.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:10.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:10 smithi172 ceph-mon[100696]: pgmap v644: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 291 op/s; 22665/236715 objects degraded (9.575%); 747/236715 objects misplaced (0.316%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:51:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:10 smithi172 ceph-mon[100696]: osdmap e223: 8 total, 8 up, 7 in 2024-09-13T14:51:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 22665/236715 objects degraded (9.575%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:10.745 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:10 smithi154 ceph-mon[107038]: pgmap v644: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 291 op/s; 22665/236715 objects degraded (9.575%); 747/236715 objects misplaced (0.316%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:51:10.745 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:10 smithi154 ceph-mon[107038]: osdmap e223: 8 total, 8 up, 7 in 2024-09-13T14:51:10.745 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 22665/236715 objects degraded (9.575%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:10.746 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:10 smithi154 ceph-mon[102314]: pgmap v644: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 291 op/s; 22665/236715 objects degraded (9.575%); 747/236715 objects misplaced (0.316%); 7.8 MiB/s, 124 objects/s recovering 2024-09-13T14:51:10.747 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:10 smithi154 ceph-mon[102314]: osdmap e223: 8 total, 8 up, 7 in 2024-09-13T14:51:10.747 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 22665/236715 objects degraded (9.575%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:10.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:10.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:10.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:10.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:10.926 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:10.926 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:11.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:11.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:11.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:11.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:11.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:11.324 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:11.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:11.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:11.498 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:51:11.499 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:51:11.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:11.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:51:11.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:11.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:11.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:11.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:11.772 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:11.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:11.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:11.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:11 smithi172 ceph-mon[100696]: osdmap e224: 8 total, 8 up, 7 in 2024-09-13T14:51:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:11 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:11.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:11 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:11.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:11.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:12.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:12.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:11 smithi154 ceph-mon[102314]: osdmap e224: 8 total, 8 up, 7 in 2024-09-13T14:51:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:11 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:12.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:11 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:12.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:11 smithi154 ceph-mon[107038]: osdmap e224: 8 total, 8 up, 7 in 2024-09-13T14:51:12.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:11 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:12.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:11 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:12.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:51:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:51:11] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:51:12.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:12.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:12.348 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:12.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:12.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:12.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:12.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:12.627 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:12.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:12.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:12.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:12.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:12.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:12 smithi154 ceph-mon[102314]: pgmap v647: 89 pgs: 1 peering, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 22510/234177 objects degraded (9.612%); 7.8 MiB/s, 150 objects/s recovering 2024-09-13T14:51:12.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:12 smithi154 ceph-mon[102314]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:12 smithi154 ceph-mon[107038]: pgmap v647: 89 pgs: 1 peering, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 22510/234177 objects degraded (9.612%); 7.8 MiB/s, 150 objects/s recovering 2024-09-13T14:51:12.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:12 smithi154 ceph-mon[107038]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:12.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:12 smithi172 ceph-mon[100696]: pgmap v647: 89 pgs: 1 peering, 1 active+remapped+backfill_wait, 8 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 22510/234177 objects degraded (9.612%); 7.8 MiB/s, 150 objects/s recovering 2024-09-13T14:51:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:12 smithi172 ceph-mon[100696]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:12.999 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:13.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:13.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:13.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:13.257 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:13.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:13.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:13.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.515 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:13.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:13.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:13.624 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:13.624 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:13.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:13.747 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:13.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:13.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:13.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:13.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:13.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:13.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:14.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:14.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:14.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:14.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:14.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:14.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:51:14.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:14.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:14.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:14.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:14.574 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:14.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:14.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:14.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:14.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:14.902 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:14.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:14.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:14 smithi172 ceph-mon[100696]: pgmap v648: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 20312/233673 objects degraded (8.692%); 1765/233673 objects misplaced (0.755%); 11 MiB/s, 198 objects/s recovering 2024-09-13T14:51:15.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:15.050 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:15.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:15.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:14 smithi154 ceph-mon[102314]: pgmap v648: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 20312/233673 objects degraded (8.692%); 1765/233673 objects misplaced (0.755%); 11 MiB/s, 198 objects/s recovering 2024-09-13T14:51:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:14 smithi154 ceph-mon[107038]: pgmap v648: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 20312/233673 objects degraded (8.692%); 1765/233673 objects misplaced (0.755%); 11 MiB/s, 198 objects/s recovering 2024-09-13T14:51:15.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:15.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:15.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:15.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:15.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:15.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:15.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:15.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:15.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:15.824 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:15.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:15.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:15.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:15.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 20312/233673 objects degraded (8.692%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:16.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:16.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:16.139 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 20312/233673 objects degraded (8.692%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:16.143 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 20312/233673 objects degraded (8.692%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:16.152 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:16.180 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2024-09-13T14:51:10.965932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5118564349,"num_objects":78059,"num_object_clones":0,"num_object_copies":234177,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":22510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78059,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":94640,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24860,"num_bytes_recovered":1600146565,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93271,"ondisk_log_size":93271,"up":267,"acting":259,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":296,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17652520,"kb_used_data":15931948,"kb_used_omap":32,"kb_used_meta":1720479,"kb_avail":638563544,"statfs":{"total":671965249536,"available":653889069056,"internally_reserved":0,"allocated":16314314752,"data_stored":16305701512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1761771056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-100990976,"num_objects":-1541,"num_object_clones":0,"num_object_copies":-4623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":-747,"num_objects_unfound":0,"num_objects_dirty":-1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1541,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":48955392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004241"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":325,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.735764+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:51:06.735764+0000","last_peered":"2024-09-13T14:51:06.735764+0000","last_clean":"2024-09-13T14:51:06.735764+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:51:06.735764+0000","last_undegraded":"2024-09-13T14:51:06.735764+0000","last_fullsized":"2024-09-13T14:51:06.735764+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":87,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018102+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:50:54.018102+0000","last_peered":"2024-09-13T14:50:54.018102+0000","last_clean":"2024-09-13T14:50:54.018102+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:50:54.018102+0000","last_undegraded":"2024-09-13T14:50:54.018102+0000","last_fullsized":"2024-09-13T14:50:54.018102+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":57,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024478+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:50:54.024478+0000","last_peered":"2024-09-13T14:50:54.024478+0000","last_clean":"2024-09-13T14:50:54.024478+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:50:54.024478+0000","last_undegraded":"2024-09-13T14:50:54.024478+0000","last_fullsized":"2024-09-13T14:50:54.024478+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":314,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.670294+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:51:06.670294+0000","last_peered":"2024-09-13T14:51:06.670294+0000","last_clean":"2024-09-13T14:51:06.670294+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:51:06.670294+0000","last_undegraded":"2024-09-13T14:51:06.670294+0000","last_fullsized":"2024-09-13T14:51:06.670294+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":314,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669176+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:51:06.669176+0000","last_peered":"2024-09-13T14:51:06.669176+0000","last_clean":"2024-09-13T14:51:06.669176+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:51:06.669176+0000","last_undegraded":"2024-09-13T14:51:06.669176+0000","last_fullsized":"2024-09-13T14:51:06.669176+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":157,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.017720+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:50:54.017720+0000","last_peered":"2024-09-13T14:50:54.017720+0000","last_clean":"2024-09-13T14:50:54.017720+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:50:54.017720+0000","last_undegraded":"2024-09-13T14:50:54.017720+0000","last_fullsized":"2024-09-13T14:50:54.017720+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":40,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.664833+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:51:06.664833+0000","last_peered":"2024-09-13T14:51:06.664833+0000","last_clean":"2024-09-13T14:51:06.664833+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:51:06.664833+0000","last_undegraded":"2024-09-13T14:51:06.664833+0000","last_fullsized":"2024-09-13T14:51:06.664833+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":133,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.036924+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:50:54.036924+0000","last_peered":"2024-09-13T14:50:54.036924+0000","last_clean":"2024-09-13T14:50:54.036924+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:50:54.036924+0000","last_undegraded":"2024-09-13T14:50:54.036924+0000","last_fullsized":"2024-09-13T14:50:54.036924+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"222'5946","reported_seq":6761,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.356767+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:51:09.356767+0000","last_peered":"2024-09-13T14:51:09.356767+0000","last_clean":"2024-09-13T14:51:09.356767+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:51:09.356767+0000","last_undegraded":"2024-09-13T14:51:09.356767+0000","last_fullsized":"2024-09-13T14:51:09.356767+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324403200,"num_objects":4950,"num_object_clones":0,"num_object_copies":14850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":249,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.661306+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:51:06.661306+0000","last_peered":"2024-09-13T14:51:06.661306+0000","last_clean":"2024-09-13T14:51:06.661306+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:51:06.661306+0000","last_undegraded":"2024-09-13T14:51:06.661306+0000","last_fullsized":"2024-09-13T14:51:06.661306+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":84,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024213+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:50:54.024213+0000","last_peered":"2024-09-13T14:50:54.024213+0000","last_clean":"2024-09-13T14:50:54.024213+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:50:54.024213+0000","last_undegraded":"2024-09-13T14:50:54.024213+0000","last_fullsized":"2024-09-13T14:50:54.024213+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"222'5753","reported_seq":7325,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:08.403687+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:51:08.403687+0000","last_peered":"2024-09-13T14:51:08.403687+0000","last_clean":"2024-09-13T14:51:08.403687+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:51:08.403687+0000","last_undegraded":"2024-09-13T14:51:08.403687+0000","last_fullsized":"2024-09-13T14:51:08.403687+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5753,"log_dups_size":0,"ondisk_log_size":5753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316211200,"num_objects":4825,"num_object_clones":0,"num_object_copies":14475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5753,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":85,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018256+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:50:54.018256+0000","last_peered":"2024-09-13T14:50:54.018256+0000","last_clean":"2024-09-13T14:50:54.018256+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:50:54.018256+0000","last_undegraded":"2024-09-13T14:50:54.018256+0000","last_fullsized":"2024-09-13T14:50:54.018256+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":237,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.017829+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:50:54.017829+0000","last_peered":"2024-09-13T14:50:54.017829+0000","last_clean":"2024-09-13T14:50:54.017829+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:50:54.017829+0000","last_undegraded":"2024-09-13T14:50:54.017829+0000","last_fullsized":"2024-09-13T14:50:54.017829+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"222'5904","reported_seq":7569,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.200105+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:51:09.200105+0000","last_peered":"2024-09-13T14:51:09.200105+0000","last_clean":"2024-09-13T14:51:09.200105+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:51:09.200105+0000","last_undegraded":"2024-09-13T14:51:09.200105+0000","last_fullsized":"2024-09-13T14:51:09.200105+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5904,"log_dups_size":0,"ondisk_log_size":5904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323747840,"num_objects":4940,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5904,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":61,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024550+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:50:54.024550+0000","last_peered":"2024-09-13T14:50:54.024550+0000","last_clean":"2024-09-13T14:50:54.024550+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:50:54.024550+0000","last_undegraded":"2024-09-13T14:50:54.024550+0000","last_fullsized":"2024-09-13T14:50:54.024550+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.036875+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:50:54.036875+0000","last_peered":"2024-09-13T14:50:54.036875+0000","last_clean":"2024-09-13T14:50:54.036875+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:50:54.036875+0000","last_undegraded":"2024-09-13T14:50:54.036875+0000","last_fullsized":"2024-09-13T14:50:54.036875+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"222'5709","reported_seq":7434,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:07.700281+0000","last_change":"2024-09-13T14:50:07.839326+0000","last_active":"2024-09-13T14:51:07.700281+0000","last_peered":"2024-09-13T14:51:07.700281+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:51:07.700281+0000","last_undegraded":"2024-09-13T14:50:07.570096+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5709,"log_dups_size":0,"ondisk_log_size":5709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314507264,"num_objects":4799,"num_object_clones":0,"num_object_copies":14397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5709,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":51445760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":2698},{"shards":"0,3,6","objects":2101}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":73,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024464+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:50:54.024464+0000","last_peered":"2024-09-13T14:50:54.024464+0000","last_clean":"2024-09-13T14:50:54.024464+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:50:54.024464+0000","last_undegraded":"2024-09-13T14:50:54.024464+0000","last_fullsized":"2024-09-13T14:50:54.024464+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":293,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669987+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:51:06.669987+0000","last_peered":"2024-09-13T14:51:06.669987+0000","last_clean":"2024-09-13T14:51:06.669987+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:51:06.669987+0000","last_undegraded":"2024-09-13T14:51:06.669987+0000","last_fullsized":"2024-09-13T14:51:06.669987+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"222'5855","reported_seq":12323,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:09.067718+0000","last_change":"2024-09-13T14:50:08.716863+0000","last_active":"2024-09-13T14:51:09.067718+0000","last_peered":"2024-09-13T14:51:09.067718+0000","last_clean":"2024-09-13T14:49:59.725669+0000","last_became_active":"2024-09-13T14:50:08.707516+0000","last_became_peered":"2024-09-13T14:50:08.707516+0000","last_unstale":"2024-09-13T14:51:09.067718+0000","last_undegraded":"2024-09-13T14:50:08.570515+0000","last_fullsized":"2024-09-13T14:50:08.544112+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:20.218839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321060864,"num_objects":4899,"num_object_clones":0,"num_object_copies":14697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5855,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3185,"num_bytes_recovered":208732160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-423},{"shards":"1,4,5","objects":4899}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":302,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.664472+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:51:06.664472+0000","last_peered":"2024-09-13T14:51:06.664472+0000","last_clean":"2024-09-13T14:51:06.664472+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:51:06.664472+0000","last_undegraded":"2024-09-13T14:51:06.664472+0000","last_fullsized":"2024-09-13T14:51:06.664472+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":204,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018088+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:50:54.018088+0000","last_peered":"2024-09-13T14:50:54.018088+0000","last_clean":"2024-09-13T14:50:54.018088+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:50:54.018088+0000","last_undegraded":"2024-09-13T14:50:54.018088+0000","last_fullsized":"2024-09-13T14:50:54.018088+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"222'5811","reported_seq":6176,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:09.109565+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:51:09.109565+0000","last_peered":"2024-09-13T14:51:09.109565+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:51:09.109565+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316473344,"num_objects":4829,"num_object_clones":0,"num_object_copies":14487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":2649},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":104,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.037060+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:50:54.037060+0000","last_peered":"2024-09-13T14:50:54.037060+0000","last_clean":"2024-09-13T14:50:54.037060+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:50:54.037060+0000","last_undegraded":"2024-09-13T14:50:54.037060+0000","last_fullsized":"2024-09-13T14:50:54.037060+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":201,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018237+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:50:54.018237+0000","last_peered":"2024-09-13T14:50:54.018237+0000","last_clean":"2024-09-13T14:50:54.018237+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:50:54.018237+0000","last_undegraded":"2024-09-13T14:50:54.018237+0000","last_fullsized":"2024-09-13T14:50:54.018237+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"222'5943","reported_seq":8056,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:08.412419+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:51:08.412419+0000","last_peered":"2024-09-13T14:51:08.412419+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:51:08.412419+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319750144,"num_objects":4879,"num_object_clones":0,"num_object_copies":14637,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":3649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5943,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73203712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":1248},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":1485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":196,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.017866+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:50:54.017866+0000","last_peered":"2024-09-13T14:50:54.017866+0000","last_clean":"2024-09-13T14:50:54.017866+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:50:54.017866+0000","last_undegraded":"2024-09-13T14:50:54.017866+0000","last_fullsized":"2024-09-13T14:50:54.017866+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":231,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018339+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:50:54.018339+0000","last_peered":"2024-09-13T14:50:54.018339+0000","last_clean":"2024-09-13T14:50:54.018339+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:50:54.018339+0000","last_undegraded":"2024-09-13T14:50:54.018339+0000","last_fullsized":"2024-09-13T14:50:54.018339+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"222'5722","reported_seq":10131,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.514590+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:51:09.514590+0000","last_peered":"2024-09-13T14:51:09.514590+0000","last_clean":"2024-09-13T14:51:09.514590+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:51:09.514590+0000","last_undegraded":"2024-09-13T14:51:09.514590+0000","last_fullsized":"2024-09-13T14:51:09.514590+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313458712,"num_objects":4784,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5722,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":40,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.667754+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:51:06.667754+0000","last_peered":"2024-09-13T14:51:06.667754+0000","last_clean":"2024-09-13T14:51:06.667754+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:51:06.667754+0000","last_undegraded":"2024-09-13T14:51:06.667754+0000","last_fullsized":"2024-09-13T14:51:06.667754+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":149,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.661415+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:51:06.661415+0000","last_peered":"2024-09-13T14:51:06.661415+0000","last_clean":"2024-09-13T14:51:06.661415+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:51:06.661415+0000","last_undegraded":"2024-09-13T14:51:06.661415+0000","last_fullsized":"2024-09-13T14:51:06.661415+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"222'5823","reported_seq":7053,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.349894+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:51:09.349894+0000","last_peered":"2024-09-13T14:51:09.349894+0000","last_clean":"2024-09-13T14:51:09.349894+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:51:09.349894+0000","last_undegraded":"2024-09-13T14:51:09.349894+0000","last_fullsized":"2024-09-13T14:51:09.349894+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322502656,"num_objects":4921,"num_object_clones":0,"num_object_copies":14763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":496,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.681192+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:51:06.681192+0000","last_peered":"2024-09-13T14:51:06.681192+0000","last_clean":"2024-09-13T14:51:06.681192+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:51:06.681192+0000","last_undegraded":"2024-09-13T14:51:06.681192+0000","last_fullsized":"2024-09-13T14:51:06.681192+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":788,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:59.578036+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:50:59.578036+0000","last_peered":"2024-09-13T14:50:59.578036+0000","last_clean":"2024-09-13T14:50:59.578036+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:50:59.578036+0000","last_undegraded":"2024-09-13T14:50:59.578036+0000","last_fullsized":"2024-09-13T14:50:59.578036+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":260,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.665265+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:51:06.665265+0000","last_peered":"2024-09-13T14:51:06.665265+0000","last_clean":"2024-09-13T14:51:06.665265+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:51:06.665265+0000","last_undegraded":"2024-09-13T14:51:06.665265+0000","last_fullsized":"2024-09-13T14:51:06.665265+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.037271+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:50:54.037271+0000","last_peered":"2024-09-13T14:50:54.037271+0000","last_clean":"2024-09-13T14:50:54.037271+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:50:54.037271+0000","last_undegraded":"2024-09-13T14:50:54.037271+0000","last_fullsized":"2024-09-13T14:50:54.037271+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"222'5926","reported_seq":10607,"reported_epoch":222,"state":"peering","last_fresh":"2024-09-13T14:51:09.661247+0000","last_change":"2024-09-13T14:51:09.709044+0000","last_active":"2024-09-13T14:51:09.709044+0000","last_peered":"2024-09-13T14:51:09.661247+0000","last_clean":"2024-09-13T14:48:19.187296+0000","last_became_active":"2024-09-13T14:50:32.492668+0000","last_became_peered":"2024-09-13T14:50:32.492668+0000","last_unstale":"2024-09-13T14:51:09.661247+0000","last_undegraded":"2024-09-13T14:51:09.661247+0000","last_fullsized":"2024-09-13T14:50:23.793735+0000","mapping_epoch":223,"log_start":"164'618","ondisk_log_start":"164'618","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322568192,"num_objects":4922,"num_object_clones":0,"num_object_copies":14766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":343,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.704333+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:51:09.704333+0000","last_peered":"2024-09-13T14:51:09.704333+0000","last_clean":"2024-09-13T14:51:09.704333+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:51:09.704333+0000","last_undegraded":"2024-09-13T14:51:09.704333+0000","last_fullsized":"2024-09-13T14:51:09.704333+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":308,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.668939+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:51:06.668939+0000","last_peered":"2024-09-13T14:51:06.668939+0000","last_clean":"2024-09-13T14:51:06.668939+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:51:06.668939+0000","last_undegraded":"2024-09-13T14:51:06.668939+0000","last_fullsized":"2024-09-13T14:51:06.668939+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":319,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669371+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:51:06.669371+0000","last_peered":"2024-09-13T14:51:06.669371+0000","last_clean":"2024-09-13T14:51:06.669371+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:51:06.669371+0000","last_undegraded":"2024-09-13T14:51:06.669371+0000","last_fullsized":"2024-09-13T14:51:06.669371+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"222'5767","reported_seq":11601,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:08.369384+0000","last_change":"2024-09-13T14:50:38.024593+0000","last_active":"2024-09-13T14:51:08.369384+0000","last_peered":"2024-09-13T14:51:08.369384+0000","last_clean":"2024-09-13T14:48:36.189224+0000","last_became_active":"2024-09-13T14:50:38.015505+0000","last_became_peered":"2024-09-13T14:50:38.015505+0000","last_unstale":"2024-09-13T14:51:08.369384+0000","last_undegraded":"2024-09-13T14:50:37.965648+0000","last_fullsized":"2024-09-13T14:50:37.942220+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5767,"log_dups_size":0,"ondisk_log_size":5767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316473344,"num_objects":4829,"num_object_clones":0,"num_object_copies":14487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5767,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2845,"num_bytes_recovered":186449920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":2768},{"shards":"2,3,5","objects":2061}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":844,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.661559+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:51:06.661559+0000","last_peered":"2024-09-13T14:51:06.661559+0000","last_clean":"2024-09-13T14:51:06.661559+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:51:06.661559+0000","last_undegraded":"2024-09-13T14:51:06.661559+0000","last_fullsized":"2024-09-13T14:51:06.661559+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":301,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.704614+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:51:09.704614+0000","last_peered":"2024-09-13T14:51:09.704614+0000","last_clean":"2024-09-13T14:51:09.704614+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:51:09.704614+0000","last_undegraded":"2024-09-13T14:51:09.704614+0000","last_fullsized":"2024-09-13T14:51:09.704614+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":54,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.705396+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:51:09.705396+0000","last_peered":"2024-09-13T14:51:09.705396+0000","last_clean":"2024-09-13T14:51:09.705396+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:51:09.705396+0000","last_undegraded":"2024-09-13T14:51:09.705396+0000","last_fullsized":"2024-09-13T14:51:09.705396+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"222'5855","reported_seq":10431,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:08.443868+0000","last_change":"2024-09-13T14:50:42.308313+0000","last_active":"2024-09-13T14:51:08.443868+0000","last_peered":"2024-09-13T14:51:08.443868+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:51:08.443868+0000","last_undegraded":"2024-09-13T14:50:40.576119+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318832640,"num_objects":4865,"num_object_clones":0,"num_object_copies":14595,"num_objects_missing_on_primary":89,"num_objects_missing":89,"num_objects_degraded":2392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5855,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":148504576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"1,4,5","objects":9},{"shards":"3,4","objects":2748},{"shards":"3,4,5","objects":2108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":75,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024264+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:50:54.024264+0000","last_peered":"2024-09-13T14:50:54.024264+0000","last_clean":"2024-09-13T14:50:54.024264+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:50:54.024264+0000","last_undegraded":"2024-09-13T14:50:54.024264+0000","last_fullsized":"2024-09-13T14:50:54.024264+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":205,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018334+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:50:54.018334+0000","last_peered":"2024-09-13T14:50:54.018334+0000","last_clean":"2024-09-13T14:50:54.018334+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:50:54.018334+0000","last_undegraded":"2024-09-13T14:50:54.018334+0000","last_fullsized":"2024-09-13T14:50:54.018334+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":304,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.710256+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:51:09.710256+0000","last_peered":"2024-09-13T14:51:09.710256+0000","last_clean":"2024-09-13T14:51:09.710256+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:51:09.710256+0000","last_undegraded":"2024-09-13T14:51:09.710256+0000","last_fullsized":"2024-09-13T14:51:09.710256+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"222'5936","reported_seq":6081,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:07.893670+0000","last_change":"2024-09-13T14:50:07.836955+0000","last_active":"2024-09-13T14:51:07.893670+0000","last_peered":"2024-09-13T14:51:07.893670+0000","last_clean":"2024-09-13T14:48:36.202432+0000","last_became_active":"2024-09-13T14:50:07.829887+0000","last_became_peered":"2024-09-13T14:50:07.829887+0000","last_unstale":"2024-09-13T14:51:07.893670+0000","last_undegraded":"2024-09-13T14:50:07.565518+0000","last_fullsized":"2024-09-13T14:50:07.543585+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5936,"log_dups_size":0,"ondisk_log_size":5936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319684608,"num_objects":4878,"num_object_clones":0,"num_object_copies":14634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5936,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":2763},{"shards":"0,3,5","objects":2115}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":75,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024384+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:50:54.024384+0000","last_peered":"2024-09-13T14:50:54.024384+0000","last_clean":"2024-09-13T14:50:54.024384+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:50:54.024384+0000","last_undegraded":"2024-09-13T14:50:54.024384+0000","last_fullsized":"2024-09-13T14:50:54.024384+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":856,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.710445+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:51:09.710445+0000","last_peered":"2024-09-13T14:51:09.710445+0000","last_clean":"2024-09-13T14:51:09.710445+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:51:09.710445+0000","last_undegraded":"2024-09-13T14:51:09.710445+0000","last_fullsized":"2024-09-13T14:51:09.710445+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":281,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.704949+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:51:09.704949+0000","last_peered":"2024-09-13T14:51:09.704949+0000","last_clean":"2024-09-13T14:51:09.704949+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:51:09.704949+0000","last_undegraded":"2024-09-13T14:51:09.704949+0000","last_fullsized":"2024-09-13T14:51:09.704949+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"222'5949","reported_seq":10629,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.706981+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:51:09.706981+0000","last_peered":"2024-09-13T14:51:09.706981+0000","last_clean":"2024-09-13T14:51:09.706981+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:51:09.706981+0000","last_undegraded":"2024-09-13T14:51:09.706981+0000","last_fullsized":"2024-09-13T14:51:09.706981+0000","mapping_epoch":218,"log_start":"164'645","ondisk_log_start":"164'645","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324993024,"num_objects":4959,"num_object_clones":0,"num_object_copies":14877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5949,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":308,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.710354+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:51:09.710354+0000","last_peered":"2024-09-13T14:51:09.710354+0000","last_clean":"2024-09-13T14:51:09.710354+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:51:09.710354+0000","last_undegraded":"2024-09-13T14:51:09.710354+0000","last_fullsized":"2024-09-13T14:51:09.710354+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":527,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.668877+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:51:06.668877+0000","last_peered":"2024-09-13T14:51:06.668877+0000","last_clean":"2024-09-13T14:51:06.668877+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:51:06.668877+0000","last_undegraded":"2024-09-13T14:51:06.668877+0000","last_fullsized":"2024-09-13T14:51:06.668877+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":304,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669919+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:51:06.669919+0000","last_peered":"2024-09-13T14:51:06.669919+0000","last_clean":"2024-09-13T14:51:06.669919+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:51:06.669919+0000","last_undegraded":"2024-09-13T14:51:06.669919+0000","last_fullsized":"2024-09-13T14:51:06.669919+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"222'5735","reported_seq":7921,"reported_epoch":222,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:08.809205+0000","last_change":"2024-09-13T14:50:19.881564+0000","last_active":"2024-09-13T14:51:08.809205+0000","last_peered":"2024-09-13T14:51:08.809205+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:51:08.809205+0000","last_undegraded":"2024-09-13T14:50:19.795499+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5735,"log_dups_size":0,"ondisk_log_size":5735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312934400,"num_objects":4775,"num_object_clones":0,"num_object_copies":14325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5735,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2684},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2090}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":324,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.665164+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:51:06.665164+0000","last_peered":"2024-09-13T14:51:06.665164+0000","last_clean":"2024-09-13T14:51:06.665164+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:51:06.665164+0000","last_undegraded":"2024-09-13T14:51:06.665164+0000","last_fullsized":"2024-09-13T14:51:06.665164+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":757,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.708305+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:51:09.708305+0000","last_peered":"2024-09-13T14:51:09.708305+0000","last_clean":"2024-09-13T14:51:09.708305+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:51:09.708305+0000","last_undegraded":"2024-09-13T14:51:09.708305+0000","last_fullsized":"2024-09-13T14:51:09.708305+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":40,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.667653+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:51:06.667653+0000","last_peered":"2024-09-13T14:51:06.667653+0000","last_clean":"2024-09-13T14:51:06.667653+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:51:06.667653+0000","last_undegraded":"2024-09-13T14:51:06.667653+0000","last_fullsized":"2024-09-13T14:51:06.667653+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"222'5883","reported_seq":7438,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:08.754517+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:51:08.754517+0000","last_peered":"2024-09-13T14:51:08.754517+0000","last_clean":"2024-09-13T14:51:08.754517+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:51:08.754517+0000","last_undegraded":"2024-09-13T14:51:08.754517+0000","last_fullsized":"2024-09-13T14:51:08.754517+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326172672,"num_objects":4977,"num_object_clones":0,"num_object_copies":14931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5883,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":299,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669043+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:51:06.669043+0000","last_peered":"2024-09-13T14:51:06.669043+0000","last_clean":"2024-09-13T14:51:06.669043+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:51:06.669043+0000","last_undegraded":"2024-09-13T14:51:06.669043+0000","last_fullsized":"2024-09-13T14:51:06.669043+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":773,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.037126+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:50:54.037126+0000","last_peered":"2024-09-13T14:50:54.037126+0000","last_clean":"2024-09-13T14:50:54.037126+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:50:54.037126+0000","last_undegraded":"2024-09-13T14:50:54.037126+0000","last_fullsized":"2024-09-13T14:50:54.037126+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":208,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.676603+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:51:06.676603+0000","last_peered":"2024-09-13T14:51:06.676603+0000","last_clean":"2024-09-13T14:51:06.676603+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:51:06.676603+0000","last_undegraded":"2024-09-13T14:51:06.676603+0000","last_fullsized":"2024-09-13T14:51:06.676603+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":203,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018255+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:50:54.018255+0000","last_peered":"2024-09-13T14:50:54.018255+0000","last_clean":"2024-09-13T14:50:54.018255+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:50:54.018255+0000","last_undegraded":"2024-09-13T14:50:54.018255+0000","last_fullsized":"2024-09-13T14:50:54.018255+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":251,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018308+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:50:54.018308+0000","last_peered":"2024-09-13T14:50:54.018308+0000","last_clean":"2024-09-13T14:50:54.018308+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:50:54.018308+0000","last_undegraded":"2024-09-13T14:50:54.018308+0000","last_fullsized":"2024-09-13T14:50:54.018308+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":305,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.704854+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:51:09.704854+0000","last_peered":"2024-09-13T14:51:09.704854+0000","last_clean":"2024-09-13T14:51:09.704854+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:51:09.704854+0000","last_undegraded":"2024-09-13T14:51:09.704854+0000","last_fullsized":"2024-09-13T14:51:09.704854+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":144,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.674038+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:51:06.674038+0000","last_peered":"2024-09-13T14:51:06.674038+0000","last_clean":"2024-09-13T14:51:06.674038+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:51:06.674038+0000","last_undegraded":"2024-09-13T14:51:06.674038+0000","last_fullsized":"2024-09-13T14:51:06.674038+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":84,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.024461+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:50:54.024461+0000","last_peered":"2024-09-13T14:50:54.024461+0000","last_clean":"2024-09-13T14:50:54.024461+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:50:54.024461+0000","last_undegraded":"2024-09-13T14:50:54.024461+0000","last_fullsized":"2024-09-13T14:50:54.024461+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":281,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.664335+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:51:06.664335+0000","last_peered":"2024-09-13T14:51:06.664335+0000","last_clean":"2024-09-13T14:51:06.664335+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:51:06.664335+0000","last_undegraded":"2024-09-13T14:51:06.664335+0000","last_fullsized":"2024-09-13T14:51:06.664335+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":335,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669543+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:51:06.669543+0000","last_peered":"2024-09-13T14:51:06.669543+0000","last_clean":"2024-09-13T14:51:06.669543+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:51:06.669543+0000","last_undegraded":"2024-09-13T14:51:06.669543+0000","last_fullsized":"2024-09-13T14:51:06.669543+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":332,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.668344+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:51:06.668344+0000","last_peered":"2024-09-13T14:51:06.668344+0000","last_clean":"2024-09-13T14:51:06.668344+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:51:06.668344+0000","last_undegraded":"2024-09-13T14:51:06.668344+0000","last_fullsized":"2024-09-13T14:51:06.668344+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":113,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.037006+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:50:54.037006+0000","last_peered":"2024-09-13T14:50:54.037006+0000","last_clean":"2024-09-13T14:50:54.037006+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:50:54.037006+0000","last_undegraded":"2024-09-13T14:50:54.037006+0000","last_fullsized":"2024-09-13T14:50:54.037006+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":298,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.679240+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:51:06.679240+0000","last_peered":"2024-09-13T14:51:06.679240+0000","last_clean":"2024-09-13T14:51:06.679240+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:51:06.679240+0000","last_undegraded":"2024-09-13T14:51:06.679240+0000","last_fullsized":"2024-09-13T14:51:06.679240+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":315,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669303+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:51:06.669303+0000","last_peered":"2024-09-13T14:51:06.669303+0000","last_clean":"2024-09-13T14:51:06.669303+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:51:06.669303+0000","last_undegraded":"2024-09-13T14:51:06.669303+0000","last_fullsized":"2024-09-13T14:51:06.669303+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":286,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.707068+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:51:09.707068+0000","last_peered":"2024-09-13T14:51:09.707068+0000","last_clean":"2024-09-13T14:51:09.707068+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:51:09.707068+0000","last_undegraded":"2024-09-13T14:51:09.707068+0000","last_fullsized":"2024-09-13T14:51:09.707068+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":140,"reported_epoch":223,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T14:51:09.705619+0000","last_change":"2024-09-13T14:50:20.795913+0000","last_active":"2024-09-13T14:51:09.705619+0000","last_peered":"2024-09-13T14:51:09.705619+0000","last_clean":"2024-09-13T14:50:08.544047+0000","last_became_active":"2024-09-13T14:50:20.795534+0000","last_became_peered":"2024-09-13T14:50:20.795534+0000","last_unstale":"2024-09-13T14:51:09.705619+0000","last_undegraded":"2024-09-13T14:51:09.705619+0000","last_fullsized":"2024-09-13T14:51:09.705619+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[1,4,5],"avail_no_missing":["1","3","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":202,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018493+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:50:54.018493+0000","last_peered":"2024-09-13T14:50:54.018493+0000","last_clean":"2024-09-13T14:50:54.018493+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:50:54.018493+0000","last_undegraded":"2024-09-13T14:50:54.018493+0000","last_fullsized":"2024-09-13T14:50:54.018493+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":43,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.661978+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:51:06.661978+0000","last_peered":"2024-09-13T14:51:06.661978+0000","last_clean":"2024-09-13T14:51:06.661978+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:51:06.661978+0000","last_undegraded":"2024-09-13T14:51:06.661978+0000","last_fullsized":"2024-09-13T14:51:06.661978+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":308,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669832+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:51:06.669832+0000","last_peered":"2024-09-13T14:51:06.669832+0000","last_clean":"2024-09-13T14:51:06.669832+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:51:06.669832+0000","last_undegraded":"2024-09-13T14:51:06.669832+0000","last_fullsized":"2024-09-13T14:51:06.669832+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":203,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018181+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:50:54.018181+0000","last_peered":"2024-09-13T14:50:54.018181+0000","last_clean":"2024-09-13T14:50:54.018181+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:50:54.018181+0000","last_undegraded":"2024-09-13T14:50:54.018181+0000","last_fullsized":"2024-09-13T14:50:54.018181+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":59,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.577594+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:50:54.577594+0000","last_peered":"2024-09-13T14:50:54.577594+0000","last_clean":"2024-09-13T14:50:54.577594+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:50:54.577594+0000","last_undegraded":"2024-09-13T14:50:54.577594+0000","last_fullsized":"2024-09-13T14:50:54.577594+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":200,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.018425+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:50:54.018425+0000","last_peered":"2024-09-13T14:50:54.018425+0000","last_clean":"2024-09-13T14:50:54.018425+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:50:54.018425+0000","last_undegraded":"2024-09-13T14:50:54.018425+0000","last_fullsized":"2024-09-13T14:50:54.018425+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":61,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.577651+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:50:54.577651+0000","last_peered":"2024-09-13T14:50:54.577651+0000","last_clean":"2024-09-13T14:50:54.577651+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:50:54.577651+0000","last_undegraded":"2024-09-13T14:50:54.577651+0000","last_fullsized":"2024-09-13T14:50:54.577651+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":301,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.664685+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:51:06.664685+0000","last_peered":"2024-09-13T14:51:06.664685+0000","last_clean":"2024-09-13T14:51:06.664685+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:51:06.664685+0000","last_undegraded":"2024-09-13T14:51:06.664685+0000","last_fullsized":"2024-09-13T14:51:06.664685+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":318,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.664953+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:51:06.664953+0000","last_peered":"2024-09-13T14:51:06.664953+0000","last_clean":"2024-09-13T14:51:06.664953+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:51:06.664953+0000","last_undegraded":"2024-09-13T14:51:06.664953+0000","last_fullsized":"2024-09-13T14:51:06.664953+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":230,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-13T14:50:54.017711+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:50:54.017711+0000","last_peered":"2024-09-13T14:50:54.017711+0000","last_clean":"2024-09-13T14:50:54.017711+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:50:54.017711+0000","last_undegraded":"2024-09-13T14:50:54.017711+0000","last_fullsized":"2024-09-13T14:50:54.017711+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":371,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-13T14:51:06.669461+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:51:06.669461+0000","last_peered":"2024-09-13T14:51:06.669461+0000","last_clean":"2024-09-13T14:51:06.669461+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:51:06.669461+0000","last_undegraded":"2024-09-13T14:51:06.669461+0000","last_fullsized":"2024-09-13T14:51:06.669461+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5113774104,"num_objects":78031,"num_object_clones":0,"num_object_copies":234093,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":22510,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93517,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24704,"num_bytes_recovered":1597767680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16375033856,"data_stored":16375021640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92089,"ondisk_log_size":92089,"up":48,"acting":40,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234285,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495921,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2942536,"kb_used_data":2704704,"kb_used_omap":1,"kb_used_meta":237822,"kb_avail":90802616,"statfs":{"total":95995035648,"available":92981878784,"internally_reserved":0,"allocated":2769616896,"data_stored":2768385813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":243530183},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757556,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2682304,"kb_used_data":2510904,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91062848,"statfs":{"total":95995035648,"available":93248356352,"internally_reserved":0,"allocated":2571165696,"data_stored":2569938408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175503815},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019196,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851080,"kb_used_data":1559104,"kb_used_omap":1,"kb_used_meta":291966,"kb_avail":91894072,"statfs":{"total":95995035648,"available":94099529728,"internally_reserved":0,"allocated":1596522496,"data_stored":1595291474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":298973638},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468081,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3516092,"kb_used_data":3322036,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":90229060,"statfs":{"total":95995035648,"available":92394557440,"internally_reserved":0,"allocated":3401764864,"data_stored":3400541422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":198700196},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631610,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2408140,"kb_used_data":2105732,"kb_used_omap":8,"kb_used_meta":302391,"kb_avail":91337012,"statfs":{"total":95995035648,"available":93529100288,"internally_reserved":0,"allocated":2156269568,"data_stored":2155034976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":309649375},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795133,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2569220,"kb_used_data":2334844,"kb_used_omap":7,"kb_used_meta":234360,"kb_avail":91175932,"statfs":{"total":95995035648,"available":93364154368,"internally_reserved":0,"allocated":2390880256,"data_stored":2389649071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":239985266},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958663,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1683148,"kb_used_data":1394624,"kb_used_omap":7,"kb_used_meta":288504,"kb_avail":92062004,"statfs":{"total":95995035648,"available":94271492096,"internally_reserved":0,"allocated":1428094976,"data_stored":1426860348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":295428583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1420496896,"data_stored":1420492824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2386694144,"data_stored":2386690072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2146893824,"data_stored":2146893824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3389915136,"data_stored":3389915136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1593114624,"data_stored":1593114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2567901184,"data_stored":2567897112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2759065600,"data_stored":2759065600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":110952448,"data_stored":110952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:16.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:51:16.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:16.207 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:16.267 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:16.267 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:16.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.333 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:16.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:16.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:16.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:16.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:16.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:16.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:16.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:16.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:16.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.834 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:16.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:16.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:16.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:16.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:16.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[102314]: pgmap v649: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 20312/229839 objects degraded (8.837%); 1765/229839 objects misplaced (0.768%); 8.6 MiB/s, 156 objects/s recovering 2024-09-13T14:51:16.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:16.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:16.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[107038]: pgmap v649: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 20312/229839 objects degraded (8.837%); 1765/229839 objects misplaced (0.768%); 8.6 MiB/s, 156 objects/s recovering 2024-09-13T14:51:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:16.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:16 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:16 smithi172 ceph-mon[100696]: pgmap v649: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 373 op/s; 20312/229839 objects degraded (8.837%); 1765/229839 objects misplaced (0.768%); 8.6 MiB/s, 156 objects/s recovering 2024-09-13T14:51:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:16 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:51:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:16 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:51:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:16 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:17.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:51:17.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:17.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:17.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:17.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:17.312 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:17.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:17.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:17.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:17.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:17.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:17.608 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:17.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:17.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:17.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.869 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:17.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:17.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:17.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:17.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:18.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:18.120 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:18.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:18.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:18.270 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:18.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:18.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:18.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:18.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:18.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:18.637 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:18.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.764 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:18.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:18.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:18.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:18.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:18.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:18 smithi154 ceph-mon[102314]: pgmap v650: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 286 op/s; 20312/229839 objects degraded (8.837%); 1765/229839 objects misplaced (0.768%); 8.6 MiB/s, 156 objects/s recovering 2024-09-13T14:51:18.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:18 smithi154 ceph-mon[107038]: pgmap v650: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 286 op/s; 20312/229839 objects degraded (8.837%); 1765/229839 objects misplaced (0.768%); 8.6 MiB/s, 156 objects/s recovering 2024-09-13T14:51:18.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:18.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:18 smithi172 ceph-mon[100696]: pgmap v650: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 286 op/s; 20312/229839 objects degraded (8.837%); 1765/229839 objects misplaced (0.768%); 8.6 MiB/s, 156 objects/s recovering 2024-09-13T14:51:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:19.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:19.052 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:19.052 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:19.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.157 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:19.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:19.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:19.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:19.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:19.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:19.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:19.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:19.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:19.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:19.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:19.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:19.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:51:19.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:19.925 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:19.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:20.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.101 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:20.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:20.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:20.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:20.311 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:20.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:20.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:20.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:20.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:20.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:20.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:20.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:20.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:20 smithi172 ceph-mon[100696]: pgmap v651: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 251 op/s; 20312/227817 objects degraded (8.916%); 1262/227817 objects misplaced (0.554%); 11 MiB/s, 189 objects/s recovering 2024-09-13T14:51:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 20312/227817 objects degraded (8.916%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:21.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:21.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:21.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:21.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:21.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:20 smithi154 ceph-mon[102314]: pgmap v651: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 251 op/s; 20312/227817 objects degraded (8.916%); 1262/227817 objects misplaced (0.554%); 11 MiB/s, 189 objects/s recovering 2024-09-13T14:51:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 20312/227817 objects degraded (8.916%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:21.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:20 smithi154 ceph-mon[107038]: pgmap v651: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 251 op/s; 20312/227817 objects degraded (8.916%); 1262/227817 objects misplaced (0.554%); 11 MiB/s, 189 objects/s recovering 2024-09-13T14:51:21.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 20312/227817 objects degraded (8.916%), 7 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:21.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.369 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:21.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:21.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.505 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:21.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:21.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:21.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:21.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:21.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:21.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:21.877 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:21.877 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:21.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:21.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:21.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:22.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:22.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:22.139 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:51:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:51:21] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-13T14:51:22.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:22.263 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:22.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:22.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:22.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:22.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:22.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:51:22.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.655 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:22.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:22.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:22.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:22.799 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:22.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:22.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:22.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:22.922 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:22 smithi172 ceph-mon[100696]: pgmap v652: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 299 op/s; 20312/224955 objects degraded (9.029%); 1262/224955 objects misplaced (0.561%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:51:22.944 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:22 smithi154 ceph-mon[102314]: pgmap v652: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 299 op/s; 20312/224955 objects degraded (9.029%); 1262/224955 objects misplaced (0.561%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:51:22.944 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:22 smithi154 ceph-mon[107038]: pgmap v652: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 299 op/s; 20312/224955 objects degraded (9.029%); 1262/224955 objects misplaced (0.561%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:51:23.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:23.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:23.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:23.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:23.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:23.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:23.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:23.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:23.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:23.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:23.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:23.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:23.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:23.812 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:23.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:23.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:23.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:24.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:24.062 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:24.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.185 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:24.278 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:51:24.279 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:51:24.279 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:24.279 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:24.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:24.305 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:51:24.305 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:24.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:24.338 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:51:24.339 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:24.365 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:51:24.365 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:24.388 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:51:24.388 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:24.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:24.420 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:51:24.420 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.437 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:24.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:24.455 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:51:24.455 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:24.527 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:51:24.527 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:51:24.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:51:24.527 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:51:24.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.528 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41401 6.5 6:a0cee5c3:::benchmark_data_smithi154_175640_object41400:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.528 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.154326+0000", 2024-09-13T14:51:24.528 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.369529599, 2024-09-13T14:51:24.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.987267173, 2024-09-13T14:51:24.529 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.529 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.530 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.530 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41401 2024-09-13T14:51:24.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.530 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154326+0000", 2024-09-13T14:51:24.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154326+0000", 2024-09-13T14:51:24.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154325+0000", 2024-09-13T14:51:24.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154365+0000", 2024-09-13T14:51:24.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9858999999999999e-05 2024-09-13T14:51:24.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154373+0000", 2024-09-13T14:51:24.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5169999999999999e-06 2024-09-13T14:51:24.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154377+0000", 2024-09-13T14:51:24.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.386e-06 2024-09-13T14:51:24.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154429+0000", 2024-09-13T14:51:24.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1792999999999997e-05 2024-09-13T14:51:24.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154440+0000", 2024-09-13T14:51:24.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1009000000000001e-05 2024-09-13T14:51:24.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.136835+0000", 2024-09-13T14:51:24.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98239497799999997 2024-09-13T14:51:24.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.136984+0000", 2024-09-13T14:51:24.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149166 2024-09-13T14:51:24.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.137156+0000", 2024-09-13T14:51:24.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171724 2024-09-13T14:51:24.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141391+0000", 2024-09-13T14:51:24.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042347770000000003 2024-09-13T14:51:24.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141397+0000", 2024-09-13T14:51:24.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3180000000000003e-06 2024-09-13T14:51:24.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141508+0000", 2024-09-13T14:51:24.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110745 2024-09-13T14:51:24.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141555+0000", 2024-09-13T14:51:24.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6873000000000003e-05 2024-09-13T14:51:24.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141593+0000", 2024-09-13T14:51:24.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8525000000000001e-05 2024-09-13T14:51:24.545 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.545 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.546 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41409 6.5 6:a7afbe70:::benchmark_data_smithi154_175640_object41408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.546 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.154474+0000", 2024-09-13T14:51:24.546 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.36938194300001, 2024-09-13T14:51:24.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98730288099999997, 2024-09-13T14:51:24.547 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.547 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.548 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41409 2024-09-13T14:51:24.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.549 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154474+0000", 2024-09-13T14:51:24.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154474+0000", 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154473+0000", 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154547+0000", 2024-09-13T14:51:24.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4348e-05 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154560+0000", 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2741e-05 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154567+0000", 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.601e-06 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154613+0000", 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6465999999999999e-05 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.154626+0000", 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2619999999999999e-05 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.138993+0000", 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98436692299999995 2024-09-13T14:51:24.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.139102+0000", 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109354 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.139209+0000", 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107131 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141630+0000", 2024-09-13T14:51:24.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024207909999999998 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141637+0000", 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6819999999999997e-06 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141695+0000", 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8863999999999997e-05 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141740+0000", 2024-09-13T14:51:24.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4885999999999997e-05 2024-09-13T14:51:24.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141777+0000", 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6353000000000002e-05 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.558 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41423 6.5 6:adf64e52:::benchmark_data_smithi154_175640_object41422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.170481+0000", 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.353374454, 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97225323200000002, 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41423 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170481+0000", 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170481+0000", 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170480+0000", 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170552+0000", 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1755999999999995e-05 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170564+0000", 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2017e-05 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170572+0000", 2024-09-13T14:51:24.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2770000000000005e-06 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170631+0000", 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9401999999999998e-05 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.170643+0000", 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.188e-05 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.140287+0000", 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96964348 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.140399+0000", 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243600000000001 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.140552+0000", 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000152318 2024-09-13T14:51:24.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142583+0000", 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020316010000000001 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142620+0000", 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6758000000000003e-05 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142667+0000", 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7546999999999997e-05 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142714+0000", 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6428999999999999e-05 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142735+0000", 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0611e-05 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.565 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41430 6.5 6:a33d844f:::benchmark_data_smithi154_175640_object41429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.176451+0000", 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.34740474899999, 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98017701400000001, 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41430 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176451+0000", 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176451+0000", 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176450+0000", 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176568+0000", 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000118829 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176585+0000", 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6586e-05 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176592+0000", 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4610000000000002e-06 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176720+0000", 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012720900000000001 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.176730+0000", 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-05 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141823+0000", 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96509276600000005 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141911+0000", 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8411000000000004e-05 2024-09-13T14:51:24.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.141991+0000", 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0037999999999995e-05 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144236+0000", 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022446940000000002 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154118+0000", 2024-09-13T14:51:24.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098819530000000006 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156564+0000", 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002446181 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156602+0000", 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8198000000000003e-05 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156628+0000", 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5703000000000001e-05 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41449 6.5 6:ab6f9780:::benchmark_data_smithi154_175640_object41448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.192954+0000", 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.33090135699999, 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96378176500000001, 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.573 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41449 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192954+0000", 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192954+0000", 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192953+0000", 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.192996+0000", 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3519000000000001e-05 2024-09-13T14:51:24.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193008+0000", 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1378e-05 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193014+0000", 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0850000000000002e-06 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193069+0000", 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5018000000000001e-05 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.193080+0000", 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1386000000000001e-05 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142763+0000", 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94968323099999996 2024-09-13T14:51:24.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142830+0000", 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7049000000000001e-05 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.142908+0000", 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7990000000000004e-05 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146064+0000", 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031560289999999999 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154178+0000", 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081134980000000002 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156666+0000", 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024880240000000001 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156716+0000", 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9897e-05 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156736+0000", 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0267000000000001e-05 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.580 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41467 6.5 6:aa02d758:::benchmark_data_smithi154_175640_object41466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.208447+0000", 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.31540846499999, 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.948363972, 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41467 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208447+0000", 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208447+0000", 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208445+0000", 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208491+0000", 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5204999999999998e-05 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208510+0000", 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9436000000000002e-05 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208520+0000", 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0275000000000001e-05 2024-09-13T14:51:24.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208572+0000", 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1277999999999999e-05 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.208579+0000", 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5159999999999997e-06 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.143489+0000", 2024-09-13T14:51:24.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.93491021900000004 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.143594+0000", 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010509999999999999 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.143678+0000", 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3837000000000004e-05 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147708+0000", 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040294559999999998 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154210+0000", 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065019739999999998 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156767+0000", 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025577299999999998 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156793+0000", 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5380000000000001e-05 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156811+0000", 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8502e-05 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.587 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41494 6.5 6:a01b1af7:::benchmark_data_smithi154_175640_object41493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.224013+0000", 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.299843154, 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.93286539300000004, 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41494 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224013+0000", 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224013+0000", 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224012+0000", 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224087+0000", 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5371999999999996e-05 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224096+0000", 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1260000000000004e-06 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224101+0000", 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.899e-06 2024-09-13T14:51:24.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224165+0000", 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3408e-05 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.224176+0000", 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1522000000000001e-05 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144255+0000", 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92007880099999995 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144322+0000", 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6680000000000005e-05 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.144399+0000", 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6947999999999996e-05 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.147713+0000", 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003314053 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.154258+0000", 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065451260000000001 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156839+0000", 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025813770000000002 2024-09-13T14:51:24.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156861+0000", 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2056e-05 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156878+0000", 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6735000000000002e-05 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41499 6.5 6:a6ba2bb0:::benchmark_data_smithi154_175640_object41498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.228394+0000", 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.29546181399999, 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.930512809, 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41499 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228394+0000", 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228394+0000", 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228392+0000", 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228458+0000", 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.635e-05 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228469+0000", 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1146e-05 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228476+0000", 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9789999999999996e-06 2024-09-13T14:51:24.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228531+0000", 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4364999999999999e-05 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.228538+0000", 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9190000000000002e-06 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.145038+0000", 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91650072000000005 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.145153+0000", 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114875 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.145286+0000", 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000133063 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.149861+0000", 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045745200000000003 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156905+0000", 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070438710000000002 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158835+0000", 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019305539999999999 2024-09-13T14:51:24.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158871+0000", 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5491999999999998e-05 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158907+0000", 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6086e-05 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:41505 6.5 6:a4d4b87f:::benchmark_data_smithi154_175640_object41504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:27.234857+0000", 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 177.28899836900001, 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92416932399999996, 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 41505 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234857+0000", 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234857+0000", 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234856+0000", 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234914+0000", 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8418999999999998e-05 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234927+0000", 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2394e-05 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234933+0000", 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1040000000000003e-06 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234982+0000", 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8931999999999998e-05 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:27.234991+0000", 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8680000000000005e-06 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146100+0000", 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91110977800000004 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146177+0000", 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6884000000000005e-05 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.146267+0000", 2024-09-13T14:51:24.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9871999999999996e-05 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.149867+0000", 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035993420000000002 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.156923+0000", 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070560579999999996 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.158959+0000", 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020363640000000001 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.159005+0000", 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6079999999999999e-05 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:28.159027+0000", 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1668999999999999e-05 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62865 6.3 6:ca841005:::benchmark_data_smithi154_175640_object62864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.046847+0000", 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.47700887900001, 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.014929846, 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.609 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62865 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046847+0000", 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046847+0000", 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046839+0000", 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999924 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046883+0000", 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4147999999999998e-05 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046898+0000", 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5157000000000001e-05 2024-09-13T14:51:24.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046906+0000", 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4050000000000003e-06 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046969+0000", 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3634999999999996e-05 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.046982+0000", 2024-09-13T14:51:24.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2079000000000001e-05 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.054841+0000", 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0078592609999999 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.054988+0000", 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147298 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.055140+0000", 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015169099999999999 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.059329+0000", 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041888409999999996 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061633+0000", 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002304352 2024-09-13T14:51:24.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061689+0000", 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5497000000000001e-05 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061736+0000", 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.706e-05 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061777+0000", 2024-09-13T14:51:24.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1149999999999997e-05 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62871 6.3 6:ced36ccb:::benchmark_data_smithi154_175640_object62870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.062936+0000", 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.46092016700001, 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0066727499999999, 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62871 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.062936+0000", 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.062936+0000", 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.062933+0000", 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999971 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.062982+0000", 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9258000000000002e-05 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.063009+0000", 2024-09-13T14:51:24.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7041000000000001e-05 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.063026+0000", 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7037e-05 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.063100+0000", 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4040000000000003e-05 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.063113+0000", 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3132999999999999e-05 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.056931+0000", 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99381736300000001 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.057030+0000", 2024-09-13T14:51:24.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9046e-05 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.057123+0000", 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3412999999999995e-05 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061591+0000", 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044675829999999998 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067234+0000", 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056434600000000003 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069536+0000", 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002301244 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069577+0000", 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1607999999999999e-05 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069608+0000", 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1171999999999999e-05 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62872 6.3 6:cc342b2b:::benchmark_data_smithi154_175640_object62871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.064305+0000", 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.45955054000001, 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.005412017, 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62872 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064305+0000", 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064305+0000", 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064303+0000", 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064346+0000", 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2871999999999999e-05 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064368+0000", 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1780000000000002e-05 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064377+0000", 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2289999999999992e-06 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064442+0000", 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5096000000000002e-05 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.064454+0000", 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1952e-05 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.058168+0000", 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99371407099999998 2024-09-13T14:51:24.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.058275+0000", 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106986 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.058415+0000", 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139603 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061789+0000", 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033740900000000002 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067286+0000", 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054964929999999999 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069655+0000", 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023689969999999999 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069695+0000", 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0645000000000001e-05 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069717+0000", 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2070999999999999e-05 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62877 6.3 6:c884fe0b:::benchmark_data_smithi154_175640_object62876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.072239+0000", 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.45161657099999, 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.007899061, 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62877 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072239+0000", 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072239+0000", 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072238+0000", 2024-09-13T14:51:24.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072295+0000", 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7420999999999998e-05 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072312+0000", 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6937999999999999e-05 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072320+0000", 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9209999999999992e-06 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072409+0000", 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9548000000000004e-05 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.072420+0000", 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0723000000000001e-05 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.059371+0000", 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98695124700000003 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.059465+0000", 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3445999999999999e-05 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.059543+0000", 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8615000000000005e-05 2024-09-13T14:51:24.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061795+0000", 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022511609999999998 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.072822+0000", 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011027444000000001 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.079955+0000", 2024-09-13T14:51:24.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071327830000000002 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.080097+0000", 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014226500000000001 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.080138+0000", 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1223999999999997e-05 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62893 6.3 6:cab6a2ce:::benchmark_data_smithi154_175640_object62892:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.089433+0000", 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.43442272499999, 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99293413699999999, 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62893 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.638 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089433+0000", 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089433+0000", 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089431+0000", 2024-09-13T14:51:24.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089519+0000", 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7792000000000002e-05 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089538+0000", 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8440999999999998e-05 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089546+0000", 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9889999999999992e-06 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089608+0000", 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1834000000000005e-05 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.089617+0000", 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0149999999999992e-06 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.060407+0000", 2024-09-13T14:51:24.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97079033199999998 2024-09-13T14:51:24.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.060517+0000", 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010979700000000001 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.060652+0000", 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013496 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.063923+0000", 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032715510000000001 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.072864+0000", 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0089403910000000007 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082264+0000", 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094001039999999994 2024-09-13T14:51:24.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082316+0000", 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1901999999999999e-05 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082367+0000", 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1604000000000003e-05 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62899 6.3 6:c7e54ea3:::benchmark_data_smithi154_175640_object62898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.095956+0000", 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.42789942900001, 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.986649051, 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62899 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.095956+0000", 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.095956+0000", 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.095954+0000", 2024-09-13T14:51:24.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.096002+0000", 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7728000000000001e-05 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.096017+0000", 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4473e-05 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.096023+0000", 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7959999999999998e-06 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.096088+0000", 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4220999999999994e-05 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.096098+0000", 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0526999999999999e-05 2024-09-13T14:51:24.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061815+0000", 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96571712099999996 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061887+0000", 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1754000000000005e-05 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.061966+0000", 2024-09-13T14:51:24.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8986999999999994e-05 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067181+0000", 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052155320000000002 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.072900+0000", 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057183609999999999 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082499+0000", 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095995649999999991 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082541+0000", 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2083999999999999e-05 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082605+0000", 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3923000000000004e-05 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.653 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62902 6.3 6:c11a8100:::benchmark_data_smithi154_175640_object62901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.098795+0000", 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.425060667, 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98393905100000001, 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62902 2024-09-13T14:51:24.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098795+0000", 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098795+0000", 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098794+0000", 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098854+0000", 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9803999999999999e-05 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098866+0000", 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2146e-05 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098872+0000", 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1829999999999999e-06 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098917+0000", 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5429000000000002e-05 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.098926+0000", 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3299999999999999e-06 2024-09-13T14:51:24.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.062935+0000", 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96400922300000003 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.063025+0000", 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0626999999999998e-05 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.063104+0000", 2024-09-13T14:51:24.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8564000000000002e-05 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067294+0000", 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041895140000000001 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082422+0000", 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015128442000000001 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082642+0000", 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021978500000000001 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082693+0000", 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1365e-05 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082734+0000", 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0998000000000003e-05 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.660 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62904 6.3 6:c8496af8:::benchmark_data_smithi154_175640_object62903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.100125+0000", 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.42373076999999, 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98273145900000003, 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.661 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62904 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100125+0000", 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100125+0000", 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100124+0000", 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100201+0000", 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7108999999999996e-05 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100211+0000", 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0135e-05 2024-09-13T14:51:24.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100216+0000", 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7650000000000001e-06 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100253+0000", 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6414000000000001e-05 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.100260+0000", 2024-09-13T14:51:24.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1969999999999999e-06 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.063948+0000", 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96368834800000003 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.064056+0000", 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107983 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.064135+0000", 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9196999999999994e-05 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067299+0000", 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031639289999999998 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082463+0000", 2024-09-13T14:51:24.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015163964 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082774+0000", 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031035500000000001 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082824+0000", 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0720000000000002e-05 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082856+0000", 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2125999999999997e-05 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62928 6.3 6:c54a0c1e:::benchmark_data_smithi154_175640_object62927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.130175+0000", 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.39368072600001, 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96209969399999995, 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.668 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62928 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130175+0000", 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130175+0000", 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130173+0000", 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130226+0000", 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2617000000000001e-05 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130245+0000", 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9230999999999999e-05 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130253+0000", 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2609999999999993e-06 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130351+0000", 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7882999999999998e-05 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.130363+0000", 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1858999999999999e-05 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.065015+0000", 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.934651542 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.065111+0000", 2024-09-13T14:51:24.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6065999999999998e-05 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.065192+0000", 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0968000000000001e-05 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067304+0000", 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021126510000000001 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082898+0000", 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015593345999999999 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092136+0000", 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009238319 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092200+0000", 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3757000000000006e-05 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092275+0000", 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4882000000000006e-05 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62942 6.3 6:c6401071:::benchmark_data_smithi154_175640_object62941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.146785+0000", 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.37707086399999, 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94559474200000004, 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.675 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62942 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146785+0000", 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146785+0000", 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146784+0000", 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146838+0000", 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4437000000000001e-05 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146846+0000", 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9719999999999992e-06 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146850+0000", 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9500000000000003e-06 2024-09-13T14:51:24.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146904+0000", 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4351999999999997e-05 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.146918+0000", 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3209e-05 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.066120+0000", 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91920194499999996 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.066207+0000", 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7795999999999998e-05 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.066298+0000", 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0167000000000006e-05 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069488+0000", 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031904450000000001 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082941+0000", 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013453316 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092317+0000", 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093758020000000008 2024-09-13T14:51:24.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092347+0000", 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9694999999999999e-05 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092380+0000", 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2784000000000003e-05 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.683 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:62951 6.3 6:c13a6f7a:::benchmark_data_smithi154_175640_object62950:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-13T14:51:24.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:48:55.170342+0000", 2024-09-13T14:51:24.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.35351403999999, 2024-09-13T14:51:24.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92212992100000002, 2024-09-13T14:51:24.683 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:24.683 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 62951 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:24.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170342+0000", 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170342+0000", 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170340+0000", 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170408+0000", 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8359000000000001e-05 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170429+0000", 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0503e-05 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170439+0000", 2024-09-13T14:51:24.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6860000000000001e-06 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170497+0000", 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8485e-05 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:55.170508+0000", 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1161e-05 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067322+0000", 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.896814204 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067393+0000", 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0482999999999996e-05 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.067472+0000", 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8557e-05 2024-09-13T14:51:24.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.069726+0000", 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002254327 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.082984+0000", 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013258244000000001 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092413+0000", 2024-09-13T14:51:24.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094287639999999992 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092442+0000", 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9065999999999999e-05 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:48:56.092472+0000", 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9728000000000001e-05 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:24.691 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:24.691 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:51:24.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:24.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:24.765 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:51:24.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:24.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[102314]: pgmap v653: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 20312/224955 objects degraded (9.029%); 1262/224955 objects misplaced (0.561%); 4.4 MiB/s, 71 objects/s recovering 2024-09-13T14:51:24.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1308310393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:24.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1197577782' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:24.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:24.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[107038]: pgmap v653: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 20312/224955 objects degraded (9.029%); 1262/224955 objects misplaced (0.561%); 4.4 MiB/s, 71 objects/s recovering 2024-09-13T14:51:24.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1308310393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:24.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1197577782' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:24.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:24 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:24.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:24.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:24.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:24.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:24.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:24 smithi172 ceph-mon[100696]: pgmap v653: 89 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 20312/224955 objects degraded (9.029%); 1262/224955 objects misplaced (0.561%); 4.4 MiB/s, 71 objects/s recovering 2024-09-13T14:51:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1308310393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1197577782' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:24.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:24 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:51:25.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20312/224955 objects degraded (9.029%), 7 pgs degraded, 5 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":900,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":225,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238500,"num_in_osds":7,"osd_in_since":1726238900,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":80},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+remapped+backfill_wait","count":1}],"num_pgs":89,"num_pools":5,"num_objects":74985,"data_bytes":4917106685,"bytes_used":17424191488,"bytes_avail":654541058048,"bytes_total":671965249536,"degraded_objects":20312,"degraded_total":224955,"degraded_ratio":0.090293614278411247,"misplaced_objects":1262,"misplaced_total":224955,"misplaced_ratio":0.0056100108910671024,"recovering_objects_per_sec":71,"recovering_bytes_per_sec":4656860,"recovering_keys_per_sec":0,"num_objects_recovered":853,"num_bytes_recovered":55902208,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":256},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi154.daekzd","status":"up:active","gid":24539}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"}},"servicemap":{"epoch":25,"modified":"2024-09-13T14:50:26.951855+0000","services":{}},"progress_events":{}} 2024-09-13T14:51:25.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:25.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:25.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:25.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:25.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:25.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:51:25.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:25.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:25.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:25.928 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:25.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:25.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:25.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:25 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:25 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:25 smithi172 ceph-mon[100696]: osdmap e225: 8 total, 8 up, 7 in 2024-09-13T14:51:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 20312/224955 objects degraded (9.029%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:25 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3827387607' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:25.999 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:51:25.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:26.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:26.070 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:26.071 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:26.071 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[102314]: osdmap e225: 8 total, 8 up, 7 in 2024-09-13T14:51:26.071 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 20312/224955 objects degraded (9.029%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:26.071 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3827387607' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:26.071 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:26.071 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:51:26.071 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[107038]: osdmap e225: 8 total, 8 up, 7 in 2024-09-13T14:51:26.072 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 20312/224955 objects degraded (9.029%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:26.072 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:25 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3827387607' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:26.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:26.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:26.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:26.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:26.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:26.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:26.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:26.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:26.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:26.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:26.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:26.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:26.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:26.962 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:26.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:26.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:27.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:27.092 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:26 smithi172 ceph-mon[100696]: pgmap v655: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 20312/220203 objects degraded (9.224%); 671/220203 objects misplaced (0.305%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:51:27.112 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:27.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:26 smithi154 ceph-mon[102314]: pgmap v655: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 20312/220203 objects degraded (9.224%); 671/220203 objects misplaced (0.305%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:51:27.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:26 smithi154 ceph-mon[107038]: pgmap v655: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 20312/220203 objects degraded (9.224%); 671/220203 objects misplaced (0.305%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:51:27.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:27.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:27.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:27.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:27.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:27.507 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:27.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:27.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:27.632 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:51:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:51:27.632 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T14:51:27.632 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:51:27.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:27.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:27.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:27.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:27.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:27.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:27.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:28.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:28.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.127 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:28.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:28.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.251 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:28.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:28.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.378 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:28.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:28.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:51:28.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:28.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:28.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:28.631 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:28.665 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:28.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:28.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:28.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:51:28.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:28.884 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:28.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:28.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:28 smithi154 ceph-mon[102314]: pgmap v656: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 20312/220203 objects degraded (9.224%); 671/220203 objects misplaced (0.305%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:51:28.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:28 smithi154 ceph-mon[102314]: osdmap e226: 8 total, 8 up, 7 in 2024-09-13T14:51:28.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:28.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:28 smithi154 ceph-mon[107038]: pgmap v656: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 20312/220203 objects degraded (9.224%); 671/220203 objects misplaced (0.305%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:51:28.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:28 smithi154 ceph-mon[107038]: osdmap e226: 8 total, 8 up, 7 in 2024-09-13T14:51:28.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:28.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:29.038 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:28 smithi172 ceph-mon[100696]: pgmap v656: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 20312/220203 objects degraded (9.224%); 671/220203 objects misplaced (0.305%); 6.8 MiB/s, 109 objects/s recovering 2024-09-13T14:51:29.038 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:28 smithi172 ceph-mon[100696]: osdmap e226: 8 total, 8 up, 7 in 2024-09-13T14:51:29.038 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:29.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:29.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:29.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:29.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:29.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:29.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:29.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:29.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:29.550 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:29.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:29.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:29.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:29.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:29.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:29.968 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:29.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:30.038 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:29 smithi154 ceph-mon[102314]: osdmap e227: 8 total, 8 up, 7 in 2024-09-13T14:51:30.038 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:29 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 20312/220203 objects degraded (9.224%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:30.039 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:29 smithi154 ceph-mon[107038]: osdmap e227: 8 total, 8 up, 7 in 2024-09-13T14:51:30.039 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:29 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 20312/220203 objects degraded (9.224%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:30.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:30.092 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:30.150 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:29 smithi172 ceph-mon[100696]: osdmap e227: 8 total, 8 up, 7 in 2024-09-13T14:51:30.150 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:29 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 20312/220203 objects degraded (9.224%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:30.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.235 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:30.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:30.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:30.354 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:51:30.358 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:51:30.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:30.426 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:51:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:51:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:51:30.426 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:51:30.427 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:51:30.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:30.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:30.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:30.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:30.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:30.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:30.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:30.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:30.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:30.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:31.012 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:31.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:31.120 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:30 smithi172 ceph-mon[100696]: pgmap v659: 89 pgs: 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 20312/218283 objects degraded (9.305%); 13 MiB/s, 224 objects/s recovering 2024-09-13T14:51:31.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:31.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:30 smithi154 ceph-mon[102314]: pgmap v659: 89 pgs: 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 20312/218283 objects degraded (9.305%); 13 MiB/s, 224 objects/s recovering 2024-09-13T14:51:31.141 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:30 smithi154 ceph-mon[107038]: pgmap v659: 89 pgs: 2 peering, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 20312/218283 objects degraded (9.305%); 13 MiB/s, 224 objects/s recovering 2024-09-13T14:51:31.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:51:31.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:31.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:31.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:31.372 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:51:31.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:31.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:31.517 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-09-13T14:51:28.971647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4771354621,"num_objects":72761,"num_object_clones":0,"num_object_copies":218283,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":20312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72761,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":99938,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27058,"num_bytes_recovered":1738755205,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98569,"ondisk_log_size":98569,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":295,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16602276,"kb_used_data":14863520,"kb_used_omap":32,"kb_used_meta":1738655,"kb_avail":639613788,"statfs":{"total":671965249536,"available":654964518912,"internally_reserved":0,"allocated":15220244480,"data_stored":15211485955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1780383280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2,1,2,2,8,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-145752064,"num_objects":-2224,"num_object_clones":0,"num_object_copies":-6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1262,"num_objects_unfound":0,"num_objects_dirty":-2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1345,"num_bytes_recovered":82706432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001446"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":333,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.901470+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:51:27.901470+0000","last_peered":"2024-09-13T14:51:27.901470+0000","last_clean":"2024-09-13T14:51:27.901470+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:51:27.901470+0000","last_undegraded":"2024-09-13T14:51:27.901470+0000","last_fullsized":"2024-09-13T14:51:27.901470+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":89,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700871+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:51:09.700871+0000","last_peered":"2024-09-13T14:51:09.700871+0000","last_clean":"2024-09-13T14:51:09.700871+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:51:09.700871+0000","last_undegraded":"2024-09-13T14:51:09.700871+0000","last_fullsized":"2024-09-13T14:51:09.700871+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":59,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.713302+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:51:09.713302+0000","last_peered":"2024-09-13T14:51:09.713302+0000","last_clean":"2024-09-13T14:51:09.713302+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:51:09.713302+0000","last_undegraded":"2024-09-13T14:51:09.713302+0000","last_fullsized":"2024-09-13T14:51:09.713302+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":322,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.850615+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:51:27.850615+0000","last_peered":"2024-09-13T14:51:27.850615+0000","last_clean":"2024-09-13T14:51:27.850615+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:51:27.850615+0000","last_undegraded":"2024-09-13T14:51:27.850615+0000","last_fullsized":"2024-09-13T14:51:27.850615+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":322,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.851753+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:51:27.851753+0000","last_peered":"2024-09-13T14:51:27.851753+0000","last_clean":"2024-09-13T14:51:27.851753+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:51:27.851753+0000","last_undegraded":"2024-09-13T14:51:27.851753+0000","last_fullsized":"2024-09-13T14:51:27.851753+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":159,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700636+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:51:09.700636+0000","last_peered":"2024-09-13T14:51:09.700636+0000","last_clean":"2024-09-13T14:51:09.700636+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:51:09.700636+0000","last_undegraded":"2024-09-13T14:51:09.700636+0000","last_fullsized":"2024-09-13T14:51:09.700636+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":48,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.850348+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:51:27.850348+0000","last_peered":"2024-09-13T14:51:27.850348+0000","last_clean":"2024-09-13T14:51:27.850348+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:51:27.850348+0000","last_undegraded":"2024-09-13T14:51:27.850348+0000","last_fullsized":"2024-09-13T14:51:27.850348+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":138,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.792542+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:51:27.792542+0000","last_peered":"2024-09-13T14:51:27.792542+0000","last_clean":"2024-09-13T14:51:27.792542+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:51:27.792542+0000","last_undegraded":"2024-09-13T14:51:27.792542+0000","last_fullsized":"2024-09-13T14:51:27.792542+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"224'6251","reported_seq":7068,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:24.539039+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:51:24.539039+0000","last_peered":"2024-09-13T14:51:24.539039+0000","last_clean":"2024-09-13T14:51:24.539039+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:51:24.539039+0000","last_undegraded":"2024-09-13T14:51:24.539039+0000","last_fullsized":"2024-09-13T14:51:24.539039+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304414720,"num_objects":4645,"num_object_clones":0,"num_object_copies":13935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":253,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.718744+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:51:10.718744+0000","last_peered":"2024-09-13T14:51:10.718744+0000","last_clean":"2024-09-13T14:51:10.718744+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:51:10.718744+0000","last_undegraded":"2024-09-13T14:51:10.718744+0000","last_fullsized":"2024-09-13T14:51:10.718744+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":86,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.712630+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:51:09.712630+0000","last_peered":"2024-09-13T14:51:09.712630+0000","last_clean":"2024-09-13T14:51:09.712630+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:51:09.712630+0000","last_undegraded":"2024-09-13T14:51:09.712630+0000","last_fullsized":"2024-09-13T14:51:09.712630+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"226'6121","reported_seq":7701,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.615742+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:51:28.615742+0000","last_peered":"2024-09-13T14:51:28.615742+0000","last_clean":"2024-09-13T14:51:28.615742+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:51:28.615742+0000","last_undegraded":"2024-09-13T14:51:28.615742+0000","last_fullsized":"2024-09-13T14:51:28.615742+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292093952,"num_objects":4457,"num_object_clones":0,"num_object_copies":13371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":87,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.701458+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:51:09.701458+0000","last_peered":"2024-09-13T14:51:09.701458+0000","last_clean":"2024-09-13T14:51:09.701458+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:51:09.701458+0000","last_undegraded":"2024-09-13T14:51:09.701458+0000","last_fullsized":"2024-09-13T14:51:09.701458+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":239,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700782+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:51:09.700782+0000","last_peered":"2024-09-13T14:51:09.700782+0000","last_clean":"2024-09-13T14:51:09.700782+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:51:09.700782+0000","last_undegraded":"2024-09-13T14:51:09.700782+0000","last_fullsized":"2024-09-13T14:51:09.700782+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"224'6226","reported_seq":7895,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:24.181284+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:51:24.181284+0000","last_peered":"2024-09-13T14:51:24.181284+0000","last_clean":"2024-09-13T14:51:24.181284+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:51:24.181284+0000","last_undegraded":"2024-09-13T14:51:24.181284+0000","last_fullsized":"2024-09-13T14:51:24.181284+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6226,"log_dups_size":0,"ondisk_log_size":6226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302645248,"num_objects":4618,"num_object_clones":0,"num_object_copies":13854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":63,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.713351+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:51:09.713351+0000","last_peered":"2024-09-13T14:51:09.713351+0000","last_clean":"2024-09-13T14:51:09.713351+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:51:09.713351+0000","last_undegraded":"2024-09-13T14:51:09.713351+0000","last_fullsized":"2024-09-13T14:51:09.713351+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793009+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:51:27.793009+0000","last_peered":"2024-09-13T14:51:27.793009+0000","last_clean":"2024-09-13T14:51:27.793009+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:51:27.793009+0000","last_undegraded":"2024-09-13T14:51:27.793009+0000","last_fullsized":"2024-09-13T14:51:27.793009+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"226'6072","reported_seq":7805,"reported_epoch":226,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:27.925448+0000","last_change":"2024-09-13T14:50:07.839326+0000","last_active":"2024-09-13T14:51:27.925448+0000","last_peered":"2024-09-13T14:51:27.925448+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:51:27.925448+0000","last_undegraded":"2024-09-13T14:50:07.570096+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6072,"log_dups_size":0,"ondisk_log_size":6072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290717696,"num_objects":4436,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6072,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":51445760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":2335},{"shards":"0,3,6","objects":2101}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":75,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.713080+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:51:09.713080+0000","last_peered":"2024-09-13T14:51:09.713080+0000","last_clean":"2024-09-13T14:51:09.713080+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:51:09.713080+0000","last_undegraded":"2024-09-13T14:51:09.713080+0000","last_fullsized":"2024-09-13T14:51:09.713080+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":302,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.849258+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:51:27.849258+0000","last_peered":"2024-09-13T14:51:27.849258+0000","last_clean":"2024-09-13T14:51:27.849258+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:51:27.849258+0000","last_undegraded":"2024-09-13T14:51:27.849258+0000","last_fullsized":"2024-09-13T14:51:27.849258+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"224'6154","reported_seq":12624,"reported_epoch":224,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:24.085301+0000","last_change":"2024-09-13T14:50:08.716863+0000","last_active":"2024-09-13T14:51:24.085301+0000","last_peered":"2024-09-13T14:51:24.085301+0000","last_clean":"2024-09-13T14:49:59.725669+0000","last_became_active":"2024-09-13T14:50:08.707516+0000","last_became_peered":"2024-09-13T14:50:08.707516+0000","last_unstale":"2024-09-13T14:51:24.085301+0000","last_undegraded":"2024-09-13T14:50:08.570515+0000","last_fullsized":"2024-09-13T14:50:08.544112+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:20.218839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301465600,"num_objects":4600,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6154,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3185,"num_bytes_recovered":208732160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-722},{"shards":"1,4,5","objects":4600}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":310,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.790995+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:51:27.790995+0000","last_peered":"2024-09-13T14:51:27.790995+0000","last_clean":"2024-09-13T14:51:27.790995+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:51:27.790995+0000","last_undegraded":"2024-09-13T14:51:27.790995+0000","last_fullsized":"2024-09-13T14:51:27.790995+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":206,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700959+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:51:09.700959+0000","last_peered":"2024-09-13T14:51:09.700959+0000","last_clean":"2024-09-13T14:51:09.700959+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:51:09.700959+0000","last_undegraded":"2024-09-13T14:51:09.700959+0000","last_fullsized":"2024-09-13T14:51:09.700959+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"224'6100","reported_seq":6467,"reported_epoch":224,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:24.045998+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:51:24.045998+0000","last_peered":"2024-09-13T14:51:24.045998+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:51:24.045998+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6100,"log_dups_size":0,"ondisk_log_size":6100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297533440,"num_objects":4540,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6100,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":2360},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":110,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.825188+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:51:28.825188+0000","last_peered":"2024-09-13T14:51:28.825188+0000","last_clean":"2024-09-13T14:51:28.825188+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:51:28.825188+0000","last_undegraded":"2024-09-13T14:51:28.825188+0000","last_fullsized":"2024-09-13T14:51:28.825188+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":203,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.701068+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:51:09.701068+0000","last_peered":"2024-09-13T14:51:09.701068+0000","last_clean":"2024-09-13T14:51:09.701068+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:51:09.701068+0000","last_undegraded":"2024-09-13T14:51:09.701068+0000","last_fullsized":"2024-09-13T14:51:09.701068+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"226'6321","reported_seq":8442,"reported_epoch":226,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:28.628626+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:51:28.628626+0000","last_peered":"2024-09-13T14:51:28.628626+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:51:28.628626+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294977536,"num_objects":4501,"num_object_clones":0,"num_object_copies":13503,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":3649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6321,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73203712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":1248},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":1107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700789+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:51:09.700789+0000","last_peered":"2024-09-13T14:51:09.700789+0000","last_clean":"2024-09-13T14:51:09.700789+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:51:09.700789+0000","last_undegraded":"2024-09-13T14:51:09.700789+0000","last_fullsized":"2024-09-13T14:51:09.700789+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":233,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.701521+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:51:09.701521+0000","last_peered":"2024-09-13T14:51:09.701521+0000","last_clean":"2024-09-13T14:51:09.701521+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:51:09.701521+0000","last_undegraded":"2024-09-13T14:51:09.701521+0000","last_fullsized":"2024-09-13T14:51:09.701521+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"224'5980","reported_seq":10391,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:24.227477+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:51:24.227477+0000","last_peered":"2024-09-13T14:51:24.227477+0000","last_clean":"2024-09-13T14:51:24.227477+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:51:24.227477+0000","last_undegraded":"2024-09-13T14:51:24.227477+0000","last_fullsized":"2024-09-13T14:51:24.227477+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5980,"log_dups_size":0,"ondisk_log_size":5980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296550424,"num_objects":4526,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5980,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":48,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.791239+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:51:27.791239+0000","last_peered":"2024-09-13T14:51:27.791239+0000","last_clean":"2024-09-13T14:51:27.791239+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:51:27.791239+0000","last_undegraded":"2024-09-13T14:51:27.791239+0000","last_fullsized":"2024-09-13T14:51:27.791239+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":153,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.718795+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:51:10.718795+0000","last_peered":"2024-09-13T14:51:10.718795+0000","last_clean":"2024-09-13T14:51:10.718795+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:51:10.718795+0000","last_undegraded":"2024-09-13T14:51:10.718795+0000","last_fullsized":"2024-09-13T14:51:10.718795+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"224'6082","reported_seq":7314,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:24.450169+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:51:24.450169+0000","last_peered":"2024-09-13T14:51:24.450169+0000","last_clean":"2024-09-13T14:51:24.450169+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:51:24.450169+0000","last_undegraded":"2024-09-13T14:51:24.450169+0000","last_fullsized":"2024-09-13T14:51:24.450169+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305528832,"num_objects":4662,"num_object_clones":0,"num_object_copies":13986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6082,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":504,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793031+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:51:27.793031+0000","last_peered":"2024-09-13T14:51:27.793031+0000","last_clean":"2024-09-13T14:51:27.793031+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:51:27.793031+0000","last_undegraded":"2024-09-13T14:51:27.793031+0000","last_fullsized":"2024-09-13T14:51:27.793031+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":790,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.712632+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:51:09.712632+0000","last_peered":"2024-09-13T14:51:09.712632+0000","last_clean":"2024-09-13T14:51:09.712632+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:51:09.712632+0000","last_undegraded":"2024-09-13T14:51:09.712632+0000","last_fullsized":"2024-09-13T14:51:09.712632+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":268,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.791107+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:51:27.791107+0000","last_peered":"2024-09-13T14:51:27.791107+0000","last_clean":"2024-09-13T14:51:27.791107+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:51:27.791107+0000","last_undegraded":"2024-09-13T14:51:27.791107+0000","last_fullsized":"2024-09-13T14:51:27.791107+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":22,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793084+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:51:27.793084+0000","last_peered":"2024-09-13T14:51:27.793084+0000","last_clean":"2024-09-13T14:51:27.793084+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:51:27.793084+0000","last_undegraded":"2024-09-13T14:51:27.793084+0000","last_fullsized":"2024-09-13T14:51:27.793084+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"224'6235","reported_seq":10924,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:24.688749+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:51:24.688749+0000","last_peered":"2024-09-13T14:51:24.688749+0000","last_clean":"2024-09-13T14:51:24.688749+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:51:24.688749+0000","last_undegraded":"2024-09-13T14:51:24.688749+0000","last_fullsized":"2024-09-13T14:51:24.688749+0000","mapping_epoch":223,"log_start":"164'618","ondisk_log_start":"164'618","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302317568,"num_objects":4613,"num_object_clones":0,"num_object_copies":13839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":345,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.711670+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:51:10.711670+0000","last_peered":"2024-09-13T14:51:10.711670+0000","last_clean":"2024-09-13T14:51:10.711670+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:51:10.711670+0000","last_undegraded":"2024-09-13T14:51:10.711670+0000","last_fullsized":"2024-09-13T14:51:10.711670+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":316,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.792961+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:51:27.792961+0000","last_peered":"2024-09-13T14:51:27.792961+0000","last_clean":"2024-09-13T14:51:27.792961+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:51:27.792961+0000","last_undegraded":"2024-09-13T14:51:27.792961+0000","last_fullsized":"2024-09-13T14:51:27.792961+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":327,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.851988+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:51:27.851988+0000","last_peered":"2024-09-13T14:51:27.851988+0000","last_clean":"2024-09-13T14:51:27.851988+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:51:27.851988+0000","last_undegraded":"2024-09-13T14:51:27.851988+0000","last_fullsized":"2024-09-13T14:51:27.851988+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"226'6166","reported_seq":12008,"reported_epoch":226,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:28.637941+0000","last_change":"2024-09-13T14:50:38.024593+0000","last_active":"2024-09-13T14:51:28.637941+0000","last_peered":"2024-09-13T14:51:28.637941+0000","last_clean":"2024-09-13T14:48:36.189224+0000","last_became_active":"2024-09-13T14:50:38.015505+0000","last_became_peered":"2024-09-13T14:50:38.015505+0000","last_unstale":"2024-09-13T14:51:28.637941+0000","last_undegraded":"2024-09-13T14:50:37.965648+0000","last_fullsized":"2024-09-13T14:50:37.942220+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290324480,"num_objects":4430,"num_object_clones":0,"num_object_copies":13290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6166,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2845,"num_bytes_recovered":186449920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":2369},{"shards":"2,3,5","objects":2061}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":848,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.718963+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:51:10.718963+0000","last_peered":"2024-09-13T14:51:10.718963+0000","last_clean":"2024-09-13T14:51:10.718963+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:51:10.718963+0000","last_undegraded":"2024-09-13T14:51:10.718963+0000","last_fullsized":"2024-09-13T14:51:10.718963+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":303,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.711783+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:51:10.711783+0000","last_peered":"2024-09-13T14:51:10.711783+0000","last_clean":"2024-09-13T14:51:10.711783+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:51:10.711783+0000","last_undegraded":"2024-09-13T14:51:10.711783+0000","last_fullsized":"2024-09-13T14:51:10.711783+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":56,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.712597+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:51:10.712597+0000","last_peered":"2024-09-13T14:51:10.712597+0000","last_clean":"2024-09-13T14:51:10.712597+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:51:10.712597+0000","last_undegraded":"2024-09-13T14:51:10.712597+0000","last_fullsized":"2024-09-13T14:51:10.712597+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"226'6245","reported_seq":10829,"reported_epoch":226,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:28.616140+0000","last_change":"2024-09-13T14:50:42.308313+0000","last_active":"2024-09-13T14:51:28.616140+0000","last_peered":"2024-09-13T14:51:28.616140+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:51:28.616140+0000","last_undegraded":"2024-09-13T14:50:40.576119+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6245,"log_dups_size":0,"ondisk_log_size":6245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293273600,"num_objects":4475,"num_object_clones":0,"num_object_copies":13425,"num_objects_missing_on_primary":89,"num_objects_missing":89,"num_objects_degraded":2392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":148504576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"1,4,5","objects":9},{"shards":"3,4","objects":2358},{"shards":"3,4,5","objects":2108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":77,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.712807+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:51:09.712807+0000","last_peered":"2024-09-13T14:51:09.712807+0000","last_clean":"2024-09-13T14:51:09.712807+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:51:09.712807+0000","last_undegraded":"2024-09-13T14:51:09.712807+0000","last_fullsized":"2024-09-13T14:51:09.712807+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":208,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.714247+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:51:10.714247+0000","last_peered":"2024-09-13T14:51:10.714247+0000","last_clean":"2024-09-13T14:51:10.714247+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:51:10.714247+0000","last_undegraded":"2024-09-13T14:51:10.714247+0000","last_fullsized":"2024-09-13T14:51:10.714247+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":306,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.714318+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:51:10.714318+0000","last_peered":"2024-09-13T14:51:10.714318+0000","last_clean":"2024-09-13T14:51:10.714318+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:51:10.714318+0000","last_undegraded":"2024-09-13T14:51:10.714318+0000","last_fullsized":"2024-09-13T14:51:10.714318+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"225'6303","reported_seq":10853,"reported_epoch":225,"state":"peering","last_fresh":"2024-09-13T14:51:27.787500+0000","last_change":"2024-09-13T14:51:27.851059+0000","last_active":"2024-09-13T14:51:27.851059+0000","last_peered":"2024-09-13T14:51:27.787500+0000","last_clean":"2024-09-13T14:48:36.202432+0000","last_became_active":"2024-09-13T14:50:07.829887+0000","last_became_peered":"2024-09-13T14:50:07.829887+0000","last_unstale":"2024-09-13T14:51:27.787500+0000","last_undegraded":"2024-09-13T14:51:27.787500+0000","last_fullsized":"2024-09-13T14:50:07.543585+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295632896,"num_objects":4511,"num_object_clones":0,"num_object_copies":13533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":77,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.713239+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:51:09.713239+0000","last_peered":"2024-09-13T14:51:09.713239+0000","last_clean":"2024-09-13T14:51:09.713239+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:51:09.713239+0000","last_undegraded":"2024-09-13T14:51:09.713239+0000","last_fullsized":"2024-09-13T14:51:09.713239+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":858,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.714575+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:51:10.714575+0000","last_peered":"2024-09-13T14:51:10.714575+0000","last_clean":"2024-09-13T14:51:10.714575+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:51:10.714575+0000","last_undegraded":"2024-09-13T14:51:10.714575+0000","last_fullsized":"2024-09-13T14:51:10.714575+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":283,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.712063+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:51:10.712063+0000","last_peered":"2024-09-13T14:51:10.712063+0000","last_clean":"2024-09-13T14:51:10.712063+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:51:10.712063+0000","last_undegraded":"2024-09-13T14:51:10.712063+0000","last_fullsized":"2024-09-13T14:51:10.712063+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"224'6262","reported_seq":10944,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:24.681490+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:51:24.681490+0000","last_peered":"2024-09-13T14:51:24.681490+0000","last_clean":"2024-09-13T14:51:24.681490+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:51:24.681490+0000","last_undegraded":"2024-09-13T14:51:24.681490+0000","last_fullsized":"2024-09-13T14:51:24.681490+0000","mapping_epoch":218,"log_start":"164'645","ondisk_log_start":"164'645","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304480256,"num_objects":4646,"num_object_clones":0,"num_object_copies":13938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6262,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":310,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.714426+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:51:10.714426+0000","last_peered":"2024-09-13T14:51:10.714426+0000","last_clean":"2024-09-13T14:51:10.714426+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:51:10.714426+0000","last_undegraded":"2024-09-13T14:51:10.714426+0000","last_fullsized":"2024-09-13T14:51:10.714426+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":535,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.849698+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:51:27.849698+0000","last_peered":"2024-09-13T14:51:27.849698+0000","last_clean":"2024-09-13T14:51:27.849698+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:51:27.849698+0000","last_undegraded":"2024-09-13T14:51:27.849698+0000","last_fullsized":"2024-09-13T14:51:27.849698+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":312,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.849335+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:51:27.849335+0000","last_peered":"2024-09-13T14:51:27.849335+0000","last_clean":"2024-09-13T14:51:27.849335+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:51:27.849335+0000","last_undegraded":"2024-09-13T14:51:27.849335+0000","last_fullsized":"2024-09-13T14:51:27.849335+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"224'6015","reported_seq":8203,"reported_epoch":224,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:23.939934+0000","last_change":"2024-09-13T14:50:19.881564+0000","last_active":"2024-09-13T14:51:23.939934+0000","last_peered":"2024-09-13T14:51:23.939934+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:51:23.939934+0000","last_undegraded":"2024-09-13T14:50:19.795499+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294584320,"num_objects":4495,"num_object_clones":0,"num_object_copies":13485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2404},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2090}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":332,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.850867+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:51:27.850867+0000","last_peered":"2024-09-13T14:51:27.850867+0000","last_clean":"2024-09-13T14:51:27.850867+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:51:27.850867+0000","last_undegraded":"2024-09-13T14:51:27.850867+0000","last_fullsized":"2024-09-13T14:51:27.850867+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":760,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.714435+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:51:10.714435+0000","last_peered":"2024-09-13T14:51:10.714435+0000","last_clean":"2024-09-13T14:51:10.714435+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:51:10.714435+0000","last_undegraded":"2024-09-13T14:51:10.714435+0000","last_fullsized":"2024-09-13T14:51:10.714435+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":48,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.791138+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:51:27.791138+0000","last_peered":"2024-09-13T14:51:27.791138+0000","last_clean":"2024-09-13T14:51:27.791138+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:51:27.791138+0000","last_undegraded":"2024-09-13T14:51:27.791138+0000","last_fullsized":"2024-09-13T14:51:27.791138+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"227'6282","reported_seq":7842,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.825128+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:51:28.825128+0000","last_peered":"2024-09-13T14:51:28.825128+0000","last_clean":"2024-09-13T14:51:28.825128+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:51:28.825128+0000","last_undegraded":"2024-09-13T14:51:28.825128+0000","last_fullsized":"2024-09-13T14:51:28.825128+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300023808,"num_objects":4578,"num_object_clones":0,"num_object_copies":13734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6282,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":307,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.851653+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:51:27.851653+0000","last_peered":"2024-09-13T14:51:27.851653+0000","last_clean":"2024-09-13T14:51:27.851653+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:51:27.851653+0000","last_undegraded":"2024-09-13T14:51:27.851653+0000","last_fullsized":"2024-09-13T14:51:27.851653+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":778,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.792952+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:51:27.792952+0000","last_peered":"2024-09-13T14:51:27.792952+0000","last_clean":"2024-09-13T14:51:27.792952+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:51:27.792952+0000","last_undegraded":"2024-09-13T14:51:27.792952+0000","last_fullsized":"2024-09-13T14:51:27.792952+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":210,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700694+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:51:09.700694+0000","last_peered":"2024-09-13T14:51:09.700694+0000","last_clean":"2024-09-13T14:51:09.700694+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:51:09.700694+0000","last_undegraded":"2024-09-13T14:51:09.700694+0000","last_fullsized":"2024-09-13T14:51:09.700694+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":205,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700957+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:51:09.700957+0000","last_peered":"2024-09-13T14:51:09.700957+0000","last_clean":"2024-09-13T14:51:09.700957+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:51:09.700957+0000","last_undegraded":"2024-09-13T14:51:09.700957+0000","last_fullsized":"2024-09-13T14:51:09.700957+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":253,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.701179+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:51:09.701179+0000","last_peered":"2024-09-13T14:51:09.701179+0000","last_clean":"2024-09-13T14:51:09.701179+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:51:09.701179+0000","last_undegraded":"2024-09-13T14:51:09.701179+0000","last_fullsized":"2024-09-13T14:51:09.701179+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":307,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.711927+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:51:10.711927+0000","last_peered":"2024-09-13T14:51:10.711927+0000","last_clean":"2024-09-13T14:51:10.711927+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:51:10.711927+0000","last_undegraded":"2024-09-13T14:51:10.711927+0000","last_fullsized":"2024-09-13T14:51:10.711927+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":152,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.854892+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:51:27.854892+0000","last_peered":"2024-09-13T14:51:27.854892+0000","last_clean":"2024-09-13T14:51:27.854892+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:51:27.854892+0000","last_undegraded":"2024-09-13T14:51:27.854892+0000","last_fullsized":"2024-09-13T14:51:27.854892+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":86,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.713204+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:51:09.713204+0000","last_peered":"2024-09-13T14:51:09.713204+0000","last_clean":"2024-09-13T14:51:09.713204+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:51:09.713204+0000","last_undegraded":"2024-09-13T14:51:09.713204+0000","last_fullsized":"2024-09-13T14:51:09.713204+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":289,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.790265+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:51:27.790265+0000","last_peered":"2024-09-13T14:51:27.790265+0000","last_clean":"2024-09-13T14:51:27.790265+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:51:27.790265+0000","last_undegraded":"2024-09-13T14:51:27.790265+0000","last_fullsized":"2024-09-13T14:51:27.790265+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":343,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.849896+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:51:27.849896+0000","last_peered":"2024-09-13T14:51:27.849896+0000","last_clean":"2024-09-13T14:51:27.849896+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:51:27.849896+0000","last_undegraded":"2024-09-13T14:51:27.849896+0000","last_fullsized":"2024-09-13T14:51:27.849896+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":340,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.848775+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:51:27.848775+0000","last_peered":"2024-09-13T14:51:27.848775+0000","last_clean":"2024-09-13T14:51:27.848775+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:51:27.848775+0000","last_undegraded":"2024-09-13T14:51:27.848775+0000","last_fullsized":"2024-09-13T14:51:27.848775+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":119,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.792882+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:51:27.792882+0000","last_peered":"2024-09-13T14:51:27.792882+0000","last_clean":"2024-09-13T14:51:27.792882+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:51:27.792882+0000","last_undegraded":"2024-09-13T14:51:27.792882+0000","last_fullsized":"2024-09-13T14:51:27.792882+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":307,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.848931+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:51:27.848931+0000","last_peered":"2024-09-13T14:51:27.848931+0000","last_clean":"2024-09-13T14:51:27.848931+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:51:27.848931+0000","last_undegraded":"2024-09-13T14:51:27.848931+0000","last_fullsized":"2024-09-13T14:51:27.848931+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":323,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.851915+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:51:27.851915+0000","last_peered":"2024-09-13T14:51:27.851915+0000","last_clean":"2024-09-13T14:51:27.851915+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:51:27.851915+0000","last_undegraded":"2024-09-13T14:51:27.851915+0000","last_fullsized":"2024-09-13T14:51:27.851915+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":288,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.711782+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:51:10.711782+0000","last_peered":"2024-09-13T14:51:10.711782+0000","last_clean":"2024-09-13T14:51:10.711782+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:51:10.711782+0000","last_undegraded":"2024-09-13T14:51:10.711782+0000","last_fullsized":"2024-09-13T14:51:10.711782+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":147,"reported_epoch":226,"state":"peering","last_fresh":"2024-09-13T14:51:27.854635+0000","last_change":"2024-09-13T14:51:27.854635+0000","last_active":"2024-09-13T14:51:27.854635+0000","last_peered":"2024-09-13T14:51:27.441350+0000","last_clean":"2024-09-13T14:50:08.544047+0000","last_became_active":"2024-09-13T14:50:20.795534+0000","last_became_peered":"2024-09-13T14:50:20.795534+0000","last_unstale":"2024-09-13T14:51:27.854635+0000","last_undegraded":"2024-09-13T14:51:27.854635+0000","last_fullsized":"2024-09-13T14:51:27.854635+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700965+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:51:09.700965+0000","last_peered":"2024-09-13T14:51:09.700965+0000","last_clean":"2024-09-13T14:51:09.700965+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:51:09.700965+0000","last_undegraded":"2024-09-13T14:51:09.700965+0000","last_fullsized":"2024-09-13T14:51:09.700965+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":47,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-13T14:51:10.718715+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:51:10.718715+0000","last_peered":"2024-09-13T14:51:10.718715+0000","last_clean":"2024-09-13T14:51:10.718715+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:51:10.718715+0000","last_undegraded":"2024-09-13T14:51:10.718715+0000","last_fullsized":"2024-09-13T14:51:10.718715+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":316,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793082+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:51:27.793082+0000","last_peered":"2024-09-13T14:51:27.793082+0000","last_clean":"2024-09-13T14:51:27.793082+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:51:27.793082+0000","last_undegraded":"2024-09-13T14:51:27.793082+0000","last_fullsized":"2024-09-13T14:51:27.793082+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":205,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700913+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:51:09.700913+0000","last_peered":"2024-09-13T14:51:09.700913+0000","last_clean":"2024-09-13T14:51:09.700913+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:51:09.700913+0000","last_undegraded":"2024-09-13T14:51:09.700913+0000","last_fullsized":"2024-09-13T14:51:09.700913+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":61,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.712831+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:51:09.712831+0000","last_peered":"2024-09-13T14:51:09.712831+0000","last_clean":"2024-09-13T14:51:09.712831+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:51:09.712831+0000","last_undegraded":"2024-09-13T14:51:09.712831+0000","last_fullsized":"2024-09-13T14:51:09.712831+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":202,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700867+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:51:09.700867+0000","last_peered":"2024-09-13T14:51:09.700867+0000","last_clean":"2024-09-13T14:51:09.700867+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:51:09.700867+0000","last_undegraded":"2024-09-13T14:51:09.700867+0000","last_fullsized":"2024-09-13T14:51:09.700867+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":63,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.713357+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:51:09.713357+0000","last_peered":"2024-09-13T14:51:09.713357+0000","last_clean":"2024-09-13T14:51:09.713357+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:51:09.713357+0000","last_undegraded":"2024-09-13T14:51:09.713357+0000","last_fullsized":"2024-09-13T14:51:09.713357+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":310,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.790672+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:51:27.790672+0000","last_peered":"2024-09-13T14:51:27.790672+0000","last_clean":"2024-09-13T14:51:27.790672+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:51:27.790672+0000","last_undegraded":"2024-09-13T14:51:27.790672+0000","last_fullsized":"2024-09-13T14:51:27.790672+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":326,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.875093+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:51:27.875093+0000","last_peered":"2024-09-13T14:51:27.875093+0000","last_clean":"2024-09-13T14:51:27.875093+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:51:27.875093+0000","last_undegraded":"2024-09-13T14:51:27.875093+0000","last_fullsized":"2024-09-13T14:51:27.875093+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":232,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-13T14:51:09.700659+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:51:09.700659+0000","last_peered":"2024-09-13T14:51:09.700659+0000","last_clean":"2024-09-13T14:51:09.700659+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:51:09.700659+0000","last_undegraded":"2024-09-13T14:51:09.700659+0000","last_fullsized":"2024-09-13T14:51:09.700659+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":379,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.849793+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:51:27.849793+0000","last_peered":"2024-09-13T14:51:27.849793+0000","last_clean":"2024-09-13T14:51:27.849793+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:51:27.849793+0000","last_undegraded":"2024-09-13T14:51:27.849793+0000","last_fullsized":"2024-09-13T14:51:27.849793+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4766564376,"num_objects":72733,"num_object_clones":0,"num_object_copies":218199,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":20312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98815,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26902,"num_bytes_recovered":1736376320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15270948864,"data_stored":15270936648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97387,"ondisk_log_size":97387,"up":48,"acting":41,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234288,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495925,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2590472,"kb_used_data":2356732,"kb_used_omap":1,"kb_used_meta":233726,"kb_avail":91154680,"statfs":{"total":95995035648,"available":93342392320,"internally_reserved":0,"allocated":2413293568,"data_stored":2412035050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":239335879},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757559,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2601440,"kb_used_data":2421848,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":91143712,"statfs":{"total":95995035648,"available":93331161088,"internally_reserved":0,"allocated":2479972352,"data_stored":2478729319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":183892423},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019199,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1752936,"kb_used_data":1473248,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91992216,"statfs":{"total":95995035648,"available":94200029184,"internally_reserved":0,"allocated":1508605952,"data_stored":1507359185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":286390726},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468085,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3248892,"kb_used_data":3054832,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":90496260,"statfs":{"total":95995035648,"available":92668170240,"internally_reserved":0,"allocated":3128147968,"data_stored":3126897091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":198700196},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,2,1,2,2,8,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631613,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2375916,"kb_used_data":2055460,"kb_used_omap":8,"kb_used_meta":320439,"kb_avail":91369236,"statfs":{"total":95995035648,"available":93562097664,"internally_reserved":0,"allocated":2104791040,"data_stored":2103540703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":328130527},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795136,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2441124,"kb_used_data":2202588,"kb_used_omap":7,"kb_used_meta":238520,"kb_avail":91304028,"statfs":{"total":95995035648,"available":93495324672,"internally_reserved":0,"allocated":2255450112,"data_stored":2254203182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":244245106},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958667,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1591496,"kb_used_data":1298812,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":92153656,"statfs":{"total":95995035648,"available":94365343744,"internally_reserved":0,"allocated":1329983488,"data_stored":1328721425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":299688423},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1321930752,"data_stored":1321926680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2242187264,"data_stored":2242183192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2097872896,"data_stored":2097872896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3114860544,"data_stored":3114860544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1500577792,"data_stored":1500577792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2475626496,"data_stored":2475622424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2406940672,"data_stored":2406940672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":110952448,"data_stored":110952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:31.527 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:31.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:31.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:31.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:31.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:31.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:31.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:31.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:32.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.090 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:31 smithi154 ceph-mon[102314]: from='client.35151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:32.090 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:31 smithi154 ceph-mon[107038]: from='client.35151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:32.090 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:51:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:51:31] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-13T14:51:32.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:32.164 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:31 smithi172 ceph-mon[100696]: from='client.35151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:32.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:32.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:32.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:32.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:32.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.669 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:32.794 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:32.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:32.940 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:32.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:33.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:33.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:33.093 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:32 smithi154 ceph-mon[102314]: pgmap v660: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 15723/215694 objects degraded (7.289%); 4429/215694 objects misplaced (2.053%); 7.8 MiB/s, 138 objects/s recovering 2024-09-13T14:51:33.094 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:32 smithi154 ceph-mon[107038]: pgmap v660: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 15723/215694 objects degraded (7.289%); 4429/215694 objects misplaced (2.053%); 7.8 MiB/s, 138 objects/s recovering 2024-09-13T14:51:33.159 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:32 smithi172 ceph-mon[100696]: pgmap v660: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 241 op/s; 15723/215694 objects degraded (7.289%); 4429/215694 objects misplaced (2.053%); 7.8 MiB/s, 138 objects/s recovering 2024-09-13T14:51:33.160 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:51:33.160 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:97375 5.9 5:9dd925ab:::benchmark_data_smithi154_144140_object97374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 424.22715822700002, 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.11292851499999999, 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.161 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 97375 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:51:33.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930165+0000", 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975030+0000", 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044864996999999997 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975066+0000", 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5460999999999998e-05 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975076+0000", 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0397999999999999e-05 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.979893+0000", 2024-09-13T14:51:33.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048163379999999999 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.980059+0000", 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016674300000000001 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.980241+0000", 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018151899999999999 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.988970+0000", 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087295719999999997 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.989030+0000", 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9859999999999999e-05 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.040919+0000", 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.051888722999999998 2024-09-13T14:51:33.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.040974+0000", 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4549999999999998e-05 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.043096+0000", 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002122199 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:113747 5.9 5:91f8fbcd:::benchmark_data_smithi154_144140_object3394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 400.91670273900002, 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.091457920999999998, 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113747 2024-09-13T14:51:33.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240622+0000", 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240625+0000", 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7530000000000002e-06 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240630+0000", 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2979999999999999e-06 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240635+0000", 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3989999999999997e-06 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247577+0000", 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0069425770000000001 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247719+0000", 2024-09-13T14:51:33.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014185900000000001 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247864+0000", 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014463900000000001 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.250756+0000", 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028924279999999998 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.251141+0000", 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038421899999999999 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.330437+0000", 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079296273 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.330474+0000", 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7418e-05 2024-09-13T14:51:33.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.332081+0000", 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001606447 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:114992 5.9 5:9bf9d9df:::benchmark_data_smithi154_144140_object4639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 399.172670692, 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.85779442800000005, 2024-09-13T14:51:33.174 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114992 2024-09-13T14:51:33.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984654+0000", 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:33.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984658+0000", 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7450000000000001e-06 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984665+0000", 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5020000000000003e-06 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984675+0000", 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.132e-06 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984715+0000", 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701000000000002e-05 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984723+0000", 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6140000000000003e-06 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824713+0000", 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.83998967899999999 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824847+0000", 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000134356 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:33.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824939+0000", 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1880999999999997e-05 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830175+0000", 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052358680000000003 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.833865+0000", 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003690189 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.840699+0000", 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00683411 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.840754+0000", 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5126e-05 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842449+0000", 2024-09-13T14:51:33.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016951170000000001 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:115007 5.9 5:93772757:::benchmark_data_smithi154_144140_object4654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 399.144658311, 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.83119907900000001, 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.183 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115007 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012666+0000", 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012675+0000", 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6880000000000005e-06 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012685+0000", 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.696e-06 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012692+0000", 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8549999999999996e-06 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012728+0000", 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6328000000000002e-05 2024-09-13T14:51:33.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012736+0000", 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.011e-06 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830212+0000", 2024-09-13T14:51:33.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81747611099999995 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830363+0000", 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015098299999999999 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:33.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830536+0000", 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173227 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834179+0000", 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003643178 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838231+0000", 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040517900000000004 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842523+0000", 2024-09-13T14:51:33.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042920570000000002 2024-09-13T14:51:33.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842581+0000", 2024-09-13T14:51:33.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7868999999999998e-05 2024-09-13T14:51:33.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843866+0000", 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001285071 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:115010 5.9 5:9c0819f0:::benchmark_data_smithi154_144140_object4657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:51:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 399.140147727, 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.82826413700000001, 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115010 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:51:33.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017177+0000", 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017181+0000", 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6619999999999998e-06 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017188+0000", 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0960000000000003e-06 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017192+0000", 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0330000000000002e-06 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017219+0000", 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7807999999999999e-05 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017226+0000", 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4629999999999998e-06 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834282+0000", 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81705591399999999 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834452+0000", 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170314 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834602+0000", 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149732 2024-09-13T14:51:33.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838287+0000", 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00368491 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838568+0000", 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028085099999999998 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843931+0000", 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053631590000000002 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843991+0000", 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0174999999999999e-05 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.845442+0000", 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001450914 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117184 5.4 5:285a1434:::benchmark_data_smithi154_144140_object6831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.98696977499998, 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15315772, 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.200 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117184 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170355+0000", 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170358+0000", 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.19e-06 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170365+0000", 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9419999999999998e-06 2024-09-13T14:51:33.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170371+0000", 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7899999999999996e-06 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174205+0000", 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038343880000000002 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174315+0000", 2024-09-13T14:51:33.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109225 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174416+0000", 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010148 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.177943+0000", 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035270409999999999 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.183599+0000", 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056554109999999999 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323412+0000", 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.13981297000000001 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323466+0000", 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4549000000000002e-05 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323513+0000", 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7312999999999999e-05 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117538 5.4 5:2cd2fa81:::benchmark_data_smithi154_144140_object7185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.51835006900001, 2024-09-13T14:51:33.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.140160653, 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117538 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638974+0000", 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638981+0000", 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3900000000000004e-06 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638999+0000", 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7948000000000002e-05 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639007+0000", 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9670000000000001e-06 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639049+0000", 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1593e-05 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639234+0000", 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185253 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639344+0000", 2024-09-13T14:51:33.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109447 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648634+0000", 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092907010000000002 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648647+0000", 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3182e-05 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779063+0000", 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.130415064 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779100+0000", 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7110000000000002e-05 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779136+0000", 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6436999999999997e-05 2024-09-13T14:51:33.212 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117539 5.4 5:27e8a4d2:::benchmark_data_smithi154_144140_object7186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.51814287100001, 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.140068675, 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.213 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117539 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639182+0000", 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639187+0000", 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.758e-06 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639198+0000", 2024-09-13T14:51:33.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1674999999999999e-05 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639205+0000", 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5409999999999997e-06 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644291+0000", 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050859499999999997 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644444+0000", 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015357700000000001 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644545+0000", 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010090999999999999 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655177+0000", 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010632157 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.776212+0000", 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.121034903 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779189+0000", 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029772129999999998 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779215+0000", 2024-09-13T14:51:33.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5933999999999999e-05 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779251+0000", 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5936000000000003e-05 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117544 5.4 5:239bf146:::benchmark_data_smithi154_144140_object7191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.51644055600002, 2024-09-13T14:51:33.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.138439275, 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117544 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640884+0000", 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640889+0000", 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7640000000000002e-06 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640902+0000", 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3046e-05 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640909+0000", 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1840000000000004e-06 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648692+0000", 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077836470000000003 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648798+0000", 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010568700000000001 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648898+0000", 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100207 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655188+0000", 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062894869999999999 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.776255+0000", 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12106747599999999 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779281+0000", 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030257999999999999 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779305+0000", 2024-09-13T14:51:33.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4136000000000001e-05 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779324+0000", 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9199999999999999e-05 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117548 5.4 5:28e11507:::benchmark_data_smithi154_144140_object7195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.51013091300001, 2024-09-13T14:51:33.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15636461800000001, 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117548 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647193+0000", 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647199+0000", 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4439999999999999e-06 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647209+0000", 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0275000000000001e-05 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647215+0000", 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1040000000000003e-06 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652094+0000", 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048785010000000004 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652244+0000", 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015033599999999999 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652372+0000", 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012824700000000001 2024-09-13T14:51:33.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.658282+0000", 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059094480000000003 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.783415+0000", 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12513291300000001 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803465+0000", 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020050179000000001 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803514+0000", 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8661e-05 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803559+0000", 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5626e-05 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117551 5.4 5:28c6b1cf:::benchmark_data_smithi154_144140_object7198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.50862447999998, 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15501442800000001, 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117551 2024-09-13T14:51:33.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648700+0000", 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648704+0000", 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4279999999999998e-06 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648712+0000", 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0320000000000003e-06 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648718+0000", 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5620000000000003e-06 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655217+0000", 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064988140000000003 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655329+0000", 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112169 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655446+0000", 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116567 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.658387+0000", 2024-09-13T14:51:33.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029411540000000001 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.783571+0000", 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12518472899999999 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803616+0000", 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020044480999999999 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803682+0000", 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6157999999999999e-05 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803715+0000", 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3432000000000001e-05 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.240 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117857 5.4 5:20ac363d:::benchmark_data_smithi154_144140_object7504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.056636744, 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12765149100000001, 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117857 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100687+0000", 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100696+0000", 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100716+0000", 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9788000000000001e-05 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100724+0000", 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4519999999999997e-06 2024-09-13T14:51:33.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100794+0000", 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9370999999999997e-05 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100907+0000", 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113435 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.101008+0000", 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100407 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107808+0000", 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068003430000000004 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107856+0000", 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7775000000000003e-05 2024-09-13T14:51:33.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228218+0000", 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12036281 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228302+0000", 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3688000000000002e-05 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228340+0000", 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090999999999998e-05 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117860 5.4 5:29f284c5:::benchmark_data_smithi154_144140_object7507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 395.05349290300001, 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.124665336, 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117860 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103831+0000", 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103839+0000", 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8129999999999996e-06 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103855+0000", 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6900000000000001e-05 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103864+0000", 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000003e-06 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104514+0000", 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000650856 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104600+0000", 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5352999999999994e-05 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104680+0000", 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0687999999999997e-05 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107887+0000", 2024-09-13T14:51:33.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032067060000000001 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228173+0000", 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12028613 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228378+0000", 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020511699999999999 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228456+0000", 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7156000000000005e-05 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228498+0000", 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2373000000000002e-05 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:172124 5.4 5:24c92774:::benchmark_data_smithi154_144140_object61771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:06.268190+0000", 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 326.88913591699998, 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.115191713, 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 172124 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268190+0000", 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268190+0000", 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268188+0000", 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268195+0000", 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8820000000000003e-06 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268210+0000", 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4799e-05 2024-09-13T14:51:33.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268215+0000", 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.604e-06 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268251+0000", 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6044000000000003e-05 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268339+0000", 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7818000000000005e-05 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268416+0000", 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7059999999999997e-05 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.271372+0000", 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029560469999999998 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.283085+0000", 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011712573 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383290+0000", 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.100204814 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383354+0000", 2024-09-13T14:51:33.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4091999999999996e-05 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383381+0000", 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7559000000000002e-05 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:172133 5.4 5:24332ad7:::benchmark_data_smithi154_144140_object61780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:51:33.260 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:06.276130+0000", 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 326.88119527700002, 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10733804399999999, 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 172133 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276130+0000", 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276130+0000", 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276129+0000", 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276134+0000", 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6999999999999999e-06 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276146+0000", 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276151+0000", 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9450000000000001e-06 2024-09-13T14:51:33.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276196+0000", 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5734e-05 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276358+0000", 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016167800000000001 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276486+0000", 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127806 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.280384+0000", 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038979990000000001 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.283921+0000", 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035375150000000002 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383414+0000", 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.099493087999999993 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383446+0000", 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1621000000000001e-05 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383468+0000", 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.266 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:173360 5.4 5:2b9b10d0:::benchmark_data_smithi154_144140_object63007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.22180248699999, 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10636651599999999, 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.267 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 173360 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935525+0000", 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935531+0000", 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7969999999999997e-06 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935536+0000", 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0270000000000003e-06 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.941411+0000", 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058755359999999998 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.941518+0000", 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106265 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.941619+0000", 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101735 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.947007+0000", 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053876490000000004 2024-09-13T14:51:33.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.948811+0000", 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001803526 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:08.041799+0000", 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.092988238000000001 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:08.041835+0000", 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5636000000000002e-05 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:08.041890+0000", 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5003000000000002e-05 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:193138 5.4 5:259154cc:::benchmark_data_smithi154_144140_object82785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-13T14:51:33.273 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:33.885070+0000", 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 299.27225584500002, 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.087953613, 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193138 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885070+0000", 2024-09-13T14:51:33.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885070+0000", 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885068+0000", 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885076+0000", 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9859999999999994e-06 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885092+0000", 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6833e-05 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885101+0000", 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3680000000000002e-06 2024-09-13T14:51:33.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885151+0000", 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0373e-05 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885278+0000", 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012707699999999999 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885400+0000", 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012204700000000001 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.888403+0000", 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030024539999999999 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.963692+0000", 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.075288788999999995 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.972936+0000", 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092441669999999993 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.972985+0000", 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9475000000000003e-05 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.973023+0000", 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8074999999999997e-05 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:215616 5.4 5:24d075d9:::benchmark_data_smithi154_144140_object105263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:47:03.377793+0000", 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 269.77953210800001, 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12910099899999999, 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 215616 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377793+0000", 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377793+0000", 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377792+0000", 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377799+0000", 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1740000000000003e-06 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377813+0000", 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3534e-05 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377820+0000", 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7279999999999995e-06 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377877+0000", 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6202000000000002e-05 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.378006+0000", 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129111 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.378116+0000", 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110344 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382011+0000", 2024-09-13T14:51:33.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038949259999999999 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.397756+0000", 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015745037 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506830+0000", 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.109073429 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506867+0000", 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7662e-05 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506894+0000", 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7200999999999999e-05 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.286 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:215621 5.4 5:28250e67:::benchmark_data_smithi154_144140_object105268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:47:03.382845+0000", 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 269.774480052, 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12415194, 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 215621 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382845+0000", 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382845+0000", 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382844+0000", 2024-09-13T14:51:33.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382851+0000", 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1600000000000001e-06 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382864+0000", 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2646999999999999e-05 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382872+0000", 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9519999999999994e-06 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382911+0000", 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9765999999999998e-05 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382986+0000", 2024-09-13T14:51:33.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5123000000000006e-05 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.383055+0000", 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8615999999999995e-05 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.386242+0000", 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031869229999999999 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.402788+0000", 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016545814999999998 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506944+0000", 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10415627099999999 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506975+0000", 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.074e-05 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506997+0000", 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2572999999999999e-05 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:215632 5.4 5:23b35507:::benchmark_data_smithi154_144140_object105279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 269.763895547, 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.113641928, 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 215632 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:51:33.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:51:33.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393432+0000", 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3949999999999999e-06 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393438+0000", 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6350000000000003e-06 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:51:33.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393442+0000", 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393493+0000", 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0192999999999999e-05 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393607+0000", 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114048 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393727+0000", 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120284 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.397785+0000", 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040582680000000003 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.402931+0000", 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051453310000000004 2024-09-13T14:51:33.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.507022+0000", 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10409193999999999 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.507047+0000", 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4746999999999999e-05 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.507072+0000", 2024-09-13T14:51:33.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4695e-05 2024-09-13T14:51:33.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:51:33.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:51:33.300 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:51:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.302 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:33.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:33.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:33.427 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:33.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:33.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:33.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:33.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:33.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:33.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:33.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:33.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:33.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:34.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:34.071 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:34.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:34.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:34.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:51:34.295 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:51:34.295 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:34.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:34.327 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:51:34.327 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:34.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:34.355 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:51:34.356 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:34.391 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:51:34.391 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:34.421 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:51:34.421 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:34.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:34.444 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:51:34.444 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:34.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:34.482 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:51:34.482 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:34.502 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:51:34.502 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:34.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:34.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:34.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:34.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:34.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:34.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:34.830 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:51:34.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:34.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:34.984 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:34.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:34 smithi172 ceph-mon[100696]: pgmap v661: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 187 op/s; 15723/215694 objects degraded (7.289%); 4429/215694 objects misplaced (2.053%); 6.1 MiB/s, 107 objects/s recovering 2024-09-13T14:51:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:34 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1304702779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:34 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:34 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1828843009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:34 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15723/215694 objects degraded (7.289%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:35.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:35.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:35.160 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[102314]: pgmap v661: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 187 op/s; 15723/215694 objects degraded (7.289%); 4429/215694 objects misplaced (2.053%); 6.1 MiB/s, 107 objects/s recovering 2024-09-13T14:51:35.160 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1304702779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1828843009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15723/215694 objects degraded (7.289%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[107038]: pgmap v661: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 187 op/s; 15723/215694 objects degraded (7.289%); 4429/215694 objects misplaced (2.053%); 6.1 MiB/s, 107 objects/s recovering 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1304702779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1828843009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:35.161 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:34 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15723/215694 objects degraded (7.289%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:35.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:35.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:35.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:35.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:35.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:35.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:35.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:35.622 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:35.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:35.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:35.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.873 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:35.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:35.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:35 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:35.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:35 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:35.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:35 smithi154 ceph-mon[102314]: osdmap e228: 8 total, 8 up, 7 in 2024-09-13T14:51:35.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:35 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:35.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:35 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:35.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:35 smithi154 ceph-mon[107038]: osdmap e228: 8 total, 8 up, 7 in 2024-09-13T14:51:35.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:35.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:35.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:36.068 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:36.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:36.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:36.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:36.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:35 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:51:36.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:35 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:36.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:35 smithi172 ceph-mon[100696]: osdmap e228: 8 total, 8 up, 7 in 2024-09-13T14:51:36.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:36.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:36.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:36.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:36.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:36.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:36.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:36.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:36.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:36.829 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:36.845 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:36 smithi154 ceph-mon[102314]: pgmap v663: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 442 op/s; 15723/210657 objects degraded (7.464%); 4136/210657 objects misplaced (1.963%); 9.3 MiB/s, 160 objects/s recovering 2024-09-13T14:51:36.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:36.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:36.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:36.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:37.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:37.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:37.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:37.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:51:37.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:36 smithi154 ceph-mon[107038]: pgmap v663: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 442 op/s; 15723/210657 objects degraded (7.464%); 4136/210657 objects misplaced (1.963%); 9.3 MiB/s, 160 objects/s recovering 2024-09-13T14:51:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:36 smithi172 ceph-mon[100696]: pgmap v663: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 442 op/s; 15723/210657 objects degraded (7.464%); 4136/210657 objects misplaced (1.963%); 9.3 MiB/s, 160 objects/s recovering 2024-09-13T14:51:37.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:37.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:37.396 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:37.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:37.520 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:37.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:37.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:37.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:37.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:37.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:37.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:37.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:37.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:51:38.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:38.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:38.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:38.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:38.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:38.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:38.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:38.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:38.618 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:38.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:38.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:38.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:38.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:38.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:38.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:38.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:39.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.116 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.152 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:38 smithi154 ceph-mon[102314]: pgmap v664: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 15723/210657 objects degraded (7.464%); 4136/210657 objects misplaced (1.963%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:51:39.153 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:38 smithi154 ceph-mon[107038]: pgmap v664: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 15723/210657 objects degraded (7.464%); 4136/210657 objects misplaced (1.963%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:51:39.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:38 smithi172 ceph-mon[100696]: pgmap v664: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 321 op/s; 15723/210657 objects degraded (7.464%); 4136/210657 objects misplaced (1.963%); 3.0 MiB/s, 48 objects/s recovering 2024-09-13T14:51:39.248 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:39.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:39.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:39.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:39.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:39.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.622 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:39.746 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:39.834 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:51:39.834 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:51:39.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:39.869 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:39.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:39.996 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:39.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:40.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:40.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:39 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15723/210657 objects degraded (7.464%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:40.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:39 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15723/210657 objects degraded (7.464%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:40.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:39 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15723/210657 objects degraded (7.464%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:40.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:40.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:40.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:40.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:40.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:40.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:51:40.749 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:51:40.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:40.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:40.885 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:40.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:40 smithi154 ceph-mon[102314]: pgmap v665: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 312 op/s; 15683/208914 objects degraded (7.507%); 4136/208914 objects misplaced (1.980%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:51:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:40 smithi154 ceph-mon[102314]: from='client.35169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:40 smithi154 ceph-mon[107038]: pgmap v665: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 312 op/s; 15683/208914 objects degraded (7.507%); 4136/208914 objects misplaced (1.980%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:51:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:40 smithi154 ceph-mon[107038]: from='client.35169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:40.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:41.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:41.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:41.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:40 smithi172 ceph-mon[100696]: pgmap v665: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 312 op/s; 15683/208914 objects degraded (7.507%); 4136/208914 objects misplaced (1.980%); 2.7 MiB/s, 43 objects/s recovering 2024-09-13T14:51:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:40 smithi172 ceph-mon[100696]: from='client.35169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:51:41.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:41.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.387 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:41.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:41.398 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:41.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:41.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:41.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:41.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:41.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:41.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:41.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:41.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:41.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:41.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:51:41.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:42.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:42.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:42.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:42.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:51:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:51:41] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-13T14:51:42.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:42.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:42.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:42.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.543 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:42.672 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:42.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:42.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:42.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:42.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:42.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:42.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:43.018 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2024-09-13T14:51:38.974362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4566685693,"num_objects":69638,"num_object_clones":0,"num_object_copies":208914,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":15683,"num_objects_misplaced":4136,"num_objects_unfound":0,"num_objects_dirty":69638,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":103061,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27496,"num_bytes_recovered":1767459973,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101592,"ondisk_log_size":101592,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":293,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15924288,"kb_used_data":14171656,"kb_used_omap":32,"kb_used_meta":1752543,"kb_avail":640291776,"statfs":{"total":671965249536,"available":655658778624,"internally_reserved":0,"allocated":14511775744,"data_stored":14502932001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1794604592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-204668928,"num_objects":-3123,"num_object_clones":0,"num_object_copies":-9369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4629,"num_objects_misplaced":4136,"num_objects_unfound":0,"num_objects_dirty":-3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3123,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":28704768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002715"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":337,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.896394+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:51:34.896394+0000","last_peered":"2024-09-13T14:51:34.896394+0000","last_clean":"2024-09-13T14:51:34.896394+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:51:34.896394+0000","last_undegraded":"2024-09-13T14:51:34.896394+0000","last_fullsized":"2024-09-13T14:51:34.896394+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":92,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793426+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:51:27.793426+0000","last_peered":"2024-09-13T14:51:27.793426+0000","last_clean":"2024-09-13T14:51:27.793426+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:51:27.793426+0000","last_undegraded":"2024-09-13T14:51:27.793426+0000","last_fullsized":"2024-09-13T14:51:27.793426+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":62,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852817+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:51:27.852817+0000","last_peered":"2024-09-13T14:51:27.852817+0000","last_clean":"2024-09-13T14:51:27.852817+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:51:27.852817+0000","last_undegraded":"2024-09-13T14:51:27.852817+0000","last_fullsized":"2024-09-13T14:51:27.852817+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":326,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.831384+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:51:34.831384+0000","last_peered":"2024-09-13T14:51:34.831384+0000","last_clean":"2024-09-13T14:51:34.831384+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:51:34.831384+0000","last_undegraded":"2024-09-13T14:51:34.831384+0000","last_fullsized":"2024-09-13T14:51:34.831384+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":326,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.832057+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:51:34.832057+0000","last_peered":"2024-09-13T14:51:34.832057+0000","last_clean":"2024-09-13T14:51:34.832057+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:51:34.832057+0000","last_undegraded":"2024-09-13T14:51:34.832057+0000","last_fullsized":"2024-09-13T14:51:34.832057+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":162,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.792911+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:51:27.792911+0000","last_peered":"2024-09-13T14:51:27.792911+0000","last_clean":"2024-09-13T14:51:27.792911+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:51:27.792911+0000","last_undegraded":"2024-09-13T14:51:27.792911+0000","last_fullsized":"2024-09-13T14:51:27.792911+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":52,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.828064+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:51:34.828064+0000","last_peered":"2024-09-13T14:51:34.828064+0000","last_clean":"2024-09-13T14:51:34.828064+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:51:34.828064+0000","last_undegraded":"2024-09-13T14:51:34.828064+0000","last_fullsized":"2024-09-13T14:51:34.828064+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":139,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.824907+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:51:28.824907+0000","last_peered":"2024-09-13T14:51:28.824907+0000","last_clean":"2024-09-13T14:51:28.824907+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:51:28.824907+0000","last_undegraded":"2024-09-13T14:51:28.824907+0000","last_fullsized":"2024-09-13T14:51:28.824907+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"227'6460","reported_seq":7280,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.503913+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:51:34.503913+0000","last_peered":"2024-09-13T14:51:34.503913+0000","last_clean":"2024-09-13T14:51:34.503913+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:51:34.503913+0000","last_undegraded":"2024-09-13T14:51:34.503913+0000","last_fullsized":"2024-09-13T14:51:34.503913+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290717696,"num_objects":4436,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":259,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.803304+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:51:28.803304+0000","last_peered":"2024-09-13T14:51:28.803304+0000","last_clean":"2024-09-13T14:51:28.803304+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:51:28.803304+0000","last_undegraded":"2024-09-13T14:51:28.803304+0000","last_fullsized":"2024-09-13T14:51:28.803304+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":89,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.853015+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:51:27.853015+0000","last_peered":"2024-09-13T14:51:27.853015+0000","last_clean":"2024-09-13T14:51:27.853015+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:51:27.853015+0000","last_undegraded":"2024-09-13T14:51:27.853015+0000","last_fullsized":"2024-09-13T14:51:27.853015+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"228'6316","reported_seq":7900,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:38.667594+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:51:38.667594+0000","last_peered":"2024-09-13T14:51:38.667594+0000","last_clean":"2024-09-13T14:51:38.667594+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:51:38.667594+0000","last_undegraded":"2024-09-13T14:51:38.667594+0000","last_fullsized":"2024-09-13T14:51:38.667594+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279314432,"num_objects":4262,"num_object_clones":0,"num_object_copies":12786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":90,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.796728+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:51:27.796728+0000","last_peered":"2024-09-13T14:51:27.796728+0000","last_clean":"2024-09-13T14:51:27.796728+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:51:27.796728+0000","last_undegraded":"2024-09-13T14:51:27.796728+0000","last_fullsized":"2024-09-13T14:51:27.796728+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":242,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793097+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:51:27.793097+0000","last_peered":"2024-09-13T14:51:27.793097+0000","last_clean":"2024-09-13T14:51:27.793097+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:51:27.793097+0000","last_undegraded":"2024-09-13T14:51:27.793097+0000","last_fullsized":"2024-09-13T14:51:27.793097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"227'6432","reported_seq":8107,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.159518+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:51:34.159518+0000","last_peered":"2024-09-13T14:51:34.159518+0000","last_clean":"2024-09-13T14:51:34.159518+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:51:34.159518+0000","last_undegraded":"2024-09-13T14:51:34.159518+0000","last_fullsized":"2024-09-13T14:51:34.159518+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6432,"log_dups_size":0,"ondisk_log_size":6432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289144832,"num_objects":4412,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6432,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":66,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.853001+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:51:27.853001+0000","last_peered":"2024-09-13T14:51:27.853001+0000","last_clean":"2024-09-13T14:51:27.853001+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:51:27.853001+0000","last_undegraded":"2024-09-13T14:51:27.853001+0000","last_fullsized":"2024-09-13T14:51:27.853001+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.825027+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:51:28.825027+0000","last_peered":"2024-09-13T14:51:28.825027+0000","last_clean":"2024-09-13T14:51:28.825027+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:51:28.825027+0000","last_undegraded":"2024-09-13T14:51:28.825027+0000","last_fullsized":"2024-09-13T14:51:28.825027+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"228'6280","reported_seq":8017,"reported_epoch":228,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:37.841106+0000","last_change":"2024-09-13T14:50:07.839326+0000","last_active":"2024-09-13T14:51:37.841106+0000","last_peered":"2024-09-13T14:51:37.841106+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:51:37.841106+0000","last_undegraded":"2024-09-13T14:50:07.570096+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6280,"log_dups_size":0,"ondisk_log_size":6280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277086208,"num_objects":4228,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6280,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":51445760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":2127},{"shards":"0,3,6","objects":2101}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":78,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852710+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:51:27.852710+0000","last_peered":"2024-09-13T14:51:27.852710+0000","last_clean":"2024-09-13T14:51:27.852710+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:51:27.852710+0000","last_undegraded":"2024-09-13T14:51:27.852710+0000","last_fullsized":"2024-09-13T14:51:27.852710+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":306,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.831725+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:51:34.831725+0000","last_peered":"2024-09-13T14:51:34.831725+0000","last_clean":"2024-09-13T14:51:34.831725+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:51:34.831725+0000","last_undegraded":"2024-09-13T14:51:34.831725+0000","last_fullsized":"2024-09-13T14:51:34.831725+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"227'6352","reported_seq":13622,"reported_epoch":227,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:34.129541+0000","last_change":"2024-09-13T14:51:27.441494+0000","last_active":"2024-09-13T14:51:34.129541+0000","last_peered":"2024-09-13T14:51:34.129541+0000","last_clean":"2024-09-13T14:49:59.725669+0000","last_became_active":"2024-09-13T14:50:08.707516+0000","last_became_peered":"2024-09-13T14:50:08.707516+0000","last_unstale":"2024-09-13T14:51:34.129541+0000","last_undegraded":"2024-09-13T14:50:08.570515+0000","last_fullsized":"2024-09-13T14:50:08.544112+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6352,"log_dups_size":0,"ondisk_log_size":6352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:20.218839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288489472,"num_objects":4402,"num_object_clones":0,"num_object_copies":13206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":4136,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3583,"num_bytes_recovered":234815488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-522},{"shards":"1,4,5","objects":4136}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":314,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.827811+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:51:34.827811+0000","last_peered":"2024-09-13T14:51:34.827811+0000","last_clean":"2024-09-13T14:51:34.827811+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:51:34.827811+0000","last_undegraded":"2024-09-13T14:51:34.827811+0000","last_fullsized":"2024-09-13T14:51:34.827811+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":209,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793757+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:51:27.793757+0000","last_peered":"2024-09-13T14:51:27.793757+0000","last_clean":"2024-09-13T14:51:27.793757+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:51:27.793757+0000","last_undegraded":"2024-09-13T14:51:27.793757+0000","last_fullsized":"2024-09-13T14:51:27.793757+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"227'6278","reported_seq":6648,"reported_epoch":227,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:34.031278+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:51:34.031278+0000","last_peered":"2024-09-13T14:51:34.031278+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:51:34.031278+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285868032,"num_objects":4362,"num_object_clones":0,"num_object_copies":13086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":2182},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":111,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.863850+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:51:28.863850+0000","last_peered":"2024-09-13T14:51:28.863850+0000","last_clean":"2024-09-13T14:51:28.863850+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:51:28.863850+0000","last_undegraded":"2024-09-13T14:51:28.863850+0000","last_fullsized":"2024-09-13T14:51:28.863850+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":206,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.794074+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:51:27.794074+0000","last_peered":"2024-09-13T14:51:27.794074+0000","last_clean":"2024-09-13T14:51:27.794074+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:51:27.794074+0000","last_undegraded":"2024-09-13T14:51:27.794074+0000","last_fullsized":"2024-09-13T14:51:27.794074+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"228'6499","reported_seq":8702,"reported_epoch":228,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:38.762433+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:51:38.762433+0000","last_peered":"2024-09-13T14:51:38.762433+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:51:38.762433+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"164'100","ondisk_log_start":"164'100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283312128,"num_objects":4323,"num_object_clones":0,"num_object_copies":12969,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":3609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6499,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":75825152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":1208},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":201,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793172+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:51:27.793172+0000","last_peered":"2024-09-13T14:51:27.793172+0000","last_clean":"2024-09-13T14:51:27.793172+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:51:27.793172+0000","last_undegraded":"2024-09-13T14:51:27.793172+0000","last_fullsized":"2024-09-13T14:51:27.793172+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":236,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.796774+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:51:27.796774+0000","last_peered":"2024-09-13T14:51:27.796774+0000","last_clean":"2024-09-13T14:51:27.796774+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:51:27.796774+0000","last_undegraded":"2024-09-13T14:51:27.796774+0000","last_fullsized":"2024-09-13T14:51:27.796774+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"227'6192","reported_seq":10606,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.442356+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:51:34.442356+0000","last_peered":"2024-09-13T14:51:34.442356+0000","last_clean":"2024-09-13T14:51:34.442356+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:51:34.442356+0000","last_undegraded":"2024-09-13T14:51:34.442356+0000","last_fullsized":"2024-09-13T14:51:34.442356+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282656792,"num_objects":4314,"num_object_clones":0,"num_object_copies":12942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6192,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":52,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.826453+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:51:34.826453+0000","last_peered":"2024-09-13T14:51:34.826453+0000","last_clean":"2024-09-13T14:51:34.826453+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:51:34.826453+0000","last_undegraded":"2024-09-13T14:51:34.826453+0000","last_fullsized":"2024-09-13T14:51:34.826453+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":159,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.803325+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:51:28.803325+0000","last_peered":"2024-09-13T14:51:28.803325+0000","last_clean":"2024-09-13T14:51:28.803325+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:51:28.803325+0000","last_undegraded":"2024-09-13T14:51:28.803325+0000","last_fullsized":"2024-09-13T14:51:28.803325+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"227'6277","reported_seq":7512,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.506779+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:51:34.506779+0000","last_peered":"2024-09-13T14:51:34.506779+0000","last_clean":"2024-09-13T14:51:34.506779+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:51:34.506779+0000","last_undegraded":"2024-09-13T14:51:34.506779+0000","last_fullsized":"2024-09-13T14:51:34.506779+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6112,"log_dups_size":0,"ondisk_log_size":6112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292749312,"num_objects":4467,"num_object_clones":0,"num_object_copies":13401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6277,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":508,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.835692+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:51:34.835692+0000","last_peered":"2024-09-13T14:51:34.835692+0000","last_clean":"2024-09-13T14:51:34.835692+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:51:34.835692+0000","last_undegraded":"2024-09-13T14:51:34.835692+0000","last_fullsized":"2024-09-13T14:51:34.835692+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":793,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852494+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:51:27.852494+0000","last_peered":"2024-09-13T14:51:27.852494+0000","last_clean":"2024-09-13T14:51:27.852494+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:51:27.852494+0000","last_undegraded":"2024-09-13T14:51:27.852494+0000","last_fullsized":"2024-09-13T14:51:27.852494+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":272,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.827555+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:51:34.827555+0000","last_peered":"2024-09-13T14:51:34.827555+0000","last_clean":"2024-09-13T14:51:34.827555+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:51:34.827555+0000","last_undegraded":"2024-09-13T14:51:34.827555+0000","last_fullsized":"2024-09-13T14:51:34.827555+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":23,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.825334+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:51:28.825334+0000","last_peered":"2024-09-13T14:51:28.825334+0000","last_clean":"2024-09-13T14:51:28.825334+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:51:28.825334+0000","last_undegraded":"2024-09-13T14:51:28.825334+0000","last_fullsized":"2024-09-13T14:51:28.825334+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"227'6431","reported_seq":11127,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.679798+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:51:34.679798+0000","last_peered":"2024-09-13T14:51:34.679798+0000","last_clean":"2024-09-13T14:51:34.679798+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:51:34.679798+0000","last_undegraded":"2024-09-13T14:51:34.679798+0000","last_fullsized":"2024-09-13T14:51:34.679798+0000","mapping_epoch":223,"log_start":"164'618","ondisk_log_start":"164'618","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289472512,"num_objects":4417,"num_object_clones":0,"num_object_copies":13251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6431,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.822181+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:51:28.822181+0000","last_peered":"2024-09-13T14:51:28.822181+0000","last_clean":"2024-09-13T14:51:28.822181+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:51:28.822181+0000","last_undegraded":"2024-09-13T14:51:28.822181+0000","last_fullsized":"2024-09-13T14:51:28.822181+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":320,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.832999+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:51:34.832999+0000","last_peered":"2024-09-13T14:51:34.832999+0000","last_clean":"2024-09-13T14:51:34.832999+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:51:34.832999+0000","last_undegraded":"2024-09-13T14:51:34.832999+0000","last_fullsized":"2024-09-13T14:51:34.832999+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":331,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.832258+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:51:34.832258+0000","last_peered":"2024-09-13T14:51:34.832258+0000","last_clean":"2024-09-13T14:51:34.832258+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:51:34.832258+0000","last_undegraded":"2024-09-13T14:51:34.832258+0000","last_fullsized":"2024-09-13T14:51:34.832258+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"228'6354","reported_seq":12200,"reported_epoch":228,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:38.802223+0000","last_change":"2024-09-13T14:50:38.024593+0000","last_active":"2024-09-13T14:51:38.802223+0000","last_peered":"2024-09-13T14:51:38.802223+0000","last_clean":"2024-09-13T14:48:36.189224+0000","last_became_active":"2024-09-13T14:50:38.015505+0000","last_became_peered":"2024-09-13T14:50:38.015505+0000","last_unstale":"2024-09-13T14:51:38.802223+0000","last_undegraded":"2024-09-13T14:50:37.965648+0000","last_fullsized":"2024-09-13T14:50:37.942220+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278003712,"num_objects":4242,"num_object_clones":0,"num_object_copies":12726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6354,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2845,"num_bytes_recovered":186449920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":2181},{"shards":"2,3,5","objects":2061}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":854,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.803052+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:51:28.803052+0000","last_peered":"2024-09-13T14:51:28.803052+0000","last_clean":"2024-09-13T14:51:28.803052+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:51:28.803052+0000","last_undegraded":"2024-09-13T14:51:28.803052+0000","last_fullsized":"2024-09-13T14:51:28.803052+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":309,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.822410+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:51:28.822410+0000","last_peered":"2024-09-13T14:51:28.822410+0000","last_clean":"2024-09-13T14:51:28.822410+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:51:28.822410+0000","last_undegraded":"2024-09-13T14:51:28.822410+0000","last_fullsized":"2024-09-13T14:51:28.822410+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":62,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.823259+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:51:28.823259+0000","last_peered":"2024-09-13T14:51:28.823259+0000","last_clean":"2024-09-13T14:51:28.823259+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:51:28.823259+0000","last_undegraded":"2024-09-13T14:51:28.823259+0000","last_fullsized":"2024-09-13T14:51:28.823259+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"228'6460","reported_seq":11048,"reported_epoch":228,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:38.658443+0000","last_change":"2024-09-13T14:50:42.308313+0000","last_active":"2024-09-13T14:51:38.658443+0000","last_peered":"2024-09-13T14:51:38.658443+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:51:38.658443+0000","last_undegraded":"2024-09-13T14:50:40.576119+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279183360,"num_objects":4260,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":89,"num_objects_missing":89,"num_objects_degraded":2392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":148504576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"1,4,5","objects":9},{"shards":"3,4","objects":2143},{"shards":"3,4,5","objects":2108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":80,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852605+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:51:27.852605+0000","last_peered":"2024-09-13T14:51:27.852605+0000","last_clean":"2024-09-13T14:51:27.852605+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:51:27.852605+0000","last_undegraded":"2024-09-13T14:51:27.852605+0000","last_fullsized":"2024-09-13T14:51:27.852605+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":211,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793646+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:51:27.793646+0000","last_peered":"2024-09-13T14:51:27.793646+0000","last_clean":"2024-09-13T14:51:27.793646+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:51:27.793646+0000","last_undegraded":"2024-09-13T14:51:27.793646+0000","last_fullsized":"2024-09-13T14:51:27.793646+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":312,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.823325+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:51:28.823325+0000","last_peered":"2024-09-13T14:51:28.823325+0000","last_clean":"2024-09-13T14:51:28.823325+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:51:28.823325+0000","last_undegraded":"2024-09-13T14:51:28.823325+0000","last_fullsized":"2024-09-13T14:51:28.823325+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"228'6500","reported_seq":11062,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:37.661498+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:51:37.661498+0000","last_peered":"2024-09-13T14:51:37.661498+0000","last_clean":"2024-09-13T14:51:37.661498+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:51:37.661498+0000","last_undegraded":"2024-09-13T14:51:37.661498+0000","last_fullsized":"2024-09-13T14:51:37.661498+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6500,"log_dups_size":0,"ondisk_log_size":6500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282722304,"num_objects":4314,"num_object_clones":0,"num_object_copies":12942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6500,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":80,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852741+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:51:27.852741+0000","last_peered":"2024-09-13T14:51:27.852741+0000","last_clean":"2024-09-13T14:51:27.852741+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:51:27.852741+0000","last_undegraded":"2024-09-13T14:51:27.852741+0000","last_fullsized":"2024-09-13T14:51:27.852741+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":864,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.823517+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:51:28.823517+0000","last_peered":"2024-09-13T14:51:28.823517+0000","last_clean":"2024-09-13T14:51:28.823517+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:51:28.823517+0000","last_undegraded":"2024-09-13T14:51:28.823517+0000","last_fullsized":"2024-09-13T14:51:28.823517+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":289,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.821993+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:51:28.821993+0000","last_peered":"2024-09-13T14:51:28.821993+0000","last_clean":"2024-09-13T14:51:28.821993+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:51:28.821993+0000","last_undegraded":"2024-09-13T14:51:28.821993+0000","last_fullsized":"2024-09-13T14:51:28.821993+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"227'6467","reported_seq":11155,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.550354+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:51:34.550354+0000","last_peered":"2024-09-13T14:51:34.550354+0000","last_clean":"2024-09-13T14:51:34.550354+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:51:34.550354+0000","last_undegraded":"2024-09-13T14:51:34.550354+0000","last_fullsized":"2024-09-13T14:51:34.550354+0000","mapping_epoch":218,"log_start":"164'645","ondisk_log_start":"164'645","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5822,"log_dups_size":0,"ondisk_log_size":5822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291045376,"num_objects":4441,"num_object_clones":0,"num_object_copies":13323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":316,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.823422+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:51:28.823422+0000","last_peered":"2024-09-13T14:51:28.823422+0000","last_clean":"2024-09-13T14:51:28.823422+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:51:28.823422+0000","last_undegraded":"2024-09-13T14:51:28.823422+0000","last_fullsized":"2024-09-13T14:51:28.823422+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":539,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.831888+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:51:34.831888+0000","last_peered":"2024-09-13T14:51:34.831888+0000","last_clean":"2024-09-13T14:51:34.831888+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:51:34.831888+0000","last_undegraded":"2024-09-13T14:51:34.831888+0000","last_fullsized":"2024-09-13T14:51:34.831888+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":316,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.831655+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:51:34.831655+0000","last_peered":"2024-09-13T14:51:34.831655+0000","last_clean":"2024-09-13T14:51:34.831655+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:51:34.831655+0000","last_undegraded":"2024-09-13T14:51:34.831655+0000","last_fullsized":"2024-09-13T14:51:34.831655+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"227'6184","reported_seq":8375,"reported_epoch":227,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:34.126767+0000","last_change":"2024-09-13T14:50:19.881564+0000","last_active":"2024-09-13T14:51:34.126767+0000","last_peered":"2024-09-13T14:51:34.126767+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:51:34.126767+0000","last_undegraded":"2024-09-13T14:50:19.795499+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6184,"log_dups_size":0,"ondisk_log_size":6184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283508736,"num_objects":4326,"num_object_clones":0,"num_object_copies":12978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6184,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2235},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2090}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":336,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.828810+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:51:34.828810+0000","last_peered":"2024-09-13T14:51:34.828810+0000","last_clean":"2024-09-13T14:51:34.828810+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:51:34.828810+0000","last_undegraded":"2024-09-13T14:51:34.828810+0000","last_fullsized":"2024-09-13T14:51:34.828810+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":766,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.823984+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:51:28.823984+0000","last_peered":"2024-09-13T14:51:28.823984+0000","last_clean":"2024-09-13T14:51:28.823984+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:51:28.823984+0000","last_undegraded":"2024-09-13T14:51:28.823984+0000","last_fullsized":"2024-09-13T14:51:28.823984+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":52,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.826396+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:51:34.826396+0000","last_peered":"2024-09-13T14:51:34.826396+0000","last_clean":"2024-09-13T14:51:34.826396+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:51:34.826396+0000","last_undegraded":"2024-09-13T14:51:34.826396+0000","last_fullsized":"2024-09-13T14:51:34.826396+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"228'6456","reported_seq":8017,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:38.597738+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:51:38.597738+0000","last_peered":"2024-09-13T14:51:38.597738+0000","last_clean":"2024-09-13T14:51:38.597738+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:51:38.597738+0000","last_undegraded":"2024-09-13T14:51:38.597738+0000","last_fullsized":"2024-09-13T14:51:38.597738+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288620544,"num_objects":4404,"num_object_clones":0,"num_object_copies":13212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6456,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":311,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.831966+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:51:34.831966+0000","last_peered":"2024-09-13T14:51:34.831966+0000","last_clean":"2024-09-13T14:51:34.831966+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:51:34.831966+0000","last_undegraded":"2024-09-13T14:51:34.831966+0000","last_fullsized":"2024-09-13T14:51:34.831966+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":779,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.825290+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:51:28.825290+0000","last_peered":"2024-09-13T14:51:28.825290+0000","last_clean":"2024-09-13T14:51:28.825290+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:51:28.825290+0000","last_undegraded":"2024-09-13T14:51:28.825290+0000","last_fullsized":"2024-09-13T14:51:28.825290+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":213,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.792979+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:51:27.792979+0000","last_peered":"2024-09-13T14:51:27.792979+0000","last_clean":"2024-09-13T14:51:27.792979+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:51:27.792979+0000","last_undegraded":"2024-09-13T14:51:27.792979+0000","last_fullsized":"2024-09-13T14:51:27.792979+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":208,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793566+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:51:27.793566+0000","last_peered":"2024-09-13T14:51:27.793566+0000","last_clean":"2024-09-13T14:51:27.793566+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:51:27.793566+0000","last_undegraded":"2024-09-13T14:51:27.793566+0000","last_fullsized":"2024-09-13T14:51:27.793566+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":256,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793869+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:51:27.793869+0000","last_peered":"2024-09-13T14:51:27.793869+0000","last_clean":"2024-09-13T14:51:27.793869+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:51:27.793869+0000","last_undegraded":"2024-09-13T14:51:27.793869+0000","last_fullsized":"2024-09-13T14:51:27.793869+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":313,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.821842+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:51:28.821842+0000","last_peered":"2024-09-13T14:51:28.821842+0000","last_clean":"2024-09-13T14:51:28.821842+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:51:28.821842+0000","last_undegraded":"2024-09-13T14:51:28.821842+0000","last_fullsized":"2024-09-13T14:51:28.821842+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":156,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.828194+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:51:34.828194+0000","last_peered":"2024-09-13T14:51:34.828194+0000","last_clean":"2024-09-13T14:51:34.828194+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:51:34.828194+0000","last_undegraded":"2024-09-13T14:51:34.828194+0000","last_fullsized":"2024-09-13T14:51:34.828194+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":89,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852751+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:51:27.852751+0000","last_peered":"2024-09-13T14:51:27.852751+0000","last_clean":"2024-09-13T14:51:27.852751+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:51:27.852751+0000","last_undegraded":"2024-09-13T14:51:27.852751+0000","last_fullsized":"2024-09-13T14:51:27.852751+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":293,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.826440+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:51:34.826440+0000","last_peered":"2024-09-13T14:51:34.826440+0000","last_clean":"2024-09-13T14:51:34.826440+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:51:34.826440+0000","last_undegraded":"2024-09-13T14:51:34.826440+0000","last_fullsized":"2024-09-13T14:51:34.826440+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":347,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.832438+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:51:34.832438+0000","last_peered":"2024-09-13T14:51:34.832438+0000","last_clean":"2024-09-13T14:51:34.832438+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:51:34.832438+0000","last_undegraded":"2024-09-13T14:51:34.832438+0000","last_fullsized":"2024-09-13T14:51:34.832438+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":344,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.826648+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:51:34.826648+0000","last_peered":"2024-09-13T14:51:34.826648+0000","last_clean":"2024-09-13T14:51:34.826648+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:51:34.826648+0000","last_undegraded":"2024-09-13T14:51:34.826648+0000","last_fullsized":"2024-09-13T14:51:34.826648+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":120,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.825229+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:51:28.825229+0000","last_peered":"2024-09-13T14:51:28.825229+0000","last_clean":"2024-09-13T14:51:28.825229+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:51:28.825229+0000","last_undegraded":"2024-09-13T14:51:28.825229+0000","last_fullsized":"2024-09-13T14:51:28.825229+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":311,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.826721+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:51:34.826721+0000","last_peered":"2024-09-13T14:51:34.826721+0000","last_clean":"2024-09-13T14:51:34.826721+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:51:34.826721+0000","last_undegraded":"2024-09-13T14:51:34.826721+0000","last_fullsized":"2024-09-13T14:51:34.826721+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":327,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.832185+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:51:34.832185+0000","last_peered":"2024-09-13T14:51:34.832185+0000","last_clean":"2024-09-13T14:51:34.832185+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:51:34.832185+0000","last_undegraded":"2024-09-13T14:51:34.832185+0000","last_fullsized":"2024-09-13T14:51:34.832185+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":294,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.822758+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:51:28.822758+0000","last_peered":"2024-09-13T14:51:28.822758+0000","last_clean":"2024-09-13T14:51:28.822758+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:51:28.822758+0000","last_undegraded":"2024-09-13T14:51:28.822758+0000","last_fullsized":"2024-09-13T14:51:28.822758+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":159,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.828040+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:51:34.828040+0000","last_peered":"2024-09-13T14:51:34.828040+0000","last_clean":"2024-09-13T14:51:34.828040+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:51:34.828040+0000","last_undegraded":"2024-09-13T14:51:34.828040+0000","last_fullsized":"2024-09-13T14:51:34.828040+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":207,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793743+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:51:27.793743+0000","last_peered":"2024-09-13T14:51:27.793743+0000","last_clean":"2024-09-13T14:51:27.793743+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:51:27.793743+0000","last_undegraded":"2024-09-13T14:51:27.793743+0000","last_fullsized":"2024-09-13T14:51:27.793743+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-13T14:51:28.803426+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:51:28.803426+0000","last_peered":"2024-09-13T14:51:28.803426+0000","last_clean":"2024-09-13T14:51:28.803426+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:51:28.803426+0000","last_undegraded":"2024-09-13T14:51:28.803426+0000","last_fullsized":"2024-09-13T14:51:28.803426+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":320,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.831566+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:51:34.831566+0000","last_peered":"2024-09-13T14:51:34.831566+0000","last_clean":"2024-09-13T14:51:34.831566+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:51:34.831566+0000","last_undegraded":"2024-09-13T14:51:34.831566+0000","last_fullsized":"2024-09-13T14:51:34.831566+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":208,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793498+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:51:27.793498+0000","last_peered":"2024-09-13T14:51:27.793498+0000","last_clean":"2024-09-13T14:51:27.793498+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:51:27.793498+0000","last_undegraded":"2024-09-13T14:51:27.793498+0000","last_fullsized":"2024-09-13T14:51:27.793498+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":64,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852578+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:51:27.852578+0000","last_peered":"2024-09-13T14:51:27.852578+0000","last_clean":"2024-09-13T14:51:27.852578+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:51:27.852578+0000","last_undegraded":"2024-09-13T14:51:27.852578+0000","last_fullsized":"2024-09-13T14:51:27.852578+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":205,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.793658+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:51:27.793658+0000","last_peered":"2024-09-13T14:51:27.793658+0000","last_clean":"2024-09-13T14:51:27.793658+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:51:27.793658+0000","last_undegraded":"2024-09-13T14:51:27.793658+0000","last_fullsized":"2024-09-13T14:51:27.793658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":66,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.852901+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:51:27.852901+0000","last_peered":"2024-09-13T14:51:27.852901+0000","last_clean":"2024-09-13T14:51:27.852901+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:51:27.852901+0000","last_undegraded":"2024-09-13T14:51:27.852901+0000","last_fullsized":"2024-09-13T14:51:27.852901+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":314,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.827239+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:51:34.827239+0000","last_peered":"2024-09-13T14:51:34.827239+0000","last_clean":"2024-09-13T14:51:34.827239+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:51:34.827239+0000","last_undegraded":"2024-09-13T14:51:34.827239+0000","last_fullsized":"2024-09-13T14:51:34.827239+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":330,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.828984+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:51:34.828984+0000","last_peered":"2024-09-13T14:51:34.828984+0000","last_clean":"2024-09-13T14:51:34.828984+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:51:34.828984+0000","last_undegraded":"2024-09-13T14:51:34.828984+0000","last_fullsized":"2024-09-13T14:51:34.828984+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":235,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-13T14:51:27.792766+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:51:27.792766+0000","last_peered":"2024-09-13T14:51:27.792766+0000","last_clean":"2024-09-13T14:51:27.792766+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:51:27.792766+0000","last_undegraded":"2024-09-13T14:51:27.792766+0000","last_fullsized":"2024-09-13T14:51:27.792766+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":383,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-13T14:51:34.832351+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:51:34.832351+0000","last_peered":"2024-09-13T14:51:34.832351+0000","last_clean":"2024-09-13T14:51:34.832351+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:51:34.832351+0000","last_undegraded":"2024-09-13T14:51:34.832351+0000","last_fullsized":"2024-09-13T14:51:34.832351+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4561895448,"num_objects":69610,"num_object_clones":0,"num_object_copies":208830,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":15683,"num_objects_misplaced":4136,"num_objects_unfound":0,"num_objects_dirty":69610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101938,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27340,"num_bytes_recovered":1765081088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14559686656,"data_stored":14559674440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100410,"ondisk_log_size":100410,"up":48,"acting":41,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234290,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495927,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2504348,"kb_used_data":2260756,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91240804,"statfs":{"total":95995035648,"available":93430583296,"internally_reserved":0,"allocated":2315014144,"data_stored":2313747928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":249428423},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757561,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2405384,"kb_used_data":2221696,"kb_used_omap":1,"kb_used_meta":183678,"kb_avail":91339768,"statfs":{"total":95995035648,"available":93531922432,"internally_reserved":0,"allocated":2275016704,"data_stored":2273758140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":188086727},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019201,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1695184,"kb_used_data":1411400,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":92049968,"statfs":{"total":95995035648,"available":94259167232,"internally_reserved":0,"allocated":1445273600,"data_stored":1444011302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290585030},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468087,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3125136,"kb_used_data":2922760,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":90620016,"statfs":{"total":95995035648,"available":92794896384,"internally_reserved":0,"allocated":2992906240,"data_stored":2991647665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":207219876},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631615,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2334868,"kb_used_data":2001932,"kb_used_omap":8,"kb_used_meta":332919,"kb_avail":91410284,"statfs":{"total":95995035648,"available":93604130816,"internally_reserved":0,"allocated":2049978368,"data_stored":2048712500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":340910047},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795138,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2330124,"kb_used_data":2104068,"kb_used_omap":7,"kb_used_meta":226040,"kb_avail":91415028,"statfs":{"total":95995035648,"available":93608988672,"internally_reserved":0,"allocated":2154565632,"data_stored":2153303171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":231465586},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958669,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529244,"kb_used_data":1249044,"kb_used_omap":7,"kb_used_meta":280184,"kb_avail":92215908,"statfs":{"total":95995035648,"available":94429089792,"internally_reserved":0,"allocated":1279021056,"data_stored":1277751295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":286908903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1271074816,"data_stored":1271070744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2142507008,"data_stored":2142502936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2039611392,"data_stored":2039611392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2978152448,"data_stored":2978152448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1437335552,"data_stored":1437335552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2270957568,"data_stored":2270953496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2309095424,"data_stored":2309095424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":110952448,"data_stored":110952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:51:43.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:51:43.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:43.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:43.113 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:42 smithi154 ceph-mon[102314]: pgmap v666: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 291 op/s; 15683/206961 objects degraded (7.578%); 3782/206961 objects misplaced (1.827%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:51:43.114 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:42 smithi154 ceph-mon[107038]: pgmap v666: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 291 op/s; 15683/206961 objects degraded (7.578%); 3782/206961 objects misplaced (1.827%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:51:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:43.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:43.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:42 smithi172 ceph-mon[100696]: pgmap v666: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 291 op/s; 15683/206961 objects degraded (7.578%); 3782/206961 objects misplaced (1.827%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:51:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:43.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:43.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:43.364 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:43.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:43.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:43.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:43.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:43.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:43.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:43.833 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:51:43.833 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:43.862 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:51:43.863 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:43.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:43.882 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:51:43.882 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:43.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:43.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:43.922 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:51:43.922 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:43.954 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:51:43.954 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:43.985 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:51:43.986 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:43.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:44.017 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:51:44.018 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:44.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:44.048 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:51:44.049 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:44.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:44.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:44.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[102314]: pgmap v667: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 290 op/s; 15683/206961 objects degraded (7.578%); 3782/206961 objects misplaced (1.827%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:51:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/811863497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[107038]: pgmap v667: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 290 op/s; 15683/206961 objects degraded (7.578%); 3782/206961 objects misplaced (1.827%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:51:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/811863497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:43 smithi172 ceph-mon[100696]: pgmap v667: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 290 op/s; 15683/206961 objects degraded (7.578%); 3782/206961 objects misplaced (1.827%); 4.3 MiB/s, 68 objects/s recovering 2024-09-13T14:51:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/811863497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:44.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:44.277 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:44.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:44.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:44.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:44.530 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:44.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:44.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:44.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:44.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:44.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:44.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:44.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:45.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:45.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:45.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:45.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:45.241 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15683/206961 objects degraded (7.578%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15683/206961 objects degraded (7.578%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:45.242 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15683/206961 objects degraded (7.578%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:45.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:45.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:45.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:45.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:45.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:45.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:45.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:45.663 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:45.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:45.792 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:45.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:45.793 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:45.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:45.918 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:45.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:46.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:46.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:46.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:46.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:45 smithi172 ceph-mon[100696]: pgmap v668: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 219 op/s; 15615/203964 objects degraded (7.656%); 3393/203964 objects misplaced (1.664%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:51:46.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:46.290 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:45 smithi154 ceph-mon[102314]: pgmap v668: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 219 op/s; 15615/203964 objects degraded (7.656%); 3393/203964 objects misplaced (1.664%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:51:46.291 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:45 smithi154 ceph-mon[107038]: pgmap v668: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 219 op/s; 15615/203964 objects degraded (7.656%); 3393/203964 objects misplaced (1.664%); 5.2 MiB/s, 83 objects/s recovering 2024-09-13T14:51:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.292 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:46.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:46.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:46.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:46.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:46.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:46.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.665 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:46.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:46.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:46.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:46.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:46.911 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:46.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:46.991 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:51:47.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:47.036 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:47.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:51:47.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:47.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:46 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3951641952' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:46 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:47.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:47.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:47.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:47.419 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:46 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3951641952' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:46 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:47.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:46 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3951641952' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:46 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:51:47.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:47.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:47.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:47.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:47.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:47.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:47.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:47.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:47.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:48.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:48.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:48.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:48.173 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[102314]: pgmap v669: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 185 op/s; 15615/203964 objects degraded (7.656%); 3393/203964 objects misplaced (1.664%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:51:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[102314]: osdmap e229: 8 total, 8 up, 7 in 2024-09-13T14:51:48.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[107038]: pgmap v669: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 185 op/s; 15615/203964 objects degraded (7.656%); 3393/203964 objects misplaced (1.664%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:51:48.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:48.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:48.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:47 smithi154 ceph-mon[107038]: osdmap e229: 8 total, 8 up, 7 in 2024-09-13T14:51:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:47 smithi172 ceph-mon[100696]: pgmap v669: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 185 op/s; 15615/203964 objects degraded (7.656%); 3393/203964 objects misplaced (1.664%); 4.4 MiB/s, 70 objects/s recovering 2024-09-13T14:51:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:47 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:47 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:51:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:47 smithi172 ceph-mon[100696]: osdmap e229: 8 total, 8 up, 7 in 2024-09-13T14:51:48.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:48.300 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:48.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.427 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:48.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:48.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.554 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:48.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:48.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:48.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:48.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.805 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:48.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:48.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:48.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:48.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:49.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:49.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:49.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:49.181 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:49.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:49.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:49.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:49.437 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:49.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:49.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:49.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:49.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:49.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:49.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:49.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:49.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:49.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:50.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:50.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:50.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:50.190 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:49 smithi154 ceph-mon[102314]: pgmap v671: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 207 op/s; 15547/202677 objects degraded (7.671%); 3393/202677 objects misplaced (1.674%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:51:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:49 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15547/202677 objects degraded (7.671%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:49 smithi154 ceph-mon[107038]: pgmap v671: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 207 op/s; 15547/202677 objects degraded (7.671%); 3393/202677 objects misplaced (1.674%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:51:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:49 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15547/202677 objects degraded (7.671%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:49 smithi172 ceph-mon[100696]: pgmap v671: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 207 op/s; 15547/202677 objects degraded (7.671%); 3393/202677 objects misplaced (1.674%); 5.5 MiB/s, 87 objects/s recovering 2024-09-13T14:51:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:49 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15547/202677 objects degraded (7.671%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:50.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:50.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:50.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:50.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:50.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:50.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:50.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:50.693 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:50.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:50.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:50.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:50.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:50.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:50.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:51.014 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:51.027 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:51:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:51:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:51:51.027 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:51:51.027 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:51:51.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:51.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:51.093 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:51:51.094 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15547/201084 objects degraded (7.732%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":926,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":229,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238500,"num_in_osds":7,"osd_in_since":1726238900,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":82},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":67028,"data_bytes":4395636733,"bytes_used":15859732480,"bytes_avail":656105517056,"bytes_total":671965249536,"degraded_objects":15547,"degraded_total":201084,"degraded_ratio":0.077315947564202028,"misplaced_objects":2952,"misplaced_total":201084,"misplaced_ratio":0.014680432058244314,"recovering_objects_per_sec":96,"recovering_bytes_per_sec":6327485,"recovering_keys_per_sec":0,"num_objects_recovered":966,"num_bytes_recovered":63307776,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi154.daekzd","status":"up:active","gid":24539}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"}},"servicemap":{"epoch":25,"modified":"2024-09-13T14:50:26.951855+0000","services":{}},"progress_events":{}} 2024-09-13T14:51:51.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:51.201 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:51.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:51.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:51.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:51.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:51.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:51.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:51.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:51.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:51:51.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:51.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:51.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:51.930 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:51:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:51:51] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-13T14:51:51.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:51.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:51.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:51.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:51:52.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:52.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:52.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:52.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:51 smithi154 ceph-mon[102314]: pgmap v672: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 15547/201084 objects degraded (7.732%); 2952/201084 objects misplaced (1.468%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:51:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3607146394' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:52.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:51 smithi154 ceph-mon[107038]: pgmap v672: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 15547/201084 objects degraded (7.732%); 2952/201084 objects misplaced (1.468%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:51:52.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3607146394' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:51 smithi172 ceph-mon[100696]: pgmap v672: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 15547/201084 objects degraded (7.732%); 2952/201084 objects misplaced (1.468%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:51:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3607146394' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:51:52.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:52.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:52.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:52.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:52.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:52.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:52.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:52.703 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:51:52.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:51:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:52.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:52.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:52.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:52.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:52.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:52.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:53.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:53.087 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:53.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:53.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:53.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:53.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:53.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:53.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:53.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:53.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:53.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:53.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:53.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:53.844 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:53.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:53.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:53.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:54.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:54.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.099 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:54.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:54.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:54.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:53 smithi172 ceph-mon[100696]: pgmap v673: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 208 op/s; 15547/200715 objects degraded (7.746%); 2952/200715 objects misplaced (1.471%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:51:54.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:54.346 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:54.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:54.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:53 smithi154 ceph-mon[102314]: pgmap v673: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 208 op/s; 15547/200715 objects degraded (7.746%); 2952/200715 objects misplaced (1.471%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:51:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:53 smithi154 ceph-mon[107038]: pgmap v673: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 208 op/s; 15547/200715 objects degraded (7.746%); 2952/200715 objects misplaced (1.471%); 6.0 MiB/s, 96 objects/s recovering 2024-09-13T14:51:54.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:54.593 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:54.594 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:54.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:54.604 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:54.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:54.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:54.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:54.842 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:54.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:54.967 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:54.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:55.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:55.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:55.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:55.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:55.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:54 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15547/200715 objects degraded (7.746%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:54 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15547/200715 objects degraded (7.746%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:55.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:55.344 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:55.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:55.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:55.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:54 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15547/200715 objects degraded (7.746%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:51:55.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:55.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:55.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:55.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:55.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:55.850 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:55.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:55.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:55.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:56.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:56.103 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:56.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.271 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:55 smithi154 ceph-mon[102314]: pgmap v674: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 15476/198057 objects degraded (7.814%); 2489/198057 objects misplaced (1.257%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:51:56.272 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:55 smithi154 ceph-mon[107038]: pgmap v674: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 15476/198057 objects degraded (7.814%); 2489/198057 objects misplaced (1.257%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:51:56.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:56.353 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:56.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:55 smithi172 ceph-mon[100696]: pgmap v674: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 15476/198057 objects degraded (7.814%); 2489/198057 objects misplaced (1.257%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:51:56.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:56.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.740 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:56.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:56.771 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:51:56.772 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:51:56.789 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:51:56.789 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:51:56.818 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:51:56.819 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:51:56.839 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:51:56.839 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:51:56.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:56.858 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:51:56.859 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:51:56.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:56.880 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:51:56.881 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:51:56.899 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:51:56.900 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:51:56.921 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:51:56.921 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:51:56.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:56.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:56.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:57.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:57.115 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/523291903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:57.115 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/523291903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:57.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:57.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:57.251 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:57.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:57.377 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:57.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:57.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/523291903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:51:57.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:57.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:57.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:57.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:57.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:57.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:57.884 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:57.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:57.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:58.014 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:51:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:58.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:58.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:58.146 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:58.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[102314]: pgmap v675: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 15476/198057 objects degraded (7.814%); 2489/198057 objects misplaced (1.257%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:51:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1243667568' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:58.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[107038]: pgmap v675: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 15476/198057 objects degraded (7.814%); 2489/198057 objects misplaced (1.257%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:51:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1243667568' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:58.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:51:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:58.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:51:58.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:58 smithi172 ceph-mon[100696]: pgmap v675: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 15476/198057 objects degraded (7.814%); 2489/198057 objects misplaced (1.257%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:51:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1243667568' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:51:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:51:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:51:58.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:58.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:58.593 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:51:58.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:58.661 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:58.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:51:58.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:51:58.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:58.918 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:58.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:59.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:51:59.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:51:59.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:51:59.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:51:59.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:51:59.283 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:59 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:59.283 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:59 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:59.283 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:51:59 smithi154 ceph-mon[102314]: osdmap e230: 8 total, 8 up, 7 in 2024-09-13T14:51:59.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:59 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:59.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:59 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:59.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:51:59 smithi154 ceph-mon[107038]: osdmap e230: 8 total, 8 up, 7 in 2024-09-13T14:51:59.304 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:51:59.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:59.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:59 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:51:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:59 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:51:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:51:59 smithi172 ceph-mon[100696]: osdmap e230: 8 total, 8 up, 7 in 2024-09-13T14:51:59.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:51:59.556 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:51:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:51:59.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:59.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:51:59.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:51:59.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:51:59.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:51:59.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:51:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:51:59.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:00.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:00.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:00.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:00.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:00.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:00.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:00.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:00.325 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:00 smithi154 ceph-mon[102314]: pgmap v677: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 174 op/s; 15476/197430 objects degraded (7.839%); 2489/197430 objects misplaced (1.261%); 6.1 MiB/s, 97 objects/s recovering 2024-09-13T14:52:00.325 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15476/197430 objects degraded (7.839%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:00.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:00 smithi154 ceph-mon[107038]: pgmap v677: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 174 op/s; 15476/197430 objects degraded (7.839%); 2489/197430 objects misplaced (1.261%); 6.1 MiB/s, 97 objects/s recovering 2024-09-13T14:52:00.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15476/197430 objects degraded (7.839%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:00.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:00.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:00.538 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:00 smithi172 ceph-mon[100696]: pgmap v677: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 174 op/s; 15476/197430 objects degraded (7.839%); 2489/197430 objects misplaced (1.261%); 6.1 MiB/s, 97 objects/s recovering 2024-09-13T14:52:00.538 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15476/197430 objects degraded (7.839%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:00.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:00.663 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:00.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:00.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:00.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:00.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:00.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:01.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:01.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:01.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:52:01.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:01.177 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:01.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:01.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:01.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:01.422 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:01.422 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:01.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:01.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:01.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:01.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:01.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:01.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.840 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:01.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:01.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:01.965 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:01.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:52:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:52:01] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:52:02.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:02.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:02.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.322 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:02 smithi154 ceph-mon[102314]: pgmap v678: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 213 op/s; 15405/194664 objects degraded (7.914%); 1976/194664 objects misplaced (1.015%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:02.323 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:02 smithi154 ceph-mon[107038]: pgmap v678: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 213 op/s; 15405/194664 objects degraded (7.914%); 1976/194664 objects misplaced (1.015%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:02.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:02.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:02.368 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:52:02.370 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:52:02.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.485 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:02.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:02 smithi172 ceph-mon[100696]: pgmap v678: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 213 op/s; 15405/194664 objects degraded (7.914%); 1976/194664 objects misplaced (1.015%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:02.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:02.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:52:02.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:02.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:02.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:02.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:02.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:02.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:03.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:03.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:03.135 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:03.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:03.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:03.345 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:03 smithi154 ceph-mon[107038]: from='client.35199 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:03.346 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:03 smithi154 ceph-mon[102314]: from='client.35199 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:03.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:03.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:03.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:03.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:03 smithi172 ceph-mon[100696]: from='client.35199 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:03.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:03.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:03.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:03.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:03.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:03.789 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:03.795 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2024-09-13T14:52:00.982910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4255389693,"num_objects":64888,"num_object_clones":0,"num_object_copies":194664,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":15405,"num_objects_misplaced":1976,"num_objects_unfound":0,"num_objects_dirty":64888,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":107811,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29934,"num_bytes_recovered":1927236741,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104742,"ondisk_log_size":104742,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":293,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15144860,"kb_used_data":13370132,"kb_used_omap":32,"kb_used_meta":1774623,"kb_avail":641071204,"statfs":{"total":671965249536,"available":656456912896,"internally_reserved":0,"allocated":13691015168,"data_stored":13682105856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1817214512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,3,1,2,3,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-140247040,"num_objects":-2140,"num_object_clones":0,"num_object_copies":-6420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":-976,"num_objects_unfound":0,"num_objects_dirty":-2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2140,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":73269248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002699"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":341,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.114743+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:51:58.114743+0000","last_peered":"2024-09-13T14:51:58.114743+0000","last_clean":"2024-09-13T14:51:58.114743+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:51:58.114743+0000","last_undegraded":"2024-09-13T14:51:58.114743+0000","last_fullsized":"2024-09-13T14:51:58.114743+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":95,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988398+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:51:46.988398+0000","last_peered":"2024-09-13T14:51:46.988398+0000","last_clean":"2024-09-13T14:51:46.988398+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:51:46.988398+0000","last_undegraded":"2024-09-13T14:51:46.988398+0000","last_fullsized":"2024-09-13T14:51:46.988398+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":65,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036522+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:51:47.036522+0000","last_peered":"2024-09-13T14:51:47.036522+0000","last_clean":"2024-09-13T14:51:47.036522+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:51:47.036522+0000","last_undegraded":"2024-09-13T14:51:47.036522+0000","last_fullsized":"2024-09-13T14:51:47.036522+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":329,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.004136+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:51:47.004136+0000","last_peered":"2024-09-13T14:51:47.004136+0000","last_clean":"2024-09-13T14:51:47.004136+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:51:47.004136+0000","last_undegraded":"2024-09-13T14:51:47.004136+0000","last_fullsized":"2024-09-13T14:51:47.004136+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":328,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003626+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:51:47.003626+0000","last_peered":"2024-09-13T14:51:47.003626+0000","last_clean":"2024-09-13T14:51:47.003626+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:51:47.003626+0000","last_undegraded":"2024-09-13T14:51:47.003626+0000","last_fullsized":"2024-09-13T14:51:47.003626+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":165,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988089+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:51:46.988089+0000","last_peered":"2024-09-13T14:51:46.988089+0000","last_clean":"2024-09-13T14:51:46.988089+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:51:46.988089+0000","last_undegraded":"2024-09-13T14:51:46.988089+0000","last_fullsized":"2024-09-13T14:51:46.988089+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":56,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.018650+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:51:58.018650+0000","last_peered":"2024-09-13T14:51:58.018650+0000","last_clean":"2024-09-13T14:51:58.018650+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:51:58.018650+0000","last_undegraded":"2024-09-13T14:51:58.018650+0000","last_fullsized":"2024-09-13T14:51:58.018650+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":142,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.021554+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:51:58.021554+0000","last_peered":"2024-09-13T14:51:58.021554+0000","last_clean":"2024-09-13T14:51:58.021554+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:51:58.021554+0000","last_undegraded":"2024-09-13T14:51:58.021554+0000","last_fullsized":"2024-09-13T14:51:58.021554+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"230'6760","reported_seq":7584,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.490925+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:51:59.490925+0000","last_peered":"2024-09-13T14:51:59.490925+0000","last_clean":"2024-09-13T14:51:59.490925+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:51:59.490925+0000","last_undegraded":"2024-09-13T14:51:59.490925+0000","last_fullsized":"2024-09-13T14:51:59.490925+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6760,"log_dups_size":0,"ondisk_log_size":6760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271056896,"num_objects":4136,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6760,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":265,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.027338+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:51:58.027338+0000","last_peered":"2024-09-13T14:51:58.027338+0000","last_clean":"2024-09-13T14:51:58.027338+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:51:58.027338+0000","last_undegraded":"2024-09-13T14:51:58.027338+0000","last_fullsized":"2024-09-13T14:51:58.027338+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":92,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036095+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:51:47.036095+0000","last_peered":"2024-09-13T14:51:47.036095+0000","last_clean":"2024-09-13T14:51:47.036095+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:51:47.036095+0000","last_undegraded":"2024-09-13T14:51:47.036095+0000","last_fullsized":"2024-09-13T14:51:47.036095+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"230'6584","reported_seq":8172,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.990560+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:51:58.990560+0000","last_peered":"2024-09-13T14:51:58.990560+0000","last_clean":"2024-09-13T14:51:58.990560+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:51:58.990560+0000","last_undegraded":"2024-09-13T14:51:58.990560+0000","last_fullsized":"2024-09-13T14:51:58.990560+0000","mapping_epoch":170,"log_start":"164'200","ondisk_log_start":"164'200","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261750784,"num_objects":3994,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":93,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.994357+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:51:46.994357+0000","last_peered":"2024-09-13T14:51:46.994357+0000","last_clean":"2024-09-13T14:51:46.994357+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:51:46.994357+0000","last_undegraded":"2024-09-13T14:51:46.994357+0000","last_fullsized":"2024-09-13T14:51:46.994357+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":245,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988270+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:51:46.988270+0000","last_peered":"2024-09-13T14:51:46.988270+0000","last_clean":"2024-09-13T14:51:46.988270+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:51:46.988270+0000","last_undegraded":"2024-09-13T14:51:46.988270+0000","last_fullsized":"2024-09-13T14:51:46.988270+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"230'6759","reported_seq":8440,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.055356+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:51:59.055356+0000","last_peered":"2024-09-13T14:51:59.055356+0000","last_clean":"2024-09-13T14:51:59.055356+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:51:59.055356+0000","last_undegraded":"2024-09-13T14:51:59.055356+0000","last_fullsized":"2024-09-13T14:51:59.055356+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6759,"log_dups_size":0,"ondisk_log_size":6759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267714560,"num_objects":4085,"num_object_clones":0,"num_object_copies":12255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6759,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":69,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036646+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:51:47.036646+0000","last_peered":"2024-09-13T14:51:47.036646+0000","last_clean":"2024-09-13T14:51:47.036646+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:51:47.036646+0000","last_undegraded":"2024-09-13T14:51:47.036646+0000","last_fullsized":"2024-09-13T14:51:47.036646+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.995433+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:51:46.995433+0000","last_peered":"2024-09-13T14:51:46.995433+0000","last_clean":"2024-09-13T14:51:46.995433+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:51:46.995433+0000","last_undegraded":"2024-09-13T14:51:46.995433+0000","last_fullsized":"2024-09-13T14:51:46.995433+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"229'6534","reported_seq":8273,"reported_epoch":229,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:57.816306+0000","last_change":"2024-09-13T14:50:07.839326+0000","last_active":"2024-09-13T14:51:57.816306+0000","last_peered":"2024-09-13T14:51:57.816306+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:51:57.816306+0000","last_undegraded":"2024-09-13T14:50:07.570096+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6534,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":51445760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":1873},{"shards":"0,3,6","objects":2101}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":81,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036312+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:51:47.036312+0000","last_peered":"2024-09-13T14:51:47.036312+0000","last_clean":"2024-09-13T14:51:47.036312+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:51:47.036312+0000","last_undegraded":"2024-09-13T14:51:47.036312+0000","last_fullsized":"2024-09-13T14:51:47.036312+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":308,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003441+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:51:47.003441+0000","last_peered":"2024-09-13T14:51:47.003441+0000","last_clean":"2024-09-13T14:51:47.003441+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:51:47.003441+0000","last_undegraded":"2024-09-13T14:51:47.003441+0000","last_fullsized":"2024-09-13T14:51:47.003441+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"230'6682","reported_seq":18275,"reported_epoch":230,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:59.153973+0000","last_change":"2024-09-13T14:51:27.441494+0000","last_active":"2024-09-13T14:51:59.153973+0000","last_peered":"2024-09-13T14:51:59.153973+0000","last_clean":"2024-09-13T14:49:59.725669+0000","last_became_active":"2024-09-13T14:50:08.707516+0000","last_became_peered":"2024-09-13T14:50:08.707516+0000","last_unstale":"2024-09-13T14:51:59.153973+0000","last_undegraded":"2024-09-13T14:50:08.570515+0000","last_fullsized":"2024-09-13T14:50:08.544112+0000","mapping_epoch":205,"log_start":"164'100","ondisk_log_start":"164'100","created":160,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6582,"log_dups_size":0,"ondisk_log_size":6582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:20.218839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266862592,"num_objects":4072,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":1976,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5743,"num_bytes_recovered":376373248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":1308},{"shards":"1,4,5","objects":1976}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":318,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.018433+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:51:58.018433+0000","last_peered":"2024-09-13T14:51:58.018433+0000","last_clean":"2024-09-13T14:51:58.018433+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:51:58.018433+0000","last_undegraded":"2024-09-13T14:51:58.018433+0000","last_fullsized":"2024-09-13T14:51:58.018433+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":212,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988798+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:51:46.988798+0000","last_peered":"2024-09-13T14:51:46.988798+0000","last_clean":"2024-09-13T14:51:46.988798+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:51:46.988798+0000","last_undegraded":"2024-09-13T14:51:46.988798+0000","last_fullsized":"2024-09-13T14:51:46.988798+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"230'6597","reported_seq":6971,"reported_epoch":230,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:59.045886+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:51:59.045886+0000","last_peered":"2024-09-13T14:51:59.045886+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:51:59.045886+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6597,"log_dups_size":0,"ondisk_log_size":6597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264962048,"num_objects":4043,"num_object_clones":0,"num_object_copies":12129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6597,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":1863},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.995519+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:51:46.995519+0000","last_peered":"2024-09-13T14:51:46.995519+0000","last_clean":"2024-09-13T14:51:46.995519+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:51:46.995519+0000","last_undegraded":"2024-09-13T14:51:46.995519+0000","last_fullsized":"2024-09-13T14:51:46.995519+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":209,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.989037+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:51:46.989037+0000","last_peered":"2024-09-13T14:51:46.989037+0000","last_clean":"2024-09-13T14:51:46.989037+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:51:46.989037+0000","last_undegraded":"2024-09-13T14:51:46.989037+0000","last_fullsized":"2024-09-13T14:51:46.989037+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"230'6777","reported_seq":9542,"reported_epoch":230,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:59.099791+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:51:59.099791+0000","last_peered":"2024-09-13T14:51:59.099791+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:51:59.099791+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"164'300","ondisk_log_start":"164'300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6477,"log_dups_size":0,"ondisk_log_size":6477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265093120,"num_objects":4045,"num_object_clones":0,"num_object_copies":12135,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":3331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6777,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1436,"num_bytes_recovered":94044160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":930},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988280+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:51:46.988280+0000","last_peered":"2024-09-13T14:51:46.988280+0000","last_clean":"2024-09-13T14:51:46.988280+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:51:46.988280+0000","last_undegraded":"2024-09-13T14:51:46.988280+0000","last_fullsized":"2024-09-13T14:51:46.988280+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":239,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.994401+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:51:46.994401+0000","last_peered":"2024-09-13T14:51:46.994401+0000","last_clean":"2024-09-13T14:51:46.994401+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:51:46.994401+0000","last_undegraded":"2024-09-13T14:51:46.994401+0000","last_fullsized":"2024-09-13T14:51:46.994401+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"230'6519","reported_seq":10936,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.497335+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:51:59.497335+0000","last_peered":"2024-09-13T14:51:59.497335+0000","last_clean":"2024-09-13T14:51:59.497335+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:51:59.497335+0000","last_undegraded":"2024-09-13T14:51:59.497335+0000","last_fullsized":"2024-09-13T14:51:59.497335+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6519,"log_dups_size":0,"ondisk_log_size":6519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261226520,"num_objects":3987,"num_object_clones":0,"num_object_copies":11961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3987,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6519,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":56,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.018103+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:51:58.018103+0000","last_peered":"2024-09-13T14:51:58.018103+0000","last_clean":"2024-09-13T14:51:58.018103+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:51:58.018103+0000","last_undegraded":"2024-09-13T14:51:58.018103+0000","last_fullsized":"2024-09-13T14:51:58.018103+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":166,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.027375+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:51:58.027375+0000","last_peered":"2024-09-13T14:51:58.027375+0000","last_clean":"2024-09-13T14:51:58.027375+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:51:58.027375+0000","last_undegraded":"2024-09-13T14:51:58.027375+0000","last_fullsized":"2024-09-13T14:51:58.027375+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"230'6618","reported_seq":7856,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.520903+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:51:59.520903+0000","last_peered":"2024-09-13T14:51:59.520903+0000","last_clean":"2024-09-13T14:51:59.520903+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:51:59.520903+0000","last_undegraded":"2024-09-13T14:51:59.520903+0000","last_fullsized":"2024-09-13T14:51:59.520903+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270401536,"num_objects":4126,"num_object_clones":0,"num_object_copies":12378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6618,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":510,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003281+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:51:47.003281+0000","last_peered":"2024-09-13T14:51:47.003281+0000","last_clean":"2024-09-13T14:51:47.003281+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:51:47.003281+0000","last_undegraded":"2024-09-13T14:51:47.003281+0000","last_fullsized":"2024-09-13T14:51:47.003281+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":796,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036199+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:51:47.036199+0000","last_peered":"2024-09-13T14:51:47.036199+0000","last_clean":"2024-09-13T14:51:47.036199+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:51:47.036199+0000","last_undegraded":"2024-09-13T14:51:47.036199+0000","last_fullsized":"2024-09-13T14:51:47.036199+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":276,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.017999+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:51:58.017999+0000","last_peered":"2024-09-13T14:51:58.017999+0000","last_clean":"2024-09-13T14:51:58.017999+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:51:58.017999+0000","last_undegraded":"2024-09-13T14:51:58.017999+0000","last_fullsized":"2024-09-13T14:51:58.017999+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":25,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.995776+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:51:46.995776+0000","last_peered":"2024-09-13T14:51:46.995776+0000","last_clean":"2024-09-13T14:51:46.995776+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:51:46.995776+0000","last_undegraded":"2024-09-13T14:51:46.995776+0000","last_fullsized":"2024-09-13T14:51:46.995776+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"230'6748","reported_seq":11451,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.338664+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:51:59.338664+0000","last_peered":"2024-09-13T14:51:59.338664+0000","last_clean":"2024-09-13T14:51:59.338664+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:51:59.338664+0000","last_undegraded":"2024-09-13T14:51:59.338664+0000","last_fullsized":"2024-09-13T14:51:59.338664+0000","mapping_epoch":223,"log_start":"168'918","ondisk_log_start":"168'918","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268697600,"num_objects":4100,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":357,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.023350+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:51:58.023350+0000","last_peered":"2024-09-13T14:51:58.023350+0000","last_clean":"2024-09-13T14:51:58.023350+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:51:58.023350+0000","last_undegraded":"2024-09-13T14:51:58.023350+0000","last_fullsized":"2024-09-13T14:51:58.023350+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":322,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.006371+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:51:47.006371+0000","last_peered":"2024-09-13T14:51:47.006371+0000","last_clean":"2024-09-13T14:51:47.006371+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:51:47.006371+0000","last_undegraded":"2024-09-13T14:51:47.006371+0000","last_fullsized":"2024-09-13T14:51:47.006371+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":333,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003871+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:51:47.003871+0000","last_peered":"2024-09-13T14:51:47.003871+0000","last_clean":"2024-09-13T14:51:47.003871+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:51:47.003871+0000","last_undegraded":"2024-09-13T14:51:47.003871+0000","last_fullsized":"2024-09-13T14:51:47.003871+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"230'6621","reported_seq":12471,"reported_epoch":230,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:59.046719+0000","last_change":"2024-09-13T14:50:38.024593+0000","last_active":"2024-09-13T14:51:59.046719+0000","last_peered":"2024-09-13T14:51:59.046719+0000","last_clean":"2024-09-13T14:48:36.189224+0000","last_became_active":"2024-09-13T14:50:38.015505+0000","last_became_peered":"2024-09-13T14:50:38.015505+0000","last_unstale":"2024-09-13T14:51:59.046719+0000","last_undegraded":"2024-09-13T14:50:37.965648+0000","last_fullsized":"2024-09-13T14:50:37.942220+0000","mapping_epoch":215,"log_start":"164'200","ondisk_log_start":"164'200","created":160,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6421,"log_dups_size":0,"ondisk_log_size":6421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260505600,"num_objects":3975,"num_object_clones":0,"num_object_copies":11925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6621,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2845,"num_bytes_recovered":186449920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":1914},{"shards":"2,3,5","objects":2061}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":860,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.027105+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:51:58.027105+0000","last_peered":"2024-09-13T14:51:58.027105+0000","last_clean":"2024-09-13T14:51:58.027105+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:51:58.027105+0000","last_undegraded":"2024-09-13T14:51:58.027105+0000","last_fullsized":"2024-09-13T14:51:58.027105+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":315,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.023472+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:51:58.023472+0000","last_peered":"2024-09-13T14:51:58.023472+0000","last_clean":"2024-09-13T14:51:58.023472+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:51:58.023472+0000","last_undegraded":"2024-09-13T14:51:58.023472+0000","last_fullsized":"2024-09-13T14:51:58.023472+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":68,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.023701+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:51:58.023701+0000","last_peered":"2024-09-13T14:51:58.023701+0000","last_clean":"2024-09-13T14:51:58.023701+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:51:58.023701+0000","last_undegraded":"2024-09-13T14:51:58.023701+0000","last_fullsized":"2024-09-13T14:51:58.023701+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"230'6706","reported_seq":11298,"reported_epoch":230,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:59.082972+0000","last_change":"2024-09-13T14:50:42.308313+0000","last_active":"2024-09-13T14:51:59.082972+0000","last_peered":"2024-09-13T14:51:59.082972+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:51:59.082972+0000","last_undegraded":"2024-09-13T14:50:40.576119+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"164'300","ondisk_log_start":"164'300","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263061504,"num_objects":4014,"num_object_clones":0,"num_object_copies":12042,"num_objects_missing_on_primary":89,"num_objects_missing":89,"num_objects_degraded":2392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6706,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":148504576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"1,4,5","objects":9},{"shards":"3,4","objects":1897},{"shards":"3,4,5","objects":2108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":83,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036257+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:51:47.036257+0000","last_peered":"2024-09-13T14:51:47.036257+0000","last_clean":"2024-09-13T14:51:47.036257+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:51:47.036257+0000","last_undegraded":"2024-09-13T14:51:47.036257+0000","last_fullsized":"2024-09-13T14:51:47.036257+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":214,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988707+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:51:46.988707+0000","last_peered":"2024-09-13T14:51:46.988707+0000","last_clean":"2024-09-13T14:51:46.988707+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:51:46.988707+0000","last_undegraded":"2024-09-13T14:51:46.988707+0000","last_fullsized":"2024-09-13T14:51:46.988707+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":318,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.024539+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:51:58.024539+0000","last_peered":"2024-09-13T14:51:58.024539+0000","last_clean":"2024-09-13T14:51:58.024539+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:51:58.024539+0000","last_undegraded":"2024-09-13T14:51:58.024539+0000","last_fullsized":"2024-09-13T14:51:58.024539+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"229'6748","reported_seq":11313,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:57.533551+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:51:57.533551+0000","last_peered":"2024-09-13T14:51:57.533551+0000","last_clean":"2024-09-13T14:51:57.533551+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:51:57.533551+0000","last_undegraded":"2024-09-13T14:51:57.533551+0000","last_fullsized":"2024-09-13T14:51:57.533551+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266469376,"num_objects":4066,"num_object_clones":0,"num_object_copies":12198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6748,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":83,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036497+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:51:47.036497+0000","last_peered":"2024-09-13T14:51:47.036497+0000","last_clean":"2024-09-13T14:51:47.036497+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:51:47.036497+0000","last_undegraded":"2024-09-13T14:51:47.036497+0000","last_fullsized":"2024-09-13T14:51:47.036497+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":870,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.024706+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:51:58.024706+0000","last_peered":"2024-09-13T14:51:58.024706+0000","last_clean":"2024-09-13T14:51:58.024706+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:51:58.024706+0000","last_undegraded":"2024-09-13T14:51:58.024706+0000","last_fullsized":"2024-09-13T14:51:58.024706+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":295,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.023637+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:51:58.023637+0000","last_peered":"2024-09-13T14:51:58.023637+0000","last_clean":"2024-09-13T14:51:58.023637+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:51:58.023637+0000","last_undegraded":"2024-09-13T14:51:58.023637+0000","last_fullsized":"2024-09-13T14:51:58.023637+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"230'6782","reported_seq":11477,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:59.331534+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:51:59.331534+0000","last_peered":"2024-09-13T14:51:59.331534+0000","last_clean":"2024-09-13T14:51:59.331534+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:51:59.331534+0000","last_undegraded":"2024-09-13T14:51:59.331534+0000","last_fullsized":"2024-09-13T14:51:59.331534+0000","mapping_epoch":218,"log_start":"168'945","ondisk_log_start":"168'945","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5837,"log_dups_size":0,"ondisk_log_size":5837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270401536,"num_objects":4126,"num_object_clones":0,"num_object_copies":12378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":322,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.024621+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:51:58.024621+0000","last_peered":"2024-09-13T14:51:58.024621+0000","last_clean":"2024-09-13T14:51:58.024621+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:51:58.024621+0000","last_undegraded":"2024-09-13T14:51:58.024621+0000","last_fullsized":"2024-09-13T14:51:58.024621+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":541,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003675+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:51:47.003675+0000","last_peered":"2024-09-13T14:51:47.003675+0000","last_clean":"2024-09-13T14:51:47.003675+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:51:47.003675+0000","last_undegraded":"2024-09-13T14:51:47.003675+0000","last_fullsized":"2024-09-13T14:51:47.003675+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":318,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003345+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:51:47.003345+0000","last_peered":"2024-09-13T14:51:47.003345+0000","last_clean":"2024-09-13T14:51:47.003345+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:51:47.003345+0000","last_undegraded":"2024-09-13T14:51:47.003345+0000","last_fullsized":"2024-09-13T14:51:47.003345+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"230'6550","reported_seq":8745,"reported_epoch":230,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:51:59.055938+0000","last_change":"2024-09-13T14:50:19.881564+0000","last_active":"2024-09-13T14:51:59.055938+0000","last_peered":"2024-09-13T14:51:59.055938+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:51:59.055938+0000","last_undegraded":"2024-09-13T14:50:19.795499+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259522560,"num_objects":3960,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":1869},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2090}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":340,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.019192+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:51:58.019192+0000","last_peered":"2024-09-13T14:51:58.019192+0000","last_clean":"2024-09-13T14:51:58.019192+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:51:58.019192+0000","last_undegraded":"2024-09-13T14:51:58.019192+0000","last_fullsized":"2024-09-13T14:51:58.019192+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":772,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.024028+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:51:58.024028+0000","last_peered":"2024-09-13T14:51:58.024028+0000","last_clean":"2024-09-13T14:51:58.024028+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:51:58.024028+0000","last_undegraded":"2024-09-13T14:51:58.024028+0000","last_fullsized":"2024-09-13T14:51:58.024028+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":56,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.017999+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:51:58.017999+0000","last_peered":"2024-09-13T14:51:58.017999+0000","last_clean":"2024-09-13T14:51:58.017999+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:51:58.017999+0000","last_undegraded":"2024-09-13T14:51:58.017999+0000","last_fullsized":"2024-09-13T14:51:58.017999+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"230'6703","reported_seq":8266,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.721882+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:51:58.721882+0000","last_peered":"2024-09-13T14:51:58.721882+0000","last_clean":"2024-09-13T14:51:58.721882+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:51:58.721882+0000","last_undegraded":"2024-09-13T14:51:58.721882+0000","last_fullsized":"2024-09-13T14:51:58.721882+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6703,"log_dups_size":0,"ondisk_log_size":6703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":12471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6703,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":313,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003523+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:51:47.003523+0000","last_peered":"2024-09-13T14:51:47.003523+0000","last_clean":"2024-09-13T14:51:47.003523+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:51:47.003523+0000","last_undegraded":"2024-09-13T14:51:47.003523+0000","last_fullsized":"2024-09-13T14:51:47.003523+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":781,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.995929+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:51:46.995929+0000","last_peered":"2024-09-13T14:51:46.995929+0000","last_clean":"2024-09-13T14:51:46.995929+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:51:46.995929+0000","last_undegraded":"2024-09-13T14:51:46.995929+0000","last_fullsized":"2024-09-13T14:51:46.995929+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":216,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988186+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:51:46.988186+0000","last_peered":"2024-09-13T14:51:46.988186+0000","last_clean":"2024-09-13T14:51:46.988186+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:51:46.988186+0000","last_undegraded":"2024-09-13T14:51:46.988186+0000","last_fullsized":"2024-09-13T14:51:46.988186+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":212,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988456+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:51:46.988456+0000","last_peered":"2024-09-13T14:51:46.988456+0000","last_clean":"2024-09-13T14:51:46.988456+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:51:46.988456+0000","last_undegraded":"2024-09-13T14:51:46.988456+0000","last_fullsized":"2024-09-13T14:51:46.988456+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":260,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988893+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:51:46.988893+0000","last_peered":"2024-09-13T14:51:46.988893+0000","last_clean":"2024-09-13T14:51:46.988893+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:51:46.988893+0000","last_undegraded":"2024-09-13T14:51:46.988893+0000","last_fullsized":"2024-09-13T14:51:46.988893+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":319,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.023495+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:51:58.023495+0000","last_peered":"2024-09-13T14:51:58.023495+0000","last_clean":"2024-09-13T14:51:58.023495+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:51:58.023495+0000","last_undegraded":"2024-09-13T14:51:58.023495+0000","last_fullsized":"2024-09-13T14:51:58.023495+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":160,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.023391+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:51:58.023391+0000","last_peered":"2024-09-13T14:51:58.023391+0000","last_clean":"2024-09-13T14:51:58.023391+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:51:58.023391+0000","last_undegraded":"2024-09-13T14:51:58.023391+0000","last_fullsized":"2024-09-13T14:51:58.023391+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":92,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036496+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:51:47.036496+0000","last_peered":"2024-09-13T14:51:47.036496+0000","last_clean":"2024-09-13T14:51:47.036496+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:51:47.036496+0000","last_undegraded":"2024-09-13T14:51:47.036496+0000","last_fullsized":"2024-09-13T14:51:47.036496+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":297,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.018548+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:51:58.018548+0000","last_peered":"2024-09-13T14:51:58.018548+0000","last_clean":"2024-09-13T14:51:58.018548+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:51:58.018548+0000","last_undegraded":"2024-09-13T14:51:58.018548+0000","last_fullsized":"2024-09-13T14:51:58.018548+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":349,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003999+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:51:47.003999+0000","last_peered":"2024-09-13T14:51:47.003999+0000","last_clean":"2024-09-13T14:51:47.003999+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:51:47.003999+0000","last_undegraded":"2024-09-13T14:51:47.003999+0000","last_fullsized":"2024-09-13T14:51:47.003999+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":348,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.018296+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:51:58.018296+0000","last_peered":"2024-09-13T14:51:58.018296+0000","last_clean":"2024-09-13T14:51:58.018296+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:51:58.018296+0000","last_undegraded":"2024-09-13T14:51:58.018296+0000","last_fullsized":"2024-09-13T14:51:58.018296+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":122,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.995868+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:51:46.995868+0000","last_peered":"2024-09-13T14:51:46.995868+0000","last_clean":"2024-09-13T14:51:46.995868+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:51:46.995868+0000","last_undegraded":"2024-09-13T14:51:46.995868+0000","last_fullsized":"2024-09-13T14:51:46.995868+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":315,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.018425+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:51:58.018425+0000","last_peered":"2024-09-13T14:51:58.018425+0000","last_clean":"2024-09-13T14:51:58.018425+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:51:58.018425+0000","last_undegraded":"2024-09-13T14:51:58.018425+0000","last_fullsized":"2024-09-13T14:51:58.018425+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":329,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003776+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:51:47.003776+0000","last_peered":"2024-09-13T14:51:47.003776+0000","last_clean":"2024-09-13T14:51:47.003776+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:51:47.003776+0000","last_undegraded":"2024-09-13T14:51:47.003776+0000","last_fullsized":"2024-09-13T14:51:47.003776+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":300,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.024396+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:51:58.024396+0000","last_peered":"2024-09-13T14:51:58.024396+0000","last_clean":"2024-09-13T14:51:58.024396+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:51:58.024396+0000","last_undegraded":"2024-09-13T14:51:58.024396+0000","last_fullsized":"2024-09-13T14:51:58.024396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":164,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.023258+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:51:58.023258+0000","last_peered":"2024-09-13T14:51:58.023258+0000","last_clean":"2024-09-13T14:51:58.023258+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:51:58.023258+0000","last_undegraded":"2024-09-13T14:51:58.023258+0000","last_fullsized":"2024-09-13T14:51:58.023258+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":210,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988595+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:51:46.988595+0000","last_peered":"2024-09-13T14:51:46.988595+0000","last_clean":"2024-09-13T14:51:46.988595+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:51:46.988595+0000","last_undegraded":"2024-09-13T14:51:46.988595+0000","last_fullsized":"2024-09-13T14:51:46.988595+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":59,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.026856+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:51:58.026856+0000","last_peered":"2024-09-13T14:51:58.026856+0000","last_clean":"2024-09-13T14:51:58.026856+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:51:58.026856+0000","last_undegraded":"2024-09-13T14:51:58.026856+0000","last_fullsized":"2024-09-13T14:51:58.026856+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":322,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003246+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:51:47.003246+0000","last_peered":"2024-09-13T14:51:47.003246+0000","last_clean":"2024-09-13T14:51:47.003246+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:51:47.003246+0000","last_undegraded":"2024-09-13T14:51:47.003246+0000","last_fullsized":"2024-09-13T14:51:47.003246+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":211,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988413+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:51:46.988413+0000","last_peered":"2024-09-13T14:51:46.988413+0000","last_clean":"2024-09-13T14:51:46.988413+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:51:46.988413+0000","last_undegraded":"2024-09-13T14:51:46.988413+0000","last_fullsized":"2024-09-13T14:51:46.988413+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":67,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036174+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:51:47.036174+0000","last_peered":"2024-09-13T14:51:47.036174+0000","last_clean":"2024-09-13T14:51:47.036174+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:51:47.036174+0000","last_undegraded":"2024-09-13T14:51:47.036174+0000","last_fullsized":"2024-09-13T14:51:47.036174+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":208,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988523+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:51:46.988523+0000","last_peered":"2024-09-13T14:51:46.988523+0000","last_clean":"2024-09-13T14:51:46.988523+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:51:46.988523+0000","last_undegraded":"2024-09-13T14:51:46.988523+0000","last_fullsized":"2024-09-13T14:51:46.988523+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":69,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.036577+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:51:47.036577+0000","last_peered":"2024-09-13T14:51:47.036577+0000","last_clean":"2024-09-13T14:51:47.036577+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:51:47.036577+0000","last_undegraded":"2024-09-13T14:51:47.036577+0000","last_fullsized":"2024-09-13T14:51:47.036577+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":318,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.018754+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:51:58.018754+0000","last_peered":"2024-09-13T14:51:58.018754+0000","last_clean":"2024-09-13T14:51:58.018754+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:51:58.018754+0000","last_undegraded":"2024-09-13T14:51:58.018754+0000","last_fullsized":"2024-09-13T14:51:58.018754+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":334,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.019141+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:51:58.019141+0000","last_peered":"2024-09-13T14:51:58.019141+0000","last_clean":"2024-09-13T14:51:58.019141+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:51:58.019141+0000","last_undegraded":"2024-09-13T14:51:58.019141+0000","last_fullsized":"2024-09-13T14:51:58.019141+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":238,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:46.988117+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:51:46.988117+0000","last_peered":"2024-09-13T14:51:46.988117+0000","last_clean":"2024-09-13T14:51:46.988117+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:51:46.988117+0000","last_undegraded":"2024-09-13T14:51:46.988117+0000","last_fullsized":"2024-09-13T14:51:46.988117+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":385,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-13T14:51:47.003448+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:51:47.003448+0000","last_peered":"2024-09-13T14:51:47.003448+0000","last_clean":"2024-09-13T14:51:47.003448+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:51:47.003448+0000","last_undegraded":"2024-09-13T14:51:47.003448+0000","last_fullsized":"2024-09-13T14:51:47.003448+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4250599448,"num_objects":64860,"num_object_clones":0,"num_object_copies":194580,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":15405,"num_objects_misplaced":1976,"num_objects_unfound":0,"num_objects_dirty":64860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106688,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29778,"num_bytes_recovered":1924857856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13749661696,"data_stored":13749649480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103560,"ondisk_log_size":103560,"up":48,"acting":41,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234295,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495931,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2382648,"kb_used_data":2130860,"kb_used_omap":1,"kb_used_meta":251774,"kb_avail":91362504,"statfs":{"total":95995035648,"available":93555204096,"internally_reserved":0,"allocated":2182000640,"data_stored":2180726710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":257817031},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757566,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2312876,"kb_used_data":2137380,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":91432276,"statfs":{"total":95995035648,"available":93626650624,"internally_reserved":0,"allocated":2188677120,"data_stored":2187406985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179698119},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019206,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1595768,"kb_used_data":1303788,"kb_used_omap":1,"kb_used_meta":291966,"kb_avail":92149384,"statfs":{"total":95995035648,"available":94360969216,"internally_reserved":0,"allocated":1335078912,"data_stored":1333805043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":298973638},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468091,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2950188,"kb_used_data":2733920,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":90794964,"statfs":{"total":95995035648,"available":92974043136,"internally_reserved":0,"allocated":2799534080,"data_stored":2798267791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":221441188},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,3,1,2,3,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631620,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2267836,"kb_used_data":1947376,"kb_used_omap":8,"kb_used_meta":320439,"kb_avail":91477316,"statfs":{"total":95995035648,"available":93672771584,"internally_reserved":0,"allocated":1994113024,"data_stored":1992835585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":328130527},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795143,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2164848,"kb_used_data":1930472,"kb_used_omap":7,"kb_used_meta":234360,"kb_avail":91580304,"statfs":{"total":95995035648,"available":93778231296,"internally_reserved":0,"allocated":1976803328,"data_stored":1975529296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":239985266},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958673,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1470696,"kb_used_data":1186336,"kb_used_omap":7,"kb_used_meta":284344,"kb_avail":92274456,"statfs":{"total":95995035648,"available":94489042944,"internally_reserved":0,"allocated":1214808064,"data_stored":1213534446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":291168743},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1206849536,"data_stored":1206845464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1967853568,"data_stored":1967849496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1986396160,"data_stored":1986396160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2788229120,"data_stored":2788229120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1327104000,"data_stored":1327104000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2185236480,"data_stored":2185232408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2177040384,"data_stored":2177040384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":110952448,"data_stored":110952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:52:03.838 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:03.844 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:03.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:03.885 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:03.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:03.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:04.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:04.119 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:04.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:04.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:04.264 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:04.343 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:04 smithi154 ceph-mon[102314]: pgmap v679: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 214 op/s; 15405/194274 objects degraded (7.930%); 1976/194274 objects misplaced (1.017%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:04.343 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:04 smithi154 ceph-mon[107038]: pgmap v679: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 214 op/s; 15405/194274 objects degraded (7.930%); 1976/194274 objects misplaced (1.017%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:04.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:04.389 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:04 smithi172 ceph-mon[100696]: pgmap v679: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 214 op/s; 15405/194274 objects degraded (7.930%); 1976/194274 objects misplaced (1.017%); 7.0 MiB/s, 111 objects/s recovering 2024-09-13T14:52:04.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:04.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:04.619 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:04.640 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:04.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:04.764 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:04.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:04.892 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:04.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:04.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:05.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:05.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:05.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:05.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15405/194274 objects degraded (7.930%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:05.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15405/194274 objects degraded (7.930%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:05.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:05.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:05.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:05.397 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15405/194274 objects degraded (7.930%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:05.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:05.524 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:05.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:05.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:05.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:05.780 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:05.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:05.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:05.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:06.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:06.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:06.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:06.169 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:06.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:06.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:06 smithi154 ceph-mon[102314]: pgmap v680: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 222 op/s; 15334/191367 objects degraded (8.013%); 1394/191367 objects misplaced (0.728%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:06 smithi154 ceph-mon[107038]: pgmap v680: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 222 op/s; 15334/191367 objects degraded (8.013%); 1394/191367 objects misplaced (0.728%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:06.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:06.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:06.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:06.421 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:06 smithi172 ceph-mon[100696]: pgmap v680: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 222 op/s; 15334/191367 objects degraded (8.013%); 1394/191367 objects misplaced (0.728%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:06.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:06.546 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:06.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:06.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:06.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.798 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:06.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:06.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:52:06.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:06.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:06.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:07.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:07.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:07.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:07.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:07.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:07.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:07.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:07.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:07.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:07.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:07.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:07.868 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:52:07.868 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:07.886 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:52:07.887 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:07.904 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:52:07.904 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:07.921 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:52:07.924 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:07.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:07.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:52:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:07.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:07.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:07.956 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:52:07.957 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:07.973 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:52:07.974 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:07.998 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:52:07.998 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:08.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:08.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:08.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:08.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:08.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:08.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:08.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:08.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:08 smithi154 ceph-mon[102314]: pgmap v681: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 222 op/s; 15334/191367 objects degraded (8.013%); 1394/191367 objects misplaced (0.728%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/79592424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:08.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:08 smithi154 ceph-mon[107038]: pgmap v681: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 222 op/s; 15334/191367 objects degraded (8.013%); 1394/191367 objects misplaced (0.728%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:08.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/79592424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:08 smithi172 ceph-mon[100696]: pgmap v681: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 222 op/s; 15334/191367 objects degraded (8.013%); 1394/191367 objects misplaced (0.728%); 7.7 MiB/s, 123 objects/s recovering 2024-09-13T14:52:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/79592424' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:08.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:08.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:08.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:08.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:08.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:08.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:08.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:08.974 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:08.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:09.048 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:52:09.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:09.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:09.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:09.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:09.296 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:09 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2971485481' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:09.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:09 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2971485481' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:09.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:09.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:09.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:09.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:09 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2971485481' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:52:09.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:09.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:09.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.597 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:09.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:09.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:09.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:09.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:09.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:09.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:09.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:09.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:10.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:10.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:10.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[102314]: pgmap v682: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 210 op/s; 15334/190677 objects degraded (8.042%); 1394/190677 objects misplaced (0.731%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:52:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2971485481' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[102314]: osdmap e231: 8 total, 8 up, 7 in 2024-09-13T14:52:10.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15334/190677 objects degraded (8.042%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:10.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[107038]: pgmap v682: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 210 op/s; 15334/190677 objects degraded (8.042%); 1394/190677 objects misplaced (0.731%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:52:10.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:10.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2971485481' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:10.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[107038]: osdmap e231: 8 total, 8 up, 7 in 2024-09-13T14:52:10.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15334/190677 objects degraded (8.042%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:10.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:10.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:10.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:10.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:10.486 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:10 smithi172 ceph-mon[100696]: pgmap v682: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 210 op/s; 15334/190677 objects degraded (8.042%); 1394/190677 objects misplaced (0.731%); 7.0 MiB/s, 112 objects/s recovering 2024-09-13T14:52:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:10 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:10 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2971485481' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:52:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:10 smithi172 ceph-mon[100696]: osdmap e231: 8 total, 8 up, 7 in 2024-09-13T14:52:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15334/190677 objects degraded (8.042%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:10.517 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:52:10.517 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:52:10.518 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T14:52:10.518 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:52:10.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:10.610 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:10.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:10.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:10.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.865 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:10.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:10.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:10.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:10.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:11.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:11.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:11.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.238 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.265 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:11.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:11.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:11.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:11.487 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:11.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.613 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:11.738 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:11.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:11.863 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:11.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:52:11.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:11.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:52:11.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:11.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:12.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:12.111 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:12.131 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:12 smithi154 ceph-mon[102314]: pgmap v684: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 229 op/s; 15261/187767 objects degraded (8.128%); 710/187767 objects misplaced (0.378%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T14:52:12.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:12 smithi154 ceph-mon[107038]: pgmap v684: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 229 op/s; 15261/187767 objects degraded (8.128%); 710/187767 objects misplaced (0.378%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T14:52:12.131 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:52:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:52:11] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:52:12.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:12.235 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:12.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:12.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:12.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:12.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:12.490 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:12 smithi172 ceph-mon[100696]: pgmap v684: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 229 op/s; 15261/187767 objects degraded (8.128%); 710/187767 objects misplaced (0.378%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T14:52:12.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:12.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:12.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:12.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:12.807 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:12.808 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:12.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:12.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:12.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:12.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:13.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:13.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:52:13.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:13.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:13.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:13.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:13.261 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:13.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:13.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:13.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:13.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:13.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:13.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:13.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:13.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:13.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:13.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:13.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:14.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:14.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:14.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:14.164 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:14.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:14.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:14.381 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:14 smithi154 ceph-mon[102314]: pgmap v685: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 230 op/s; 15261/187359 objects degraded (8.145%); 710/187359 objects misplaced (0.379%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T14:52:14.381 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:14 smithi154 ceph-mon[107038]: pgmap v685: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 230 op/s; 15261/187359 objects degraded (8.145%); 710/187359 objects misplaced (0.379%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T14:52:14.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:14.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:14 smithi172 ceph-mon[100696]: pgmap v685: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 230 op/s; 15261/187359 objects degraded (8.145%); 710/187359 objects misplaced (0.379%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T14:52:14.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:14.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:14.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:14.665 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:14.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:14.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:14.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:14.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:14.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:14.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:15.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:15.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:15.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:15.172 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:15.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:15.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:15.313 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:15.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:15.423 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15261/187359 objects degraded (8.145%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:15.423 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15261/187359 objects degraded (8.145%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:15.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15261/187359 objects degraded (8.145%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:15.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:15.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:15.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:15.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:15.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:15.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:15.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:15.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:15.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:15.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:16.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:16.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:16.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.195 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:16.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:16 smithi154 ceph-mon[102314]: pgmap v686: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 15006/184269 objects degraded (8.144%); 9.6 MiB/s, 177 objects/s recovering 2024-09-13T14:52:16.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:16 smithi154 ceph-mon[107038]: pgmap v686: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 15006/184269 objects degraded (8.144%); 9.6 MiB/s, 177 objects/s recovering 2024-09-13T14:52:16.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:16.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:16.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:16.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:16 smithi172 ceph-mon[100696]: pgmap v686: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 15006/184269 objects degraded (8.144%); 9.6 MiB/s, 177 objects/s recovering 2024-09-13T14:52:16.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:16.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:16.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:16.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:16.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:16.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:16.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:16.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:17.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:17.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:17.166 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:17 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:17.167 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:17 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:17.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:17.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:17.332 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:52:17.333 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:52:17.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:17.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:17 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:52:17.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:17.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:17.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:17.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:17.642 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:17.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:17.773 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:17.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:17.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:17.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:17.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.021 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:18.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.090 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:52:18.090 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:52:18.090 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:52:18.090 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:52:18.090 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:52:18.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:18.142 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:18 smithi154 ceph-mon[102314]: pgmap v687: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 15006/184269 objects degraded (8.144%); 9.6 MiB/s, 177 objects/s recovering 2024-09-13T14:52:18.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:18 smithi154 ceph-mon[102314]: osdmap e232: 8 total, 8 up, 7 in 2024-09-13T14:52:18.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:18 smithi154 ceph-mon[102314]: from='client.45158 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:18 smithi154 ceph-mon[107038]: pgmap v687: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 15006/184269 objects degraded (8.144%); 9.6 MiB/s, 177 objects/s recovering 2024-09-13T14:52:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:18 smithi154 ceph-mon[107038]: osdmap e232: 8 total, 8 up, 7 in 2024-09-13T14:52:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:18 smithi154 ceph-mon[107038]: from='client.45158 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:18.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:18.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:18.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:18.405 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:18 smithi172 ceph-mon[100696]: pgmap v687: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 15006/184269 objects degraded (8.144%); 9.6 MiB/s, 177 objects/s recovering 2024-09-13T14:52:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:18 smithi172 ceph-mon[100696]: osdmap e232: 8 total, 8 up, 7 in 2024-09-13T14:52:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:18 smithi172 ceph-mon[100696]: from='client.45158 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:18.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:18.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:18.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:18.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:18.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:18.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:18.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:18.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:18.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:19.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:19.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:19.097 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:52:19.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:19.179 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:19 smithi172 ceph-mon[100696]: osdmap e233: 8 total, 8 up, 7 in 2024-09-13T14:52:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3867380228' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:19 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:19.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:19.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:19.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:19.427 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:19.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:19 smithi154 ceph-mon[102314]: osdmap e233: 8 total, 8 up, 7 in 2024-09-13T14:52:19.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3867380228' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:19.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:19 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:19.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:19 smithi154 ceph-mon[107038]: osdmap e233: 8 total, 8 up, 7 in 2024-09-13T14:52:19.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3867380228' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:19.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:19 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:19.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:19.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:19.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.676 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:19.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:19.804 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:19.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:19.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:19.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:20.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:20.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:20.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.198 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:20.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:20.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:20.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:20.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:20.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[102314]: pgmap v690: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 169 op/s; 15006/183699 objects degraded (8.169%); 6.1 MiB/s, 127 objects/s recovering 2024-09-13T14:52:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[102314]: osdmap e234: 8 total, 8 up, 7 in 2024-09-13T14:52:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15006/183699 objects degraded (8.169%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[107038]: pgmap v690: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 169 op/s; 15006/183699 objects degraded (8.169%); 6.1 MiB/s, 127 objects/s recovering 2024-09-13T14:52:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:20.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[107038]: osdmap e234: 8 total, 8 up, 7 in 2024-09-13T14:52:20.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15006/183699 objects degraded (8.169%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:20 smithi172 ceph-mon[100696]: pgmap v690: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 169 op/s; 15006/183699 objects degraded (8.169%); 6.1 MiB/s, 127 objects/s recovering 2024-09-13T14:52:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:20 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:52:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:20 smithi172 ceph-mon[100696]: osdmap e234: 8 total, 8 up, 7 in 2024-09-13T14:52:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15006/183699 objects degraded (8.169%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:20.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:20.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:20.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:20.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:20.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:20.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:20.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:20.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:21.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:21.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:21.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:21.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:21.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:21.344 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:21.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:21.473 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2024-09-13T14:52:17.075606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4028307453,"num_objects":61423,"num_object_clones":0,"num_object_copies":184269,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":15006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61423,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":111276,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32364,"num_bytes_recovered":2071088261,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106307,"ondisk_log_size":106307,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":293,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14613484,"kb_used_data":12818100,"kb_used_omap":32,"kb_used_meta":1795231,"kb_avail":641602580,"statfs":{"total":671965249536,"available":657001041920,"internally_reserved":0,"allocated":13125734400,"data_stored":13116794232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1838317104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,0,0,0,0,5,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-139984896,"num_objects":-2136,"num_object_clones":0,"num_object_copies":-6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-328,"num_objects_misplaced":-1394,"num_objects_unfound":0,"num_objects_dirty":-2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2136,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1777,"num_bytes_recovered":101056512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.090340"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":343,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.096692+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:52:09.096692+0000","last_peered":"2024-09-13T14:52:09.096692+0000","last_clean":"2024-09-13T14:52:09.096692+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:52:09.096692+0000","last_undegraded":"2024-09-13T14:52:09.096692+0000","last_fullsized":"2024-09-13T14:52:09.096692+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":96,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.015921+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:51:58.015921+0000","last_peered":"2024-09-13T14:51:58.015921+0000","last_clean":"2024-09-13T14:51:58.015921+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:51:58.015921+0000","last_undegraded":"2024-09-13T14:51:58.015921+0000","last_fullsized":"2024-09-13T14:51:58.015921+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":66,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.020165+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:51:58.020165+0000","last_peered":"2024-09-13T14:51:58.020165+0000","last_clean":"2024-09-13T14:51:58.020165+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:51:58.020165+0000","last_undegraded":"2024-09-13T14:51:58.020165+0000","last_fullsized":"2024-09-13T14:51:58.020165+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":333,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.053697+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:52:09.053697+0000","last_peered":"2024-09-13T14:52:09.053697+0000","last_clean":"2024-09-13T14:52:09.053697+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:52:09.053697+0000","last_undegraded":"2024-09-13T14:52:09.053697+0000","last_fullsized":"2024-09-13T14:52:09.053697+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":333,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.052143+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:52:09.052143+0000","last_peered":"2024-09-13T14:52:09.052143+0000","last_clean":"2024-09-13T14:52:09.052143+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:52:09.052143+0000","last_undegraded":"2024-09-13T14:52:09.052143+0000","last_fullsized":"2024-09-13T14:52:09.052143+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":166,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.015699+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:51:58.015699+0000","last_peered":"2024-09-13T14:51:58.015699+0000","last_clean":"2024-09-13T14:51:58.015699+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:51:58.015699+0000","last_undegraded":"2024-09-13T14:51:58.015699+0000","last_fullsized":"2024-09-13T14:51:58.015699+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":58,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.055839+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:52:09.055839+0000","last_peered":"2024-09-13T14:52:09.055839+0000","last_clean":"2024-09-13T14:52:09.055839+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:52:09.055839+0000","last_undegraded":"2024-09-13T14:52:09.055839+0000","last_fullsized":"2024-09-13T14:52:09.055839+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":143,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.866427+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:51:58.866427+0000","last_peered":"2024-09-13T14:51:58.866427+0000","last_clean":"2024-09-13T14:51:58.866427+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:51:58.866427+0000","last_undegraded":"2024-09-13T14:51:58.866427+0000","last_fullsized":"2024-09-13T14:51:58.866427+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"231'6983","reported_seq":7808,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:14.228205+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:52:14.228205+0000","last_peered":"2024-09-13T14:52:14.228205+0000","last_clean":"2024-09-13T14:52:14.228205+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:52:14.228205+0000","last_undegraded":"2024-09-13T14:52:14.228205+0000","last_fullsized":"2024-09-13T14:52:14.228205+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256442368,"num_objects":3913,"num_object_clones":0,"num_object_copies":11739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6983,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":267,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.058519+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:52:09.058519+0000","last_peered":"2024-09-13T14:52:09.058519+0000","last_clean":"2024-09-13T14:52:09.058519+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:52:09.058519+0000","last_undegraded":"2024-09-13T14:52:09.058519+0000","last_fullsized":"2024-09-13T14:52:09.058519+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":93,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.019660+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:51:58.019660+0000","last_peered":"2024-09-13T14:51:58.019660+0000","last_clean":"2024-09-13T14:51:58.019660+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:51:58.019660+0000","last_undegraded":"2024-09-13T14:51:58.019660+0000","last_fullsized":"2024-09-13T14:51:58.019660+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"231'6782","reported_seq":8373,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:13.958138+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:52:13.958138+0000","last_peered":"2024-09-13T14:52:13.958138+0000","last_clean":"2024-09-13T14:52:13.958138+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:52:13.958138+0000","last_undegraded":"2024-09-13T14:52:13.958138+0000","last_fullsized":"2024-09-13T14:52:13.958138+0000","mapping_epoch":170,"log_start":"164'300","ondisk_log_start":"164'300","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6482,"log_dups_size":0,"ondisk_log_size":6482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248774656,"num_objects":3796,"num_object_clones":0,"num_object_copies":11388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":94,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.016350+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:51:58.016350+0000","last_peered":"2024-09-13T14:51:58.016350+0000","last_clean":"2024-09-13T14:51:58.016350+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:51:58.016350+0000","last_undegraded":"2024-09-13T14:51:58.016350+0000","last_fullsized":"2024-09-13T14:51:58.016350+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":247,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.051734+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:52:09.051734+0000","last_peered":"2024-09-13T14:52:09.051734+0000","last_clean":"2024-09-13T14:52:09.051734+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:52:09.051734+0000","last_undegraded":"2024-09-13T14:52:09.051734+0000","last_fullsized":"2024-09-13T14:52:09.051734+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"231'6984","reported_seq":8667,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:14.200747+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:52:14.200747+0000","last_peered":"2024-09-13T14:52:14.200747+0000","last_clean":"2024-09-13T14:52:14.200747+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:52:14.200747+0000","last_undegraded":"2024-09-13T14:52:14.200747+0000","last_fullsized":"2024-09-13T14:52:14.200747+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6984,"log_dups_size":0,"ondisk_log_size":6984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252968960,"num_objects":3860,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":70,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.020319+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:51:58.020319+0000","last_peered":"2024-09-13T14:51:58.020319+0000","last_clean":"2024-09-13T14:51:58.020319+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:51:58.020319+0000","last_undegraded":"2024-09-13T14:51:58.020319+0000","last_fullsized":"2024-09-13T14:51:58.020319+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.021652+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:51:58.021652+0000","last_peered":"2024-09-13T14:51:58.021652+0000","last_clean":"2024-09-13T14:51:58.021652+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:51:58.021652+0000","last_undegraded":"2024-09-13T14:51:58.021652+0000","last_fullsized":"2024-09-13T14:51:58.021652+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"231'6739","reported_seq":8483,"reported_epoch":231,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:12.787692+0000","last_change":"2024-09-13T14:50:07.839326+0000","last_active":"2024-09-13T14:52:12.787692+0000","last_peered":"2024-09-13T14:52:12.787692+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:52:12.787692+0000","last_undegraded":"2024-09-13T14:50:07.570096+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247005184,"num_objects":3769,"num_object_clones":0,"num_object_copies":11307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6739,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":785,"num_bytes_recovered":51445760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":1668},{"shards":"0,3,6","objects":2101}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":82,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.020588+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:51:58.020588+0000","last_peered":"2024-09-13T14:51:58.020588+0000","last_clean":"2024-09-13T14:51:58.020588+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:51:58.020588+0000","last_undegraded":"2024-09-13T14:51:58.020588+0000","last_fullsized":"2024-09-13T14:51:58.020588+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":312,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.051685+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:52:09.051685+0000","last_peered":"2024-09-13T14:52:09.051685+0000","last_clean":"2024-09-13T14:52:09.051685+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:52:09.051685+0000","last_undegraded":"2024-09-13T14:52:09.051685+0000","last_fullsized":"2024-09-13T14:52:09.051685+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"231'6882","reported_seq":22899,"reported_epoch":231,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:14.163554+0000","last_change":"2024-09-13T14:51:27.441494+0000","last_active":"2024-09-13T14:52:14.163554+0000","last_peered":"2024-09-13T14:52:14.163554+0000","last_clean":"2024-09-13T14:49:59.725669+0000","last_became_active":"2024-09-13T14:50:08.707516+0000","last_became_peered":"2024-09-13T14:50:08.707516+0000","last_unstale":"2024-09-13T14:52:14.163554+0000","last_undegraded":"2024-09-13T14:50:08.570515+0000","last_fullsized":"2024-09-13T14:50:08.544112+0000","mapping_epoch":205,"log_start":"164'300","ondisk_log_start":"164'300","created":160,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6582,"log_dups_size":0,"ondisk_log_size":6582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:20.218839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253755392,"num_objects":3872,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6882,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7954,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":3264}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":320,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.056062+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:52:09.056062+0000","last_peered":"2024-09-13T14:52:09.056062+0000","last_clean":"2024-09-13T14:52:09.056062+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:52:09.056062+0000","last_undegraded":"2024-09-13T14:52:09.056062+0000","last_fullsized":"2024-09-13T14:52:09.056062+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":213,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.016085+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:51:58.016085+0000","last_peered":"2024-09-13T14:51:58.016085+0000","last_clean":"2024-09-13T14:51:58.016085+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:51:58.016085+0000","last_undegraded":"2024-09-13T14:51:58.016085+0000","last_fullsized":"2024-09-13T14:51:58.016085+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"231'6824","reported_seq":7199,"reported_epoch":231,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:13.926020+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:52:13.926020+0000","last_peered":"2024-09-13T14:52:13.926020+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:52:13.926020+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"164'300","ondisk_log_start":"164'300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6524,"log_dups_size":0,"ondisk_log_size":6524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250085376,"num_objects":3816,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6824,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":1636},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":114,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.021845+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:51:58.021845+0000","last_peered":"2024-09-13T14:51:58.021845+0000","last_clean":"2024-09-13T14:51:58.021845+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:51:58.021845+0000","last_undegraded":"2024-09-13T14:51:58.021845+0000","last_fullsized":"2024-09-13T14:51:58.021845+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":210,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.016124+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:51:58.016124+0000","last_peered":"2024-09-13T14:51:58.016124+0000","last_clean":"2024-09-13T14:51:58.016124+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:51:58.016124+0000","last_undegraded":"2024-09-13T14:51:58.016124+0000","last_fullsized":"2024-09-13T14:51:58.016124+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"231'6996","reported_seq":10200,"reported_epoch":231,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:14.235481+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:52:14.235481+0000","last_peered":"2024-09-13T14:52:14.235481+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:52:14.235481+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"164'600","ondisk_log_start":"164'600","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250740736,"num_objects":3826,"num_object_clones":0,"num_object_copies":11478,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":3112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":108396544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":711},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":206,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.051754+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:52:09.051754+0000","last_peered":"2024-09-13T14:52:09.051754+0000","last_clean":"2024-09-13T14:52:09.051754+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:52:09.051754+0000","last_undegraded":"2024-09-13T14:52:09.051754+0000","last_fullsized":"2024-09-13T14:52:09.051754+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":240,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.016411+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:51:58.016411+0000","last_peered":"2024-09-13T14:51:58.016411+0000","last_clean":"2024-09-13T14:51:58.016411+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:51:58.016411+0000","last_undegraded":"2024-09-13T14:51:58.016411+0000","last_fullsized":"2024-09-13T14:51:58.016411+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"231'6758","reported_seq":11176,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:14.474509+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:52:14.474509+0000","last_peered":"2024-09-13T14:52:14.474509+0000","last_clean":"2024-09-13T14:52:14.474509+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:52:14.474509+0000","last_undegraded":"2024-09-13T14:52:14.474509+0000","last_fullsized":"2024-09-13T14:52:14.474509+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245563416,"num_objects":3748,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6758,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":58,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.055697+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:52:09.055697+0000","last_peered":"2024-09-13T14:52:09.055697+0000","last_clean":"2024-09-13T14:52:09.055697+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:52:09.055697+0000","last_undegraded":"2024-09-13T14:52:09.055697+0000","last_fullsized":"2024-09-13T14:52:09.055697+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":168,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.058668+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:52:09.058668+0000","last_peered":"2024-09-13T14:52:09.058668+0000","last_clean":"2024-09-13T14:52:09.058668+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:52:09.058668+0000","last_undegraded":"2024-09-13T14:52:09.058668+0000","last_fullsized":"2024-09-13T14:52:09.058668+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"231'6831","reported_seq":8070,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:14.210513+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:52:14.210513+0000","last_peered":"2024-09-13T14:52:14.210513+0000","last_clean":"2024-09-13T14:52:14.210513+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:52:14.210513+0000","last_undegraded":"2024-09-13T14:52:14.210513+0000","last_fullsized":"2024-09-13T14:52:14.210513+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6666,"log_dups_size":0,"ondisk_log_size":6666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256442368,"num_objects":3913,"num_object_clones":0,"num_object_copies":11739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":515,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.053115+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:52:09.053115+0000","last_peered":"2024-09-13T14:52:09.053115+0000","last_clean":"2024-09-13T14:52:09.053115+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:52:09.053115+0000","last_undegraded":"2024-09-13T14:52:09.053115+0000","last_fullsized":"2024-09-13T14:52:09.053115+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":797,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.019887+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:51:58.019887+0000","last_peered":"2024-09-13T14:51:58.019887+0000","last_clean":"2024-09-13T14:51:58.019887+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:51:58.019887+0000","last_undegraded":"2024-09-13T14:51:58.019887+0000","last_fullsized":"2024-09-13T14:51:58.019887+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":278,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.055236+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:52:09.055236+0000","last_peered":"2024-09-13T14:52:09.055236+0000","last_clean":"2024-09-13T14:52:09.055236+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:52:09.055236+0000","last_undegraded":"2024-09-13T14:52:09.055236+0000","last_fullsized":"2024-09-13T14:52:09.055236+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":26,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.021775+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:51:58.021775+0000","last_peered":"2024-09-13T14:51:58.021775+0000","last_clean":"2024-09-13T14:51:58.021775+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:51:58.021775+0000","last_undegraded":"2024-09-13T14:51:58.021775+0000","last_fullsized":"2024-09-13T14:51:58.021775+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"231'6958","reported_seq":11663,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:14.701394+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:52:14.701394+0000","last_peered":"2024-09-13T14:52:14.701394+0000","last_clean":"2024-09-13T14:52:14.701394+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:52:14.701394+0000","last_undegraded":"2024-09-13T14:52:14.701394+0000","last_fullsized":"2024-09-13T14:52:14.701394+0000","mapping_epoch":223,"log_start":"169'1118","ondisk_log_start":"169'1118","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5840,"log_dups_size":0,"ondisk_log_size":5840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254935040,"num_objects":3890,"num_object_clones":0,"num_object_copies":11670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6958,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":360,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.049322+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:52:09.049322+0000","last_peered":"2024-09-13T14:52:09.049322+0000","last_clean":"2024-09-13T14:52:09.049322+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:52:09.049322+0000","last_undegraded":"2024-09-13T14:52:09.049322+0000","last_fullsized":"2024-09-13T14:52:09.049322+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":326,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.056924+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:52:09.056924+0000","last_peered":"2024-09-13T14:52:09.056924+0000","last_clean":"2024-09-13T14:52:09.056924+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:52:09.056924+0000","last_undegraded":"2024-09-13T14:52:09.056924+0000","last_fullsized":"2024-09-13T14:52:09.056924+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":337,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.052336+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:52:09.052336+0000","last_peered":"2024-09-13T14:52:09.052336+0000","last_clean":"2024-09-13T14:52:09.052336+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:52:09.052336+0000","last_undegraded":"2024-09-13T14:52:09.052336+0000","last_fullsized":"2024-09-13T14:52:09.052336+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"231'6831","reported_seq":12683,"reported_epoch":231,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:14.171403+0000","last_change":"2024-09-13T14:50:38.024593+0000","last_active":"2024-09-13T14:52:14.171403+0000","last_peered":"2024-09-13T14:52:14.171403+0000","last_clean":"2024-09-13T14:48:36.189224+0000","last_became_active":"2024-09-13T14:50:38.015505+0000","last_became_peered":"2024-09-13T14:50:38.015505+0000","last_unstale":"2024-09-13T14:52:14.171403+0000","last_undegraded":"2024-09-13T14:50:37.965648+0000","last_fullsized":"2024-09-13T14:50:37.942220+0000","mapping_epoch":215,"log_start":"164'400","ondisk_log_start":"164'400","created":160,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6431,"log_dups_size":0,"ondisk_log_size":6431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246743040,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2845,"num_bytes_recovered":186449920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":1704},{"shards":"2,3,5","objects":2061}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":863,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.058456+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:52:09.058456+0000","last_peered":"2024-09-13T14:52:09.058456+0000","last_clean":"2024-09-13T14:52:09.058456+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:52:09.058456+0000","last_undegraded":"2024-09-13T14:52:09.058456+0000","last_fullsized":"2024-09-13T14:52:09.058456+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":317,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.049423+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:52:09.049423+0000","last_peered":"2024-09-13T14:52:09.049423+0000","last_clean":"2024-09-13T14:52:09.049423+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:52:09.049423+0000","last_undegraded":"2024-09-13T14:52:09.049423+0000","last_fullsized":"2024-09-13T14:52:09.049423+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":70,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.049989+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:52:09.049989+0000","last_peered":"2024-09-13T14:52:09.049989+0000","last_clean":"2024-09-13T14:52:09.049989+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:52:09.049989+0000","last_undegraded":"2024-09-13T14:52:09.049989+0000","last_fullsized":"2024-09-13T14:52:09.049989+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"231'6913","reported_seq":11507,"reported_epoch":231,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:14.147625+0000","last_change":"2024-09-13T14:50:42.308313+0000","last_active":"2024-09-13T14:52:14.147625+0000","last_peered":"2024-09-13T14:52:14.147625+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:52:14.147625+0000","last_undegraded":"2024-09-13T14:50:40.576119+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"164'500","ondisk_log_start":"164'500","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6413,"log_dups_size":0,"ondisk_log_size":6413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249495552,"num_objects":3807,"num_object_clones":0,"num_object_copies":11421,"num_objects_missing_on_primary":89,"num_objects_missing":89,"num_objects_degraded":2392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6913,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":148504576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"1,4,5","objects":9},{"shards":"3,4","objects":1690},{"shards":"3,4,5","objects":2108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":84,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.019874+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:51:58.019874+0000","last_peered":"2024-09-13T14:51:58.019874+0000","last_clean":"2024-09-13T14:51:58.019874+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:51:58.019874+0000","last_undegraded":"2024-09-13T14:51:58.019874+0000","last_fullsized":"2024-09-13T14:51:58.019874+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":215,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.016027+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:51:58.016027+0000","last_peered":"2024-09-13T14:51:58.016027+0000","last_clean":"2024-09-13T14:51:58.016027+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:51:58.016027+0000","last_undegraded":"2024-09-13T14:51:58.016027+0000","last_fullsized":"2024-09-13T14:51:58.016027+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":320,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.050032+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:52:09.050032+0000","last_peered":"2024-09-13T14:52:09.050032+0000","last_clean":"2024-09-13T14:52:09.050032+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:52:09.050032+0000","last_undegraded":"2024-09-13T14:52:09.050032+0000","last_fullsized":"2024-09-13T14:52:09.050032+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"231'6967","reported_seq":11536,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:12.764377+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:52:12.764377+0000","last_peered":"2024-09-13T14:52:12.764377+0000","last_clean":"2024-09-13T14:52:12.764377+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:52:12.764377+0000","last_undegraded":"2024-09-13T14:52:12.764377+0000","last_fullsized":"2024-09-13T14:52:12.764377+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6967,"log_dups_size":0,"ondisk_log_size":6967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252116992,"num_objects":3847,"num_object_clones":0,"num_object_copies":11541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.020097+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:51:58.020097+0000","last_peered":"2024-09-13T14:51:58.020097+0000","last_clean":"2024-09-13T14:51:58.020097+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:51:58.020097+0000","last_undegraded":"2024-09-13T14:51:58.020097+0000","last_fullsized":"2024-09-13T14:51:58.020097+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":872,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.051927+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:52:09.051927+0000","last_peered":"2024-09-13T14:52:09.051927+0000","last_clean":"2024-09-13T14:52:09.051927+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:52:09.051927+0000","last_undegraded":"2024-09-13T14:52:09.051927+0000","last_fullsized":"2024-09-13T14:52:09.051927+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":297,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.050302+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:52:09.050302+0000","last_peered":"2024-09-13T14:52:09.050302+0000","last_clean":"2024-09-13T14:52:09.050302+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:52:09.050302+0000","last_undegraded":"2024-09-13T14:52:09.050302+0000","last_fullsized":"2024-09-13T14:52:09.050302+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"231'7012","reported_seq":11709,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:14.688208+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:52:14.688208+0000","last_peered":"2024-09-13T14:52:14.688208+0000","last_clean":"2024-09-13T14:52:14.688208+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:52:14.688208+0000","last_undegraded":"2024-09-13T14:52:14.688208+0000","last_fullsized":"2024-09-13T14:52:14.688208+0000","mapping_epoch":218,"log_start":"169'1145","ondisk_log_start":"169'1145","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255328256,"num_objects":3896,"num_object_clones":0,"num_object_copies":11688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7012,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":324,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.051816+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:52:09.051816+0000","last_peered":"2024-09-13T14:52:09.051816+0000","last_clean":"2024-09-13T14:52:09.051816+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:52:09.051816+0000","last_undegraded":"2024-09-13T14:52:09.051816+0000","last_fullsized":"2024-09-13T14:52:09.051816+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":546,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.053429+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:52:09.053429+0000","last_peered":"2024-09-13T14:52:09.053429+0000","last_clean":"2024-09-13T14:52:09.053429+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:52:09.053429+0000","last_undegraded":"2024-09-13T14:52:09.053429+0000","last_fullsized":"2024-09-13T14:52:09.053429+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":322,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.051608+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:52:09.051608+0000","last_peered":"2024-09-13T14:52:09.051608+0000","last_clean":"2024-09-13T14:52:09.051608+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:52:09.051608+0000","last_undegraded":"2024-09-13T14:52:09.051608+0000","last_fullsized":"2024-09-13T14:52:09.051608+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"231'6762","reported_seq":8958,"reported_epoch":231,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:13.856609+0000","last_change":"2024-09-13T14:50:19.881564+0000","last_active":"2024-09-13T14:52:13.856609+0000","last_peered":"2024-09-13T14:52:13.856609+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:52:13.856609+0000","last_undegraded":"2024-09-13T14:50:19.795499+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"164'200","ondisk_log_start":"164'200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245628928,"num_objects":3748,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6762,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":1657},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2090}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":342,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.056219+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:52:09.056219+0000","last_peered":"2024-09-13T14:52:09.056219+0000","last_clean":"2024-09-13T14:52:09.056219+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:52:09.056219+0000","last_undegraded":"2024-09-13T14:52:09.056219+0000","last_fullsized":"2024-09-13T14:52:09.056219+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":774,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.053981+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:52:09.053981+0000","last_peered":"2024-09-13T14:52:09.053981+0000","last_clean":"2024-09-13T14:52:09.053981+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:52:09.053981+0000","last_undegraded":"2024-09-13T14:52:09.053981+0000","last_fullsized":"2024-09-13T14:52:09.053981+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":58,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.055618+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:52:09.055618+0000","last_peered":"2024-09-13T14:52:09.055618+0000","last_clean":"2024-09-13T14:52:09.055618+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:52:09.055618+0000","last_undegraded":"2024-09-13T14:52:09.055618+0000","last_fullsized":"2024-09-13T14:52:09.055618+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"231'6931","reported_seq":8496,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:13.778280+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:52:13.778280+0000","last_peered":"2024-09-13T14:52:13.778280+0000","last_clean":"2024-09-13T14:52:13.778280+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:52:13.778280+0000","last_undegraded":"2024-09-13T14:52:13.778280+0000","last_fullsized":"2024-09-13T14:52:13.778280+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6931,"log_dups_size":0,"ondisk_log_size":6931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257490944,"num_objects":3929,"num_object_clones":0,"num_object_copies":11787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6931,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":317,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.052034+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:52:09.052034+0000","last_peered":"2024-09-13T14:52:09.052034+0000","last_clean":"2024-09-13T14:52:09.052034+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:52:09.052034+0000","last_undegraded":"2024-09-13T14:52:09.052034+0000","last_fullsized":"2024-09-13T14:52:09.052034+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":782,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.022064+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:51:58.022064+0000","last_peered":"2024-09-13T14:51:58.022064+0000","last_clean":"2024-09-13T14:51:58.022064+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:51:58.022064+0000","last_undegraded":"2024-09-13T14:51:58.022064+0000","last_fullsized":"2024-09-13T14:51:58.022064+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":217,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.015771+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:51:58.015771+0000","last_peered":"2024-09-13T14:51:58.015771+0000","last_clean":"2024-09-13T14:51:58.015771+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:51:58.015771+0000","last_undegraded":"2024-09-13T14:51:58.015771+0000","last_fullsized":"2024-09-13T14:51:58.015771+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":213,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.016007+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:51:58.016007+0000","last_peered":"2024-09-13T14:51:58.016007+0000","last_clean":"2024-09-13T14:51:58.016007+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:51:58.016007+0000","last_undegraded":"2024-09-13T14:51:58.016007+0000","last_fullsized":"2024-09-13T14:51:58.016007+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":261,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.016216+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:51:58.016216+0000","last_peered":"2024-09-13T14:51:58.016216+0000","last_clean":"2024-09-13T14:51:58.016216+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:51:58.016216+0000","last_undegraded":"2024-09-13T14:51:58.016216+0000","last_fullsized":"2024-09-13T14:51:58.016216+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":321,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.049788+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:52:09.049788+0000","last_peered":"2024-09-13T14:52:09.049788+0000","last_clean":"2024-09-13T14:52:09.049788+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:52:09.049788+0000","last_undegraded":"2024-09-13T14:52:09.049788+0000","last_fullsized":"2024-09-13T14:52:09.049788+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":162,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.061263+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:52:09.061263+0000","last_peered":"2024-09-13T14:52:09.061263+0000","last_clean":"2024-09-13T14:52:09.061263+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:52:09.061263+0000","last_undegraded":"2024-09-13T14:52:09.061263+0000","last_fullsized":"2024-09-13T14:52:09.061263+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":93,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.020156+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:51:58.020156+0000","last_peered":"2024-09-13T14:51:58.020156+0000","last_clean":"2024-09-13T14:51:58.020156+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:51:58.020156+0000","last_undegraded":"2024-09-13T14:51:58.020156+0000","last_fullsized":"2024-09-13T14:51:58.020156+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":299,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.055236+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:52:09.055236+0000","last_peered":"2024-09-13T14:52:09.055236+0000","last_clean":"2024-09-13T14:52:09.055236+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:52:09.055236+0000","last_undegraded":"2024-09-13T14:52:09.055236+0000","last_fullsized":"2024-09-13T14:52:09.055236+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":354,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.052510+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:52:09.052510+0000","last_peered":"2024-09-13T14:52:09.052510+0000","last_clean":"2024-09-13T14:52:09.052510+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:52:09.052510+0000","last_undegraded":"2024-09-13T14:52:09.052510+0000","last_fullsized":"2024-09-13T14:52:09.052510+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":350,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.055900+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:52:09.055900+0000","last_peered":"2024-09-13T14:52:09.055900+0000","last_clean":"2024-09-13T14:52:09.055900+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:52:09.055900+0000","last_undegraded":"2024-09-13T14:52:09.055900+0000","last_fullsized":"2024-09-13T14:52:09.055900+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":123,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.021983+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:51:58.021983+0000","last_peered":"2024-09-13T14:51:58.021983+0000","last_clean":"2024-09-13T14:51:58.021983+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:51:58.021983+0000","last_undegraded":"2024-09-13T14:51:58.021983+0000","last_fullsized":"2024-09-13T14:51:58.021983+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":317,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.057808+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:52:09.057808+0000","last_peered":"2024-09-13T14:52:09.057808+0000","last_clean":"2024-09-13T14:52:09.057808+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:52:09.057808+0000","last_undegraded":"2024-09-13T14:52:09.057808+0000","last_fullsized":"2024-09-13T14:52:09.057808+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":334,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.052265+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:52:09.052265+0000","last_peered":"2024-09-13T14:52:09.052265+0000","last_clean":"2024-09-13T14:52:09.052265+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:52:09.052265+0000","last_undegraded":"2024-09-13T14:52:09.052265+0000","last_fullsized":"2024-09-13T14:52:09.052265+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":302,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.049524+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:52:09.049524+0000","last_peered":"2024-09-13T14:52:09.049524+0000","last_clean":"2024-09-13T14:52:09.049524+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:52:09.049524+0000","last_undegraded":"2024-09-13T14:52:09.049524+0000","last_fullsized":"2024-09-13T14:52:09.049524+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":166,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.061158+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:52:09.061158+0000","last_peered":"2024-09-13T14:52:09.061158+0000","last_clean":"2024-09-13T14:52:09.061158+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:52:09.061158+0000","last_undegraded":"2024-09-13T14:52:09.061158+0000","last_fullsized":"2024-09-13T14:52:09.061158+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":211,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.015994+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:51:58.015994+0000","last_peered":"2024-09-13T14:51:58.015994+0000","last_clean":"2024-09-13T14:51:58.015994+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:51:58.015994+0000","last_undegraded":"2024-09-13T14:51:58.015994+0000","last_fullsized":"2024-09-13T14:51:58.015994+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":61,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.058692+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:52:09.058692+0000","last_peered":"2024-09-13T14:52:09.058692+0000","last_clean":"2024-09-13T14:52:09.058692+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:52:09.058692+0000","last_undegraded":"2024-09-13T14:52:09.058692+0000","last_fullsized":"2024-09-13T14:52:09.058692+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":326,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.051509+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:52:09.051509+0000","last_peered":"2024-09-13T14:52:09.051509+0000","last_clean":"2024-09-13T14:52:09.051509+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:52:09.051509+0000","last_undegraded":"2024-09-13T14:52:09.051509+0000","last_fullsized":"2024-09-13T14:52:09.051509+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":212,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.015915+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:51:58.015915+0000","last_peered":"2024-09-13T14:51:58.015915+0000","last_clean":"2024-09-13T14:51:58.015915+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:51:58.015915+0000","last_undegraded":"2024-09-13T14:51:58.015915+0000","last_fullsized":"2024-09-13T14:51:58.015915+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":68,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.020519+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:51:58.020519+0000","last_peered":"2024-09-13T14:51:58.020519+0000","last_clean":"2024-09-13T14:51:58.020519+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:51:58.020519+0000","last_undegraded":"2024-09-13T14:51:58.020519+0000","last_fullsized":"2024-09-13T14:51:58.020519+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":209,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.015907+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:51:58.015907+0000","last_peered":"2024-09-13T14:51:58.015907+0000","last_clean":"2024-09-13T14:51:58.015907+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:51:58.015907+0000","last_undegraded":"2024-09-13T14:51:58.015907+0000","last_fullsized":"2024-09-13T14:51:58.015907+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":70,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.020239+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:51:58.020239+0000","last_peered":"2024-09-13T14:51:58.020239+0000","last_clean":"2024-09-13T14:51:58.020239+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:51:58.020239+0000","last_undegraded":"2024-09-13T14:51:58.020239+0000","last_fullsized":"2024-09-13T14:51:58.020239+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":320,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.055751+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:52:09.055751+0000","last_peered":"2024-09-13T14:52:09.055751+0000","last_clean":"2024-09-13T14:52:09.055751+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:52:09.055751+0000","last_undegraded":"2024-09-13T14:52:09.055751+0000","last_fullsized":"2024-09-13T14:52:09.055751+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":336,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.073967+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:52:09.073967+0000","last_peered":"2024-09-13T14:52:09.073967+0000","last_clean":"2024-09-13T14:52:09.073967+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:52:09.073967+0000","last_undegraded":"2024-09-13T14:52:09.073967+0000","last_fullsized":"2024-09-13T14:52:09.073967+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":239,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-13T14:51:58.015720+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:51:58.015720+0000","last_peered":"2024-09-13T14:51:58.015720+0000","last_clean":"2024-09-13T14:51:58.015720+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:51:58.015720+0000","last_undegraded":"2024-09-13T14:51:58.015720+0000","last_fullsized":"2024-09-13T14:51:58.015720+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":389,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-13T14:52:09.052427+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:52:09.052427+0000","last_peered":"2024-09-13T14:52:09.052427+0000","last_clean":"2024-09-13T14:52:09.052427+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:52:09.052427+0000","last_undegraded":"2024-09-13T14:52:09.052427+0000","last_fullsized":"2024-09-13T14:52:09.052427+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4023517208,"num_objects":61395,"num_object_clones":0,"num_object_copies":184185,"num_objects_missing_on_primary":176,"num_objects_missing":176,"num_objects_degraded":15006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110153,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32208,"num_bytes_recovered":2068709376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13179826176,"data_stored":13179813960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105125,"ondisk_log_size":105125,"up":48,"acting":41,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234298,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495934,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2276224,"kb_used_data":2020280,"kb_used_omap":1,"kb_used_meta":255870,"kb_avail":91468928,"statfs":{"total":95995035648,"available":93664182272,"internally_reserved":0,"allocated":2068766720,"data_stored":2067488933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":262011335},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757569,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2260472,"kb_used_data":2080880,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":91484680,"statfs":{"total":95995035648,"available":93680312320,"internally_reserved":0,"allocated":2130821120,"data_stored":2129547128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":183892423},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019209,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1532096,"kb_used_data":1236024,"kb_used_omap":1,"kb_used_meta":296062,"kb_avail":92213056,"statfs":{"total":95995035648,"available":94426169344,"internally_reserved":0,"allocated":1265688576,"data_stored":1264410850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":303167942},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468094,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2795636,"kb_used_data":2583532,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":90949516,"statfs":{"total":95995035648,"available":93132304384,"internally_reserved":0,"allocated":2645536768,"data_stored":2644266622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":217181348},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,0,0,0,0,5,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631623,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2270084,"kb_used_data":1949628,"kb_used_omap":8,"kb_used_meta":320439,"kb_avail":91475068,"statfs":{"total":95995035648,"available":93670469632,"internally_reserved":0,"allocated":1996419072,"data_stored":1995137776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":328130527},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795146,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2060348,"kb_used_data":1817652,"kb_used_omap":7,"kb_used_meta":242680,"kb_avail":91684804,"statfs":{"total":95995035648,"available":93885239296,"internally_reserved":0,"allocated":1861275648,"data_stored":1859997759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":248504946},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958676,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418624,"kb_used_data":1130104,"kb_used_omap":7,"kb_used_meta":288504,"kb_avail":92326528,"statfs":{"total":95995035648,"available":94542364672,"internally_reserved":0,"allocated":1157226496,"data_stored":1155945164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":295428583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1149440000,"data_stored":1149435928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1852313600,"data_stored":1852309528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1985675264,"data_stored":1985675264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2633367552,"data_stored":2633367552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1257701376,"data_stored":1257701376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2127302656,"data_stored":2127298584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2063073280,"data_stored":2063073280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":110952448,"data_stored":110952448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:52:21.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:52:21.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:21.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:21.614 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:21.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:21.738 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:21.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:52:21.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:21.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:21.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:21.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:21.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:21.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:22.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:22.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:22.173 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:52:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:52:21] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-09-13T14:52:22.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:22.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:22.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:22.368 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:22.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:22 smithi154 ceph-mon[102314]: pgmap v692: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 12110/181047 objects degraded (6.689%); 1769/181047 objects misplaced (0.977%); 3.7 MiB/s, 161 objects/s recovering 2024-09-13T14:52:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:22.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:22 smithi154 ceph-mon[107038]: pgmap v692: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 12110/181047 objects degraded (6.689%); 1769/181047 objects misplaced (0.977%); 3.7 MiB/s, 161 objects/s recovering 2024-09-13T14:52:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:22.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:22.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:22.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:22.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:22.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:22 smithi172 ceph-mon[100696]: pgmap v692: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 178 op/s; 12110/181047 objects degraded (6.689%); 1769/181047 objects misplaced (0.977%); 3.7 MiB/s, 161 objects/s recovering 2024-09-13T14:52:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:22.747 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:22.807 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:22.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:22.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:22.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:22.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:23.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:23.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:23.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:23.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:23.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:23.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:23.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:23.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:23.512 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:52:23.512 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:23.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:23.537 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:52:23.537 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:23.558 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:52:23.559 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:23.581 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:52:23.581 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:23.604 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:52:23.604 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:23.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:23.632 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:52:23.633 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:23.652 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:52:23.652 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:23.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:23.679 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:52:23.679 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:23.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:23.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:23.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:23.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:23.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:24.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:24.038 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:24.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:24.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:24.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:24.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:24.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:24.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:24.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:24.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:24.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:24.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:24.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:24 smithi154 ceph-mon[102314]: pgmap v693: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 211 op/s; 12110/180453 objects degraded (6.711%); 1769/180453 objects misplaced (0.980%); 3.7 MiB/s, 161 objects/s recovering 2024-09-13T14:52:24.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2335041904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:24.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:24 smithi154 ceph-mon[107038]: pgmap v693: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 211 op/s; 12110/180453 objects degraded (6.711%); 1769/180453 objects misplaced (0.980%); 3.7 MiB/s, 161 objects/s recovering 2024-09-13T14:52:24.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2335041904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:24 smithi172 ceph-mon[100696]: pgmap v693: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 211 op/s; 12110/180453 objects degraded (6.711%); 1769/180453 objects misplaced (0.980%); 3.7 MiB/s, 161 objects/s recovering 2024-09-13T14:52:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2335041904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:24.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:52:24.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:24.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:24.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:24.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:24.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:25.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:25.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.335 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:25.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.461 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:25.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:25.594 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 12110/180453 objects degraded (6.711%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:25.595 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 12110/180453 objects degraded (6.711%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:25.664 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:25.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:25.710 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:25.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 12110/180453 objects degraded (6.711%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:25.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:25.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:25.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:25.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:25.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:26.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:26.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:26.189 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:26.211 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:26.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:26.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:26.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:26.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:26.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:26.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:26.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:26.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:26 smithi154 ceph-mon[102314]: pgmap v694: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 12037/176562 objects degraded (6.817%); 1197/176562 objects misplaced (0.678%); 7.9 MiB/s, 203 objects/s recovering 2024-09-13T14:52:26.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:26 smithi154 ceph-mon[107038]: pgmap v694: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 12037/176562 objects degraded (6.817%); 1197/176562 objects misplaced (0.678%); 7.9 MiB/s, 203 objects/s recovering 2024-09-13T14:52:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:26 smithi172 ceph-mon[100696]: pgmap v694: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 324 op/s; 12037/176562 objects degraded (6.817%); 1197/176562 objects misplaced (0.678%); 7.9 MiB/s, 203 objects/s recovering 2024-09-13T14:52:26.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:26.843 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:26.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:26.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:26.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:27.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:27.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:27.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:52:27.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:27.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:27.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:27.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:27.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:27.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:27.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.601 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:27.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:27.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.725 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:27.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:27.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:27.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:27.928 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:27.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:27.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:27.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:28.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:28.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:52:28.156 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12037/176562 objects degraded (6.817%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":963,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":234,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238500,"num_in_osds":7,"osd_in_since":1726238900,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":83},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":58854,"data_bytes":3859945469,"bytes_used":14185832448,"bytes_avail":657779417088,"bytes_total":671965249536,"degraded_objects":12037,"degraded_total":176562,"degraded_ratio":0.068174352352148249,"misplaced_objects":1197,"misplaced_total":176562,"misplaced_ratio":0.0067794882250993985,"recovering_objects_per_sec":180,"recovering_bytes_per_sec":7385866,"recovering_keys_per_sec":0,"num_objects_recovered":1613,"num_bytes_recovered":65863680,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":274},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi154.daekzd","status":"up:active","gid":24539}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"}},"servicemap":{"epoch":25,"modified":"2024-09-13T14:50:26.951855+0000","services":{}},"progress_events":{}} 2024-09-13T14:52:28.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:28.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:28.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:28.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:28.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:28.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:28.587 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:28.610 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:28.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[102314]: pgmap v695: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 274 op/s; 12037/176562 objects degraded (6.817%); 1197/176562 objects misplaced (0.678%); 7.0 MiB/s, 180 objects/s recovering 2024-09-13T14:52:28.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:28.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:28.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4193223811' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:52:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[107038]: pgmap v695: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 274 op/s; 12037/176562 objects degraded (6.817%); 1197/176562 objects misplaced (0.678%); 7.0 MiB/s, 180 objects/s recovering 2024-09-13T14:52:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:28 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4193223811' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:52:28.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:28 smithi172 ceph-mon[100696]: pgmap v695: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 274 op/s; 12037/176562 objects degraded (6.817%); 1197/176562 objects misplaced (0.678%); 7.0 MiB/s, 180 objects/s recovering 2024-09-13T14:52:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:28 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4193223811' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:52:28.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:28.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:28.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:28.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:28.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:29.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:29.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:29.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:29.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:29.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:29.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:29.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:29.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:29.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:29.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:29.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:29.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:29.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:29.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:30.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:30.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:30.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:30.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:30.255 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:30.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:30.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:30.438 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:52:30.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:52:30.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:30.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:30.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:30.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:30 smithi154 ceph-mon[102314]: pgmap v696: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 12037/175755 objects degraded (6.849%); 1197/175755 objects misplaced (0.681%); 6.3 MiB/s, 161 objects/s recovering 2024-09-13T14:52:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 12037/175755 objects degraded (6.849%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:30.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:30 smithi154 ceph-mon[107038]: pgmap v696: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 12037/175755 objects degraded (6.849%); 1197/175755 objects misplaced (0.681%); 6.3 MiB/s, 161 objects/s recovering 2024-09-13T14:52:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 12037/175755 objects degraded (6.849%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:30.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:30 smithi172 ceph-mon[100696]: pgmap v696: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 264 op/s; 12037/175755 objects degraded (6.849%); 1197/175755 objects misplaced (0.681%); 6.3 MiB/s, 161 objects/s recovering 2024-09-13T14:52:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 12037/175755 objects degraded (6.849%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:30.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:30.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:30.895 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:30.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:30.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:31.019 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:31.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:31.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:31.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:31.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:31.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:31.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:31.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:31.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:31.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:31.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:31.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:31.785 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:31.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:31.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:31.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:31.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:32.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.033 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:32.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:32.043 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:32.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:32.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:32.179 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:52:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:52:31] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:52:32.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:32.291 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:32.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:32.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:32.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:32 smithi154 ceph-mon[107038]: pgmap v697: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 309 op/s; 11958/172635 objects degraded (6.927%); 340/172635 objects misplaced (0.197%); 10 MiB/s, 214 objects/s recovering 2024-09-13T14:52:32.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:32 smithi154 ceph-mon[107038]: osdmap e235: 8 total, 8 up, 7 in 2024-09-13T14:52:32.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:32 smithi154 ceph-mon[102314]: pgmap v697: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 309 op/s; 11958/172635 objects degraded (6.927%); 340/172635 objects misplaced (0.197%); 10 MiB/s, 214 objects/s recovering 2024-09-13T14:52:32.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:32 smithi154 ceph-mon[102314]: osdmap e235: 8 total, 8 up, 7 in 2024-09-13T14:52:32.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:32.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:32.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:32.678 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:52:32.679 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:52:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:32.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:32.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:32 smithi172 ceph-mon[100696]: pgmap v697: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 309 op/s; 11958/172635 objects degraded (6.927%); 340/172635 objects misplaced (0.197%); 10 MiB/s, 214 objects/s recovering 2024-09-13T14:52:32.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:32 smithi172 ceph-mon[100696]: osdmap e235: 8 total, 8 up, 7 in 2024-09-13T14:52:32.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:32.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:32.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:32.975 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:32.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:33.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:33.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:33.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:33.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:33.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:33.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:33.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:33.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:33.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:33.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:33.654 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:33 smithi154 ceph-mon[102314]: osdmap e236: 8 total, 8 up, 7 in 2024-09-13T14:52:33.655 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:33 smithi154 ceph-mon[107038]: osdmap e236: 8 total, 8 up, 7 in 2024-09-13T14:52:33.723 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:33 smithi172 ceph-mon[100696]: osdmap e236: 8 total, 8 up, 7 in 2024-09-13T14:52:33.746 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:33.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:33.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:33.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:33.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:34.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:34.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:34.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:34.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:34.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:34.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:34.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:34.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:34.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:34 smithi154 ceph-mon[102314]: pgmap v700: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 163 op/s; 11958/172635 objects degraded (6.927%); 340/172635 objects misplaced (0.197%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:34 smithi154 ceph-mon[107038]: pgmap v700: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 163 op/s; 11958/172635 objects degraded (6.927%); 340/172635 objects misplaced (0.197%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:34.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:34.543 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:34.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:34.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:34 smithi172 ceph-mon[100696]: pgmap v700: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 163 op/s; 11958/172635 objects degraded (6.927%); 340/172635 objects misplaced (0.197%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T14:52:34.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:34.799 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:34.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:34.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:34.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:35.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:35.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:35.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:35.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:35.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:35.307 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:35.336 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:35.386 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:52:35.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:35.432 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:35.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:35.562 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:35.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:35.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 11958/172635 objects degraded (6.927%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3788701397' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:35 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 11958/172635 objects degraded (6.927%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3788701397' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:35 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 11958/172635 objects degraded (6.927%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3788701397' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:35 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:52:35.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:35.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:52:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:35.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:35.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:35.944 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:35.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:36.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:36.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:36.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:36.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:36.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:36.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:36.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:36.487 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:36.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:36.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[107038]: pgmap v701: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 300 op/s; 7326/169341 objects degraded (4.326%); 4047/169341 objects misplaced (2.390%); 12 MiB/s, 218 objects/s recovering 2024-09-13T14:52:36.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[107038]: osdmap e237: 8 total, 8 up, 7 in 2024-09-13T14:52:36.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[102314]: pgmap v701: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 300 op/s; 7326/169341 objects degraded (4.326%); 4047/169341 objects misplaced (2.390%); 12 MiB/s, 218 objects/s recovering 2024-09-13T14:52:36.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:36.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:36.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:36 smithi154 ceph-mon[102314]: osdmap e237: 8 total, 8 up, 7 in 2024-09-13T14:52:36.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:36 smithi172 ceph-mon[100696]: pgmap v701: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 300 op/s; 7326/169341 objects degraded (4.326%); 4047/169341 objects misplaced (2.390%); 12 MiB/s, 218 objects/s recovering 2024-09-13T14:52:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:36 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:36 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:52:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:36 smithi172 ceph-mon[100696]: osdmap e237: 8 total, 8 up, 7 in 2024-09-13T14:52:36.746 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:36.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:36.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:36.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:36.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:37.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:37.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:37.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:37.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:37.236 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:52:37.237 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:52:37.250 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:37.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:37.374 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:37.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:37.500 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:37.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:37.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:37.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:37.750 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:37.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:37.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:37.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:37.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:38.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:38.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:38.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:38.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:38.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:38.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:38.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:38.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:38.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:38.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:38.527 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:38.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:38.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:38 smithi154 ceph-mon[102314]: pgmap v703: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 182 op/s; 7326/169341 objects degraded (4.326%); 4047/169341 objects misplaced (2.390%); 6.1 MiB/s, 135 objects/s recovering 2024-09-13T14:52:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:38 smithi154 ceph-mon[102314]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:38 smithi154 ceph-mon[107038]: pgmap v703: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 182 op/s; 7326/169341 objects degraded (4.326%); 4047/169341 objects misplaced (2.390%); 6.1 MiB/s, 135 objects/s recovering 2024-09-13T14:52:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:38 smithi154 ceph-mon[107038]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:38 smithi172 ceph-mon[100696]: pgmap v703: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 182 op/s; 7326/169341 objects degraded (4.326%); 4047/169341 objects misplaced (2.390%); 6.1 MiB/s, 135 objects/s recovering 2024-09-13T14:52:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:38 smithi172 ceph-mon[100696]: from='client.35238 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:38.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:38.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:38.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:38.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:38.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:39.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:39.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:39.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:39.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:39.279 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:39.363 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:39.365 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-09-13T14:52:36.995022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3702200317,"num_objects":56447,"num_object_clones":0,"num_object_copies":169341,"num_objects_missing_on_primary":101,"num_objects_missing":101,"num_objects_degraded":7326,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":56447,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":116252,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35725,"num_bytes_recovered":2236828805,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108383,"ondisk_log_size":108383,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13164700,"kb_used_data":11374348,"kb_used_omap":32,"kb_used_meta":1790239,"kb_avail":643051364,"statfs":{"total":671965249536,"available":658484596736,"internally_reserved":0,"allocated":11647332352,"data_stored":11638260193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1833205296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-71958528,"num_objects":-1098,"num_object_clones":0,"num_object_copies":-3294,"num_objects_missing_on_primary":-75,"num_objects_missing":-75,"num_objects_degraded":-4632,"num_objects_misplaced":3707,"num_objects_unfound":0,"num_objects_dirty":-1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1098,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":812,"num_bytes_recovered":38535168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001962"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":353,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.417260+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:52:32.417260+0000","last_peered":"2024-09-13T14:52:32.417260+0000","last_clean":"2024-09-13T14:52:32.417260+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:52:32.417260+0000","last_undegraded":"2024-09-13T14:52:32.417260+0000","last_fullsized":"2024-09-13T14:52:32.417260+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":101,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350685+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:52:31.350685+0000","last_peered":"2024-09-13T14:52:31.350685+0000","last_clean":"2024-09-13T14:52:31.350685+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:52:31.350685+0000","last_undegraded":"2024-09-13T14:52:31.350685+0000","last_fullsized":"2024-09-13T14:52:31.350685+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":71,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.357298+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:52:31.357298+0000","last_peered":"2024-09-13T14:52:31.357298+0000","last_clean":"2024-09-13T14:52:31.357298+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:52:31.357298+0000","last_undegraded":"2024-09-13T14:52:31.357298+0000","last_fullsized":"2024-09-13T14:52:31.357298+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":343,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.351685+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:52:32.351685+0000","last_peered":"2024-09-13T14:52:32.351685+0000","last_clean":"2024-09-13T14:52:32.351685+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:52:32.351685+0000","last_undegraded":"2024-09-13T14:52:32.351685+0000","last_fullsized":"2024-09-13T14:52:32.351685+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":343,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352142+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:52:32.352142+0000","last_peered":"2024-09-13T14:52:32.352142+0000","last_clean":"2024-09-13T14:52:32.352142+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:52:32.352142+0000","last_undegraded":"2024-09-13T14:52:32.352142+0000","last_fullsized":"2024-09-13T14:52:32.352142+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":171,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350353+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:52:31.350353+0000","last_peered":"2024-09-13T14:52:31.350353+0000","last_clean":"2024-09-13T14:52:31.350353+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:52:31.350353+0000","last_undegraded":"2024-09-13T14:52:31.350353+0000","last_fullsized":"2024-09-13T14:52:31.350353+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":68,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.360280+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:52:32.360280+0000","last_peered":"2024-09-13T14:52:32.360280+0000","last_clean":"2024-09-13T14:52:32.360280+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:52:32.360280+0000","last_undegraded":"2024-09-13T14:52:32.360280+0000","last_fullsized":"2024-09-13T14:52:32.360280+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":148,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.355019+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:52:31.355019+0000","last_peered":"2024-09-13T14:52:31.355019+0000","last_clean":"2024-09-13T14:52:31.355019+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:52:31.355019+0000","last_undegraded":"2024-09-13T14:52:31.355019+0000","last_fullsized":"2024-09-13T14:52:31.355019+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"236'7309","reported_seq":8139,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.439877+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:52:34.439877+0000","last_peered":"2024-09-13T14:52:34.439877+0000","last_clean":"2024-09-13T14:52:34.439877+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:52:34.439877+0000","last_undegraded":"2024-09-13T14:52:34.439877+0000","last_fullsized":"2024-09-13T14:52:34.439877+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7309,"log_dups_size":0,"ondisk_log_size":7309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235077632,"num_objects":3587,"num_object_clones":0,"num_object_copies":10761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":278,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.356634+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:52:32.356634+0000","last_peered":"2024-09-13T14:52:32.356634+0000","last_clean":"2024-09-13T14:52:32.356634+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:52:32.356634+0000","last_undegraded":"2024-09-13T14:52:32.356634+0000","last_fullsized":"2024-09-13T14:52:32.356634+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":98,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.356733+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:52:31.356733+0000","last_peered":"2024-09-13T14:52:31.356733+0000","last_clean":"2024-09-13T14:52:31.356733+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:52:31.356733+0000","last_undegraded":"2024-09-13T14:52:31.356733+0000","last_fullsized":"2024-09-13T14:52:31.356733+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"236'7101","reported_seq":8702,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.478231+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:52:34.478231+0000","last_peered":"2024-09-13T14:52:34.478231+0000","last_clean":"2024-09-13T14:52:34.478231+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:52:34.478231+0000","last_undegraded":"2024-09-13T14:52:34.478231+0000","last_fullsized":"2024-09-13T14:52:34.478231+0000","mapping_epoch":170,"log_start":"164'700","ondisk_log_start":"164'700","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6401,"log_dups_size":0,"ondisk_log_size":6401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227868672,"num_objects":3477,"num_object_clones":0,"num_object_copies":10431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":100,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.388177+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:52:32.388177+0000","last_peered":"2024-09-13T14:52:32.388177+0000","last_clean":"2024-09-13T14:52:32.388177+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:52:32.388177+0000","last_undegraded":"2024-09-13T14:52:32.388177+0000","last_fullsized":"2024-09-13T14:52:32.388177+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":252,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350452+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:52:31.350452+0000","last_peered":"2024-09-13T14:52:31.350452+0000","last_clean":"2024-09-13T14:52:31.350452+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:52:31.350452+0000","last_undegraded":"2024-09-13T14:52:31.350452+0000","last_fullsized":"2024-09-13T14:52:31.350452+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"236'7284","reported_seq":8978,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:33.932153+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:52:33.932153+0000","last_peered":"2024-09-13T14:52:33.932153+0000","last_clean":"2024-09-13T14:52:33.932153+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:52:33.932153+0000","last_undegraded":"2024-09-13T14:52:33.932153+0000","last_fullsized":"2024-09-13T14:52:33.932153+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233308160,"num_objects":3560,"num_object_clones":0,"num_object_copies":10680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":75,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.357376+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:52:31.357376+0000","last_peered":"2024-09-13T14:52:31.357376+0000","last_clean":"2024-09-13T14:52:31.357376+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:52:31.357376+0000","last_undegraded":"2024-09-13T14:52:31.357376+0000","last_fullsized":"2024-09-13T14:52:31.357376+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.355083+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:52:31.355083+0000","last_peered":"2024-09-13T14:52:31.355083+0000","last_clean":"2024-09-13T14:52:31.355083+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:52:31.355083+0000","last_undegraded":"2024-09-13T14:52:31.355083+0000","last_fullsized":"2024-09-13T14:52:31.355083+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"236'7073","reported_seq":9152,"reported_epoch":236,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:52:32.987027+0000","last_change":"2024-09-13T14:52:31.266342+0000","last_active":"2024-09-13T14:52:32.987027+0000","last_peered":"2024-09-13T14:52:32.987027+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:52:32.987027+0000","last_undegraded":"2024-09-13T14:52:32.987027+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7073,"log_dups_size":0,"ondisk_log_size":7073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225116160,"num_objects":3435,"num_object_clones":0,"num_object_copies":10305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1939,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7073,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62062592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":1496},{"shards":"0,3,6","objects":1939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":87,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.357069+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:52:31.357069+0000","last_peered":"2024-09-13T14:52:31.357069+0000","last_clean":"2024-09-13T14:52:31.357069+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:52:31.357069+0000","last_undegraded":"2024-09-13T14:52:31.357069+0000","last_fullsized":"2024-09-13T14:52:31.357069+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":322,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352676+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:52:32.352676+0000","last_peered":"2024-09-13T14:52:32.352676+0000","last_clean":"2024-09-13T14:52:32.352676+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:52:32.352676+0000","last_undegraded":"2024-09-13T14:52:32.352676+0000","last_fullsized":"2024-09-13T14:52:32.352676+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"236'7184","reported_seq":24432,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.012025+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:52:34.012025+0000","last_peered":"2024-09-13T14:52:34.012025+0000","last_clean":"2024-09-13T14:52:34.012025+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:52:34.012025+0000","last_undegraded":"2024-09-13T14:52:34.012025+0000","last_fullsized":"2024-09-13T14:52:34.012025+0000","mapping_epoch":232,"log_start":"164'600","ondisk_log_start":"164'600","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6584,"log_dups_size":0,"ondisk_log_size":6584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233963520,"num_objects":3570,"num_object_clones":0,"num_object_copies":10710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7184,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":330,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.366541+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:52:32.366541+0000","last_peered":"2024-09-13T14:52:32.366541+0000","last_clean":"2024-09-13T14:52:32.366541+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:52:32.366541+0000","last_undegraded":"2024-09-13T14:52:32.366541+0000","last_fullsized":"2024-09-13T14:52:32.366541+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":218,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350534+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:52:31.350534+0000","last_peered":"2024-09-13T14:52:31.350534+0000","last_clean":"2024-09-13T14:52:31.350534+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:52:31.350534+0000","last_undegraded":"2024-09-13T14:52:31.350534+0000","last_fullsized":"2024-09-13T14:52:31.350534+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"236'7149","reported_seq":7529,"reported_epoch":236,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:34.036929+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:52:34.036929+0000","last_peered":"2024-09-13T14:52:34.036929+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:52:34.036929+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"164'600","ondisk_log_start":"164'600","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6549,"log_dups_size":0,"ondisk_log_size":6549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228786176,"num_objects":3491,"num_object_clones":0,"num_object_copies":10473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7149,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":1311},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":119,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.355186+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:52:31.355186+0000","last_peered":"2024-09-13T14:52:31.355186+0000","last_clean":"2024-09-13T14:52:31.355186+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:52:31.355186+0000","last_undegraded":"2024-09-13T14:52:31.355186+0000","last_fullsized":"2024-09-13T14:52:31.355186+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":215,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350689+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:52:31.350689+0000","last_peered":"2024-09-13T14:52:31.350689+0000","last_clean":"2024-09-13T14:52:31.350689+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:52:31.350689+0000","last_undegraded":"2024-09-13T14:52:31.350689+0000","last_fullsized":"2024-09-13T14:52:31.350689+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"236'7292","reported_seq":11100,"reported_epoch":236,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:34.496635+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:52:34.496635+0000","last_peered":"2024-09-13T14:52:34.496635+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:52:34.496635+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"167'900","ondisk_log_start":"167'900","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231342080,"num_objects":3530,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":2815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7292,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1952,"num_bytes_recovered":127860736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":414},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":211,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350487+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:52:31.350487+0000","last_peered":"2024-09-13T14:52:31.350487+0000","last_clean":"2024-09-13T14:52:31.350487+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:52:31.350487+0000","last_undegraded":"2024-09-13T14:52:31.350487+0000","last_fullsized":"2024-09-13T14:52:31.350487+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":245,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350954+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:52:31.350954+0000","last_peered":"2024-09-13T14:52:31.350954+0000","last_clean":"2024-09-13T14:52:31.350954+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:52:31.350954+0000","last_undegraded":"2024-09-13T14:52:31.350954+0000","last_fullsized":"2024-09-13T14:52:31.350954+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"236'7057","reported_seq":11480,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.380599+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:52:34.380599+0000","last_peered":"2024-09-13T14:52:34.380599+0000","last_clean":"2024-09-13T14:52:34.380599+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:52:34.380599+0000","last_undegraded":"2024-09-13T14:52:34.380599+0000","last_fullsized":"2024-09-13T14:52:34.380599+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7057,"log_dups_size":0,"ondisk_log_size":7057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225968152,"num_objects":3449,"num_object_clones":0,"num_object_copies":10347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7057,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":69,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.359466+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:52:32.359466+0000","last_peered":"2024-09-13T14:52:32.359466+0000","last_clean":"2024-09-13T14:52:32.359466+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:52:32.359466+0000","last_undegraded":"2024-09-13T14:52:32.359466+0000","last_fullsized":"2024-09-13T14:52:32.359466+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":178,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.356681+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:52:32.356681+0000","last_peered":"2024-09-13T14:52:32.356681+0000","last_clean":"2024-09-13T14:52:32.356681+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:52:32.356681+0000","last_undegraded":"2024-09-13T14:52:32.356681+0000","last_fullsized":"2024-09-13T14:52:32.356681+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"236'7139","reported_seq":8384,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.444495+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:52:34.444495+0000","last_peered":"2024-09-13T14:52:34.444495+0000","last_clean":"2024-09-13T14:52:34.444495+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:52:34.444495+0000","last_undegraded":"2024-09-13T14:52:34.444495+0000","last_fullsized":"2024-09-13T14:52:34.444495+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6974,"log_dups_size":0,"ondisk_log_size":6974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236257280,"num_objects":3605,"num_object_clones":0,"num_object_copies":10815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":525,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.351630+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:52:32.351630+0000","last_peered":"2024-09-13T14:52:32.351630+0000","last_clean":"2024-09-13T14:52:32.351630+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:52:32.351630+0000","last_undegraded":"2024-09-13T14:52:32.351630+0000","last_fullsized":"2024-09-13T14:52:32.351630+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":802,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.356858+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:52:31.356858+0000","last_peered":"2024-09-13T14:52:31.356858+0000","last_clean":"2024-09-13T14:52:31.356858+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:52:31.356858+0000","last_undegraded":"2024-09-13T14:52:31.356858+0000","last_fullsized":"2024-09-13T14:52:31.356858+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":289,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.361306+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:52:32.361306+0000","last_peered":"2024-09-13T14:52:32.361306+0000","last_clean":"2024-09-13T14:52:32.361306+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:52:32.361306+0000","last_undegraded":"2024-09-13T14:52:32.361306+0000","last_fullsized":"2024-09-13T14:52:32.361306+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":31,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.355780+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:52:31.355780+0000","last_peered":"2024-09-13T14:52:31.355780+0000","last_clean":"2024-09-13T14:52:31.355780+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:52:31.355780+0000","last_undegraded":"2024-09-13T14:52:31.355780+0000","last_fullsized":"2024-09-13T14:52:31.355780+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"236'7264","reported_seq":11980,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.541778+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:52:34.541778+0000","last_peered":"2024-09-13T14:52:34.541778+0000","last_clean":"2024-09-13T14:52:34.541778+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:52:34.541778+0000","last_undegraded":"2024-09-13T14:52:34.541778+0000","last_fullsized":"2024-09-13T14:52:34.541778+0000","mapping_epoch":223,"log_start":"169'1418","ondisk_log_start":"169'1418","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234881024,"num_objects":3584,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7264,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":370,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.410866+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:52:32.410866+0000","last_peered":"2024-09-13T14:52:32.410866+0000","last_clean":"2024-09-13T14:52:32.410866+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:52:32.410866+0000","last_undegraded":"2024-09-13T14:52:32.410866+0000","last_fullsized":"2024-09-13T14:52:32.410866+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":337,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.354242+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:52:32.354242+0000","last_peered":"2024-09-13T14:52:32.354242+0000","last_clean":"2024-09-13T14:52:32.354242+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:52:32.354242+0000","last_undegraded":"2024-09-13T14:52:32.354242+0000","last_fullsized":"2024-09-13T14:52:32.354242+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":347,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352364+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:52:32.352364+0000","last_peered":"2024-09-13T14:52:32.352364+0000","last_clean":"2024-09-13T14:52:32.352364+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:52:32.352364+0000","last_undegraded":"2024-09-13T14:52:32.352364+0000","last_fullsized":"2024-09-13T14:52:32.352364+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"236'7140","reported_seq":17452,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:33.846633+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:52:33.846633+0000","last_peered":"2024-09-13T14:52:33.846633+0000","last_clean":"2024-09-13T14:52:33.846633+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:52:33.846633+0000","last_undegraded":"2024-09-13T14:52:33.846633+0000","last_fullsized":"2024-09-13T14:52:33.846633+0000","mapping_epoch":235,"log_start":"164'700","ondisk_log_start":"164'700","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7140,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":873,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.359617+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:52:32.359617+0000","last_peered":"2024-09-13T14:52:32.359617+0000","last_clean":"2024-09-13T14:52:32.359617+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:52:32.359617+0000","last_undegraded":"2024-09-13T14:52:32.359617+0000","last_fullsized":"2024-09-13T14:52:32.359617+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":327,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.411098+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:52:32.411098+0000","last_peered":"2024-09-13T14:52:32.411098+0000","last_clean":"2024-09-13T14:52:32.411098+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:52:32.411098+0000","last_undegraded":"2024-09-13T14:52:32.411098+0000","last_fullsized":"2024-09-13T14:52:32.411098+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":81,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.411701+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:52:32.411701+0000","last_peered":"2024-09-13T14:52:32.411701+0000","last_clean":"2024-09-13T14:52:32.411701+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:52:32.411701+0000","last_undegraded":"2024-09-13T14:52:32.411701+0000","last_fullsized":"2024-09-13T14:52:32.411701+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"236'7222","reported_seq":12051,"reported_epoch":236,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:52:34.543990+0000","last_change":"2024-09-13T14:52:31.275276+0000","last_active":"2024-09-13T14:52:34.543990+0000","last_peered":"2024-09-13T14:52:34.543990+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:52:34.543990+0000","last_undegraded":"2024-09-13T14:52:34.543990+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"167'800","ondisk_log_start":"167'800","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229244928,"num_objects":3498,"num_object_clones":0,"num_object_copies":10494,"num_objects_missing_on_primary":14,"num_objects_missing":14,"num_objects_degraded":0,"num_objects_misplaced":2108,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2340,"num_bytes_recovered":149094400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"3,4","objects":1390},{"shards":"3,4,5","objects":2108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":90,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.360181+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:52:32.360181+0000","last_peered":"2024-09-13T14:52:32.360181+0000","last_clean":"2024-09-13T14:52:32.360181+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:52:32.360181+0000","last_undegraded":"2024-09-13T14:52:32.360181+0000","last_fullsized":"2024-09-13T14:52:32.360181+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":220,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350888+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:52:31.350888+0000","last_peered":"2024-09-13T14:52:31.350888+0000","last_clean":"2024-09-13T14:52:31.350888+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:52:31.350888+0000","last_undegraded":"2024-09-13T14:52:31.350888+0000","last_fullsized":"2024-09-13T14:52:31.350888+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":330,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.411789+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:52:32.411789+0000","last_peered":"2024-09-13T14:52:32.411789+0000","last_clean":"2024-09-13T14:52:32.411789+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:52:32.411789+0000","last_undegraded":"2024-09-13T14:52:32.411789+0000","last_fullsized":"2024-09-13T14:52:32.411789+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"236'7261","reported_seq":11840,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.784134+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:52:32.784134+0000","last_peered":"2024-09-13T14:52:32.784134+0000","last_clean":"2024-09-13T14:52:32.784134+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:52:32.784134+0000","last_undegraded":"2024-09-13T14:52:32.784134+0000","last_fullsized":"2024-09-13T14:52:32.784134+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7261,"log_dups_size":0,"ondisk_log_size":7261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":10659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7261,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":90,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.360372+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:52:32.360372+0000","last_peered":"2024-09-13T14:52:32.360372+0000","last_clean":"2024-09-13T14:52:32.360372+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:52:32.360372+0000","last_undegraded":"2024-09-13T14:52:32.360372+0000","last_fullsized":"2024-09-13T14:52:32.360372+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":882,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.412249+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:52:32.412249+0000","last_peered":"2024-09-13T14:52:32.412249+0000","last_clean":"2024-09-13T14:52:32.412249+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:52:32.412249+0000","last_undegraded":"2024-09-13T14:52:32.412249+0000","last_fullsized":"2024-09-13T14:52:32.412249+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":307,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.411613+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:52:32.411613+0000","last_peered":"2024-09-13T14:52:32.411613+0000","last_clean":"2024-09-13T14:52:32.411613+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:52:32.411613+0000","last_undegraded":"2024-09-13T14:52:32.411613+0000","last_fullsized":"2024-09-13T14:52:32.411613+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"236'7335","reported_seq":12042,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.544455+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:52:34.544455+0000","last_peered":"2024-09-13T14:52:34.544455+0000","last_clean":"2024-09-13T14:52:34.544455+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:52:34.544455+0000","last_undegraded":"2024-09-13T14:52:34.544455+0000","last_fullsized":"2024-09-13T14:52:34.544455+0000","mapping_epoch":218,"log_start":"171'1545","ondisk_log_start":"171'1545","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234160128,"num_objects":3573,"num_object_clones":0,"num_object_copies":10719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7335,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":334,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.412164+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:52:32.412164+0000","last_peered":"2024-09-13T14:52:32.412164+0000","last_clean":"2024-09-13T14:52:32.412164+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:52:32.412164+0000","last_undegraded":"2024-09-13T14:52:32.412164+0000","last_fullsized":"2024-09-13T14:52:32.412164+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":556,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352255+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:52:32.352255+0000","last_peered":"2024-09-13T14:52:32.352255+0000","last_clean":"2024-09-13T14:52:32.352255+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:52:32.352255+0000","last_undegraded":"2024-09-13T14:52:32.352255+0000","last_fullsized":"2024-09-13T14:52:32.352255+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":333,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352631+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:52:32.352631+0000","last_peered":"2024-09-13T14:52:32.352631+0000","last_clean":"2024-09-13T14:52:32.352631+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:52:32.352631+0000","last_undegraded":"2024-09-13T14:52:32.352631+0000","last_fullsized":"2024-09-13T14:52:32.352631+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"236'7066","reported_seq":9267,"reported_epoch":236,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:34.007733+0000","last_change":"2024-09-13T14:50:19.881564+0000","last_active":"2024-09-13T14:52:34.007733+0000","last_peered":"2024-09-13T14:52:34.007733+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:52:34.007733+0000","last_undegraded":"2024-09-13T14:50:19.795499+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"164'500","ondisk_log_start":"164'500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6566,"log_dups_size":0,"ondisk_log_size":6566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225705984,"num_objects":3444,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7066,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":68550656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":1353},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2090}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":353,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.360656+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:52:32.360656+0000","last_peered":"2024-09-13T14:52:32.360656+0000","last_clean":"2024-09-13T14:52:32.360656+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:52:32.360656+0000","last_undegraded":"2024-09-13T14:52:32.360656+0000","last_fullsized":"2024-09-13T14:52:32.360656+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":784,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.412159+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:52:32.412159+0000","last_peered":"2024-09-13T14:52:32.412159+0000","last_clean":"2024-09-13T14:52:32.412159+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:52:32.412159+0000","last_undegraded":"2024-09-13T14:52:32.412159+0000","last_fullsized":"2024-09-13T14:52:32.412159+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":69,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.359310+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:52:32.359310+0000","last_peered":"2024-09-13T14:52:32.359310+0000","last_clean":"2024-09-13T14:52:32.359310+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:52:32.359310+0000","last_undegraded":"2024-09-13T14:52:32.359310+0000","last_fullsized":"2024-09-13T14:52:32.359310+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"236'7253","reported_seq":8823,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:33.681381+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:52:33.681381+0000","last_peered":"2024-09-13T14:52:33.681381+0000","last_clean":"2024-09-13T14:52:33.681381+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:52:33.681381+0000","last_undegraded":"2024-09-13T14:52:33.681381+0000","last_fullsized":"2024-09-13T14:52:33.681381+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7253,"log_dups_size":0,"ondisk_log_size":7253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236388352,"num_objects":3607,"num_object_clones":0,"num_object_copies":10821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7253,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":327,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352044+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:52:32.352044+0000","last_peered":"2024-09-13T14:52:32.352044+0000","last_clean":"2024-09-13T14:52:32.352044+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:52:32.352044+0000","last_undegraded":"2024-09-13T14:52:32.352044+0000","last_fullsized":"2024-09-13T14:52:32.352044+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":787,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.355290+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:52:31.355290+0000","last_peered":"2024-09-13T14:52:31.355290+0000","last_clean":"2024-09-13T14:52:31.355290+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:52:31.355290+0000","last_undegraded":"2024-09-13T14:52:31.355290+0000","last_fullsized":"2024-09-13T14:52:31.355290+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":222,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350378+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:52:31.350378+0000","last_peered":"2024-09-13T14:52:31.350378+0000","last_clean":"2024-09-13T14:52:31.350378+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:52:31.350378+0000","last_undegraded":"2024-09-13T14:52:31.350378+0000","last_fullsized":"2024-09-13T14:52:31.350378+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":218,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350819+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:52:31.350819+0000","last_peered":"2024-09-13T14:52:31.350819+0000","last_clean":"2024-09-13T14:52:31.350819+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:52:31.350819+0000","last_undegraded":"2024-09-13T14:52:31.350819+0000","last_fullsized":"2024-09-13T14:52:31.350819+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":266,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350781+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:52:31.350781+0000","last_peered":"2024-09-13T14:52:31.350781+0000","last_clean":"2024-09-13T14:52:31.350781+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:52:31.350781+0000","last_undegraded":"2024-09-13T14:52:31.350781+0000","last_fullsized":"2024-09-13T14:52:31.350781+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":332,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.411459+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:52:32.411459+0000","last_peered":"2024-09-13T14:52:32.411459+0000","last_clean":"2024-09-13T14:52:32.411459+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:52:32.411459+0000","last_undegraded":"2024-09-13T14:52:32.411459+0000","last_fullsized":"2024-09-13T14:52:32.411459+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":172,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.362196+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:52:32.362196+0000","last_peered":"2024-09-13T14:52:32.362196+0000","last_clean":"2024-09-13T14:52:32.362196+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:52:32.362196+0000","last_undegraded":"2024-09-13T14:52:32.362196+0000","last_fullsized":"2024-09-13T14:52:32.362196+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":98,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.357004+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:52:31.357004+0000","last_peered":"2024-09-13T14:52:31.357004+0000","last_clean":"2024-09-13T14:52:31.357004+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:52:31.357004+0000","last_undegraded":"2024-09-13T14:52:31.357004+0000","last_fullsized":"2024-09-13T14:52:31.357004+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":309,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.359310+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:52:32.359310+0000","last_peered":"2024-09-13T14:52:32.359310+0000","last_clean":"2024-09-13T14:52:32.359310+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:52:32.359310+0000","last_undegraded":"2024-09-13T14:52:32.359310+0000","last_fullsized":"2024-09-13T14:52:32.359310+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":364,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352499+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:52:32.352499+0000","last_peered":"2024-09-13T14:52:32.352499+0000","last_clean":"2024-09-13T14:52:32.352499+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:52:32.352499+0000","last_undegraded":"2024-09-13T14:52:32.352499+0000","last_fullsized":"2024-09-13T14:52:32.352499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":360,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.359619+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:52:32.359619+0000","last_peered":"2024-09-13T14:52:32.359619+0000","last_clean":"2024-09-13T14:52:32.359619+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:52:32.359619+0000","last_undegraded":"2024-09-13T14:52:32.359619+0000","last_fullsized":"2024-09-13T14:52:32.359619+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":128,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.355226+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:52:31.355226+0000","last_peered":"2024-09-13T14:52:31.355226+0000","last_clean":"2024-09-13T14:52:31.355226+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:52:31.355226+0000","last_undegraded":"2024-09-13T14:52:31.355226+0000","last_fullsized":"2024-09-13T14:52:31.355226+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":327,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.359749+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:52:32.359749+0000","last_peered":"2024-09-13T14:52:32.359749+0000","last_clean":"2024-09-13T14:52:32.359749+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:52:32.359749+0000","last_undegraded":"2024-09-13T14:52:32.359749+0000","last_fullsized":"2024-09-13T14:52:32.359749+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":344,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352287+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:52:32.352287+0000","last_peered":"2024-09-13T14:52:32.352287+0000","last_clean":"2024-09-13T14:52:32.352287+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:52:32.352287+0000","last_undegraded":"2024-09-13T14:52:32.352287+0000","last_fullsized":"2024-09-13T14:52:32.352287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":312,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.411216+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:52:32.411216+0000","last_peered":"2024-09-13T14:52:32.411216+0000","last_clean":"2024-09-13T14:52:32.411216+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:52:32.411216+0000","last_undegraded":"2024-09-13T14:52:32.411216+0000","last_fullsized":"2024-09-13T14:52:32.411216+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":176,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.362103+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:52:32.362103+0000","last_peered":"2024-09-13T14:52:32.362103+0000","last_clean":"2024-09-13T14:52:32.362103+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:52:32.362103+0000","last_undegraded":"2024-09-13T14:52:32.362103+0000","last_fullsized":"2024-09-13T14:52:32.362103+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":216,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.352876+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:52:31.352876+0000","last_peered":"2024-09-13T14:52:31.352876+0000","last_clean":"2024-09-13T14:52:31.352876+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:52:31.352876+0000","last_undegraded":"2024-09-13T14:52:31.352876+0000","last_fullsized":"2024-09-13T14:52:31.352876+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":71,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.355571+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:52:32.355571+0000","last_peered":"2024-09-13T14:52:32.355571+0000","last_clean":"2024-09-13T14:52:32.355571+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:52:32.355571+0000","last_undegraded":"2024-09-13T14:52:32.355571+0000","last_fullsized":"2024-09-13T14:52:32.355571+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":337,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352551+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:52:32.352551+0000","last_peered":"2024-09-13T14:52:32.352551+0000","last_clean":"2024-09-13T14:52:32.352551+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:52:32.352551+0000","last_undegraded":"2024-09-13T14:52:32.352551+0000","last_fullsized":"2024-09-13T14:52:32.352551+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":217,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350766+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:52:31.350766+0000","last_peered":"2024-09-13T14:52:31.350766+0000","last_clean":"2024-09-13T14:52:31.350766+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:52:31.350766+0000","last_undegraded":"2024-09-13T14:52:31.350766+0000","last_fullsized":"2024-09-13T14:52:31.350766+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":73,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.356940+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:52:31.356940+0000","last_peered":"2024-09-13T14:52:31.356940+0000","last_clean":"2024-09-13T14:52:31.356940+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:52:31.356940+0000","last_undegraded":"2024-09-13T14:52:31.356940+0000","last_fullsized":"2024-09-13T14:52:31.356940+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":214,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.352823+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:52:31.352823+0000","last_peered":"2024-09-13T14:52:31.352823+0000","last_clean":"2024-09-13T14:52:31.352823+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:52:31.352823+0000","last_undegraded":"2024-09-13T14:52:31.352823+0000","last_fullsized":"2024-09-13T14:52:31.352823+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":75,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.357345+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:52:31.357345+0000","last_peered":"2024-09-13T14:52:31.357345+0000","last_clean":"2024-09-13T14:52:31.357345+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:52:31.357345+0000","last_undegraded":"2024-09-13T14:52:31.357345+0000","last_fullsized":"2024-09-13T14:52:31.357345+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":330,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.361122+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:52:32.361122+0000","last_peered":"2024-09-13T14:52:32.361122+0000","last_clean":"2024-09-13T14:52:32.361122+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:52:32.361122+0000","last_undegraded":"2024-09-13T14:52:32.361122+0000","last_fullsized":"2024-09-13T14:52:32.361122+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":347,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.360803+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:52:32.360803+0000","last_peered":"2024-09-13T14:52:32.360803+0000","last_clean":"2024-09-13T14:52:32.360803+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:52:32.360803+0000","last_undegraded":"2024-09-13T14:52:32.360803+0000","last_fullsized":"2024-09-13T14:52:32.360803+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":245,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T14:52:31.350367+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:52:31.350367+0000","last_peered":"2024-09-13T14:52:31.350367+0000","last_clean":"2024-09-13T14:52:31.350367+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:52:31.350367+0000","last_undegraded":"2024-09-13T14:52:31.350367+0000","last_fullsized":"2024-09-13T14:52:31.350367+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":400,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.352449+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:52:32.352449+0000","last_peered":"2024-09-13T14:52:32.352449+0000","last_clean":"2024-09-13T14:52:32.352449+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:52:32.352449+0000","last_undegraded":"2024-09-13T14:52:32.352449+0000","last_fullsized":"2024-09-13T14:52:32.352449+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3697410072,"num_objects":56419,"num_object_clones":0,"num_object_copies":169257,"num_objects_missing_on_primary":101,"num_objects_missing":101,"num_objects_degraded":7326,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":56419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115129,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35569,"num_bytes_recovered":2234449920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11700219904,"data_stored":11700207688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107201,"ondisk_log_size":107201,"up":48,"acting":43,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234302,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495938,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2221632,"kb_used_data":1978036,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91523520,"statfs":{"total":95995035648,"available":93720084480,"internally_reserved":0,"allocated":2025508864,"data_stored":2024211663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":249428423},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757573,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1572920,"kb_used_data":1401516,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92172232,"statfs":{"total":95995035648,"available":94384365568,"internally_reserved":0,"allocated":1435152384,"data_stored":1433858978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175503815},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019213,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418048,"kb_used_data":1134260,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":92327104,"statfs":{"total":95995035648,"available":94542954496,"internally_reserved":0,"allocated":1161482240,"data_stored":1160185100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":290585030},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468098,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2570228,"kb_used_data":2362280,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":91174924,"statfs":{"total":95995035648,"available":93363122176,"internally_reserved":0,"allocated":2418974720,"data_stored":2417685160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":212921508},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631627,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2142400,"kb_used_data":1785848,"kb_used_omap":8,"kb_used_meta":356535,"kb_avail":91602752,"statfs":{"total":95995035648,"available":93801218048,"internally_reserved":0,"allocated":1828708352,"data_stored":1827407642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":365092831},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795150,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1894460,"kb_used_data":1660080,"kb_used_omap":7,"kb_used_meta":234360,"kb_avail":91850692,"statfs":{"total":95995035648,"available":94055108608,"internally_reserved":0,"allocated":1699921920,"data_stored":1698624617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":239985266},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958680,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1345012,"kb_used_data":1052328,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":92400140,"statfs":{"total":95995035648,"available":94617743360,"internally_reserved":0,"allocated":1077583872,"data_stored":1076287033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":299688423},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1065684992,"data_stored":1065680920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1690767360,"data_stored":1690763288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1822162944,"data_stored":1822162944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2405761024,"data_stored":2405761024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1156251648,"data_stored":1156251648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1431310336,"data_stored":1431306264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2021195776,"data_stored":2021195776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":107085824,"data_stored":107085824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:52:39.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.403 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:39.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:39.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:39.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:39.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:52:39.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:39.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:39.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:39.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:39.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:40.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.046 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:40.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:40.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:40.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:40.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:40.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:40.414 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:40.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:40.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:40.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:40.626 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:40 smithi154 ceph-mon[102314]: pgmap v704: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 183 op/s; 7326/168435 objects degraded (4.349%); 3538/168435 objects misplaced (2.101%); 9.0 MiB/s, 172 objects/s recovering 2024-09-13T14:52:40.627 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7326/168435 objects degraded (4.349%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:40.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:40 smithi154 ceph-mon[107038]: pgmap v704: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 183 op/s; 7326/168435 objects degraded (4.349%); 3538/168435 objects misplaced (2.101%); 9.0 MiB/s, 172 objects/s recovering 2024-09-13T14:52:40.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7326/168435 objects degraded (4.349%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:40.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:40.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:40.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:40 smithi172 ceph-mon[100696]: pgmap v704: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 183 op/s; 7326/168435 objects degraded (4.349%); 3538/168435 objects misplaced (2.101%); 9.0 MiB/s, 172 objects/s recovering 2024-09-13T14:52:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7326/168435 objects degraded (4.349%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:40.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:40.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:40.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:40.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:40.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:40.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:41.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:41.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:41.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:41.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:41.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:41.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:41.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:41.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:41.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:41.599 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:41.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:41.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:41.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:41.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:41.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:41.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:41.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:42.045 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:52:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:52:41] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:52:42.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:42.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:42.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:42.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:42.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:42.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:42.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:52:42.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:42.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:42.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:42.617 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:42.675 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:42 smithi154 ceph-mon[102314]: pgmap v705: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 7246/165609 objects degraded (4.375%); 3061/165609 objects misplaced (1.848%); 12 MiB/s, 218 objects/s recovering 2024-09-13T14:52:42.675 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:42 smithi154 ceph-mon[107038]: pgmap v705: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 7246/165609 objects degraded (4.375%); 3061/165609 objects misplaced (1.848%); 12 MiB/s, 218 objects/s recovering 2024-09-13T14:52:42.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:42 smithi172 ceph-mon[100696]: pgmap v705: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 270 op/s; 7246/165609 objects degraded (4.375%); 3061/165609 objects misplaced (1.848%); 12 MiB/s, 218 objects/s recovering 2024-09-13T14:52:42.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:42.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:42.844 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:52:42.844 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:42.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:42.872 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:52:42.872 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:42.898 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:52:42.898 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:42.930 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:52:42.930 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:42.958 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:52:42.959 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:42.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:42.990 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:52:42.990 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:42.994 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:42.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:43.014 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:52:43.015 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:43.042 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:52:43.042 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:43.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:43.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:43.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.244 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:43.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:43.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:43.368 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:43.460 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3643862112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:43.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:43.461 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:43.461 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3643862112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:43.461 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:43.461 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:43.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:43.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:43.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:43.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:43.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3643862112' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:43.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:43.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:43.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:43.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:43.903 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:43.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:44.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.028 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:44.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:44.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:44.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:44.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:44.278 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:44.380 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:44.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:44.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:44.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:44.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:44.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:44.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:44 smithi154 ceph-mon[102314]: pgmap v706: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 234 op/s; 7246/165609 objects degraded (4.375%); 3061/165609 objects misplaced (1.848%); 10 MiB/s, 189 objects/s recovering 2024-09-13T14:52:44.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:44 smithi154 ceph-mon[107038]: pgmap v706: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 234 op/s; 7246/165609 objects degraded (4.375%); 3061/165609 objects misplaced (1.848%); 10 MiB/s, 189 objects/s recovering 2024-09-13T14:52:44.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:44 smithi172 ceph-mon[100696]: pgmap v706: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 234 op/s; 7246/165609 objects degraded (4.375%); 3061/165609 objects misplaced (1.848%); 10 MiB/s, 189 objects/s recovering 2024-09-13T14:52:44.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:44.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:44.883 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:44.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:44.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:44.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:45.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:45.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:45.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:45.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:45.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:45.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:45.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:45.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:45.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:45.599 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:45.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:45.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:45.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:45 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7246/165609 objects degraded (4.375%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:45.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:45 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7246/165609 objects degraded (4.375%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:45.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:45 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7246/165609 objects degraded (4.375%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:45.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:45.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:45.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:45.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:45.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:45.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:46.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:46.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:46.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:46.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:46.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:46.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:52:46.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:46.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:46.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:46.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:46.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:46.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:46 smithi154 ceph-mon[102314]: pgmap v707: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 288 op/s; 7166/160698 objects degraded (4.459%); 1895/160698 objects misplaced (1.179%); 14 MiB/s, 232 objects/s recovering 2024-09-13T14:52:46.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:46 smithi154 ceph-mon[107038]: pgmap v707: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 288 op/s; 7166/160698 objects degraded (4.459%); 1895/160698 objects misplaced (1.179%); 14 MiB/s, 232 objects/s recovering 2024-09-13T14:52:46.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:46.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:46 smithi172 ceph-mon[100696]: pgmap v707: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 288 op/s; 7166/160698 objects degraded (4.459%); 1895/160698 objects misplaced (1.179%); 14 MiB/s, 232 objects/s recovering 2024-09-13T14:52:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:46.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:46.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:46.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:46.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:46.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:47.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:47.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:47.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:47.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:47.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:47.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:47.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:47.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:47.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:47.629 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:47.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:47.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:47.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:47.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:47.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:47.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:48.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:48.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:48.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:48.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:48.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:48.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:48.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:48.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.524 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:48.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:48.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:48.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:48.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:48 smithi154 ceph-mon[102314]: pgmap v708: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 247 op/s; 7166/160698 objects degraded (4.459%); 1895/160698 objects misplaced (1.179%); 12 MiB/s, 200 objects/s recovering 2024-09-13T14:52:48.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:48 smithi154 ceph-mon[107038]: pgmap v708: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 247 op/s; 7166/160698 objects degraded (4.459%); 1895/160698 objects misplaced (1.179%); 12 MiB/s, 200 objects/s recovering 2024-09-13T14:52:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:48 smithi172 ceph-mon[100696]: pgmap v708: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 247 op/s; 7166/160698 objects degraded (4.459%); 1895/160698 objects misplaced (1.179%); 12 MiB/s, 200 objects/s recovering 2024-09-13T14:52:48.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:48.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:48.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:48.901 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:48.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:49.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:49.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:49.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:49.280 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:49.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:49.407 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:49.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:49.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:49.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:49.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:49.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:49.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:49.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:49.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:49.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:50.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.032 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:52:50.032 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.44843.0:119396 6.f e238/208 6:f038eff2:::benchmark_data_smithi154_175640_object33620:head v 238'7419, mlcod=181'3140)", 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:50.028923+0000", 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0016994390000000001, 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017317210000000001, 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:52:50.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.028923+0000", 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.028923+0000", 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.028925+0000", 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.234e-06 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.028933+0000", 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3219999999999993e-06 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.028954+0000", 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0883e-05 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.028963+0000", 2024-09-13T14:52:50.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5820000000000008e-06 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.029033+0000", 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9406999999999995e-05 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.029070+0000", 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7218999999999997e-05 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:52:50.036 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:52:50.037 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:52:50.037 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:52:50.037 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 1 2024-09-13T14:52:50.037 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:52:50.039 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:50.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:50.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:50.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:50.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:50.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:50.416 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:50.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:50.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:50.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:50.676 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[107038]: pgmap v709: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 268 op/s; 7166/159666 objects degraded (4.488%); 1177/159666 objects misplaced (0.737%); 16 MiB/s, 253 objects/s recovering 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[107038]: osdmap e238: 8 total, 8 up, 7 in 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7166/159666 objects degraded (4.488%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[102314]: pgmap v709: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 268 op/s; 7166/159666 objects degraded (4.488%); 1177/159666 objects misplaced (0.737%); 16 MiB/s, 253 objects/s recovering 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:50.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[102314]: osdmap e238: 8 total, 8 up, 7 in 2024-09-13T14:52:50.678 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:50 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7166/159666 objects degraded (4.488%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:50.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:50 smithi172 ceph-mon[100696]: pgmap v709: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 268 op/s; 7166/159666 objects degraded (4.488%); 1177/159666 objects misplaced (0.737%); 16 MiB/s, 253 objects/s recovering 2024-09-13T14:52:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:50 smithi172 ceph-mon[100696]: osdmap e238: 8 total, 8 up, 7 in 2024-09-13T14:52:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:50 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7166/159666 objects degraded (4.488%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:50.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:50.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:50.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:50.937 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:50.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:51.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:51.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:51.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:51.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:51.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:51.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:51.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:51.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:51.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:51.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:51.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[102314]: osdmap e239: 8 total, 8 up, 7 in 2024-09-13T14:52:51.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:51.638 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:51.639 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:51.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[107038]: osdmap e239: 8 total, 8 up, 7 in 2024-09-13T14:52:51.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:51.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:51.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:51.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:51.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:51 smithi172 ceph-mon[100696]: osdmap e239: 8 total, 8 up, 7 in 2024-09-13T14:52:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:52:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:52:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:52:51.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:51.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:51.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:51.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:51.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:51.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:51.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:52:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:52:51] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T14:52:52.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:52.073 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:52.075 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:52.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:52:52.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:52.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:52.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.331 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:52.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:52.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:52.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.594 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:52.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:52.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:52 smithi154 ceph-mon[107038]: pgmap v712: 89 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 359 op/s; 4913/156972 objects degraded (3.130%); 2399/156972 objects misplaced (1.528%); 23 MiB/s, 364 objects/s recovering 2024-09-13T14:52:52.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:52 smithi154 ceph-mon[107038]: osdmap e240: 8 total, 8 up, 7 in 2024-09-13T14:52:52.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:52 smithi154 ceph-mon[102314]: pgmap v712: 89 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 359 op/s; 4913/156972 objects degraded (3.130%); 2399/156972 objects misplaced (1.528%); 23 MiB/s, 364 objects/s recovering 2024-09-13T14:52:52.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:52 smithi154 ceph-mon[102314]: osdmap e240: 8 total, 8 up, 7 in 2024-09-13T14:52:52.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:52.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:52 smithi172 ceph-mon[100696]: pgmap v712: 89 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 359 op/s; 4913/156972 objects degraded (3.130%); 2399/156972 objects misplaced (1.528%); 23 MiB/s, 364 objects/s recovering 2024-09-13T14:52:52.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:52 smithi172 ceph-mon[100696]: osdmap e240: 8 total, 8 up, 7 in 2024-09-13T14:52:52.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:52.845 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:52.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:52.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:52.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:53.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:53.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:53.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:53.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:53.348 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:53.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:53.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:53.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:53.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:53.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:53.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:53 smithi154 ceph-mon[102314]: osdmap e241: 8 total, 8 up, 7 in 2024-09-13T14:52:53.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:53 smithi154 ceph-mon[107038]: osdmap e241: 8 total, 8 up, 7 in 2024-09-13T14:52:53.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:53 smithi172 ceph-mon[100696]: osdmap e241: 8 total, 8 up, 7 in 2024-09-13T14:52:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.745 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:53.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:53.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:53.871 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:53.876 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:52:53.877 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:52:53.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:53.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:53.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:54.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:54.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:54.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:54.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:54.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.382 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:54.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:54.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:54.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:54.565 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:54 smithi154 ceph-mon[102314]: pgmap v715: 89 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4913/156972 objects degraded (3.130%); 2399/156972 objects misplaced (1.528%); 15 MiB/s, 238 objects/s recovering 2024-09-13T14:52:54.565 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:54 smithi154 ceph-mon[102314]: from='client.45188 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:54.566 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:54 smithi154 ceph-mon[107038]: pgmap v715: 89 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4913/156972 objects degraded (3.130%); 2399/156972 objects misplaced (1.528%); 15 MiB/s, 238 objects/s recovering 2024-09-13T14:52:54.566 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:54 smithi154 ceph-mon[107038]: from='client.45188 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:54.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.641 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:54.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:54.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:54 smithi172 ceph-mon[100696]: pgmap v715: 89 pgs: 3 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 224 op/s; 4913/156972 objects degraded (3.130%); 2399/156972 objects misplaced (1.528%); 15 MiB/s, 238 objects/s recovering 2024-09-13T14:52:54.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:54 smithi172 ceph-mon[100696]: from='client.45188 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:52:54.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:54.764 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:54.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:54.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:54.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:54.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:55.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:55.051 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:52:55.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.138 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:55.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:55.316 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:52:55.317 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:52:55.337 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:52:55.337 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:52:55.354 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:52:55.354 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:52:55.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:55.374 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:52:55.374 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:52:55.389 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:55.395 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:52:55.395 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:52:55.414 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:52:55.414 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:52:55.437 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:52:55.438 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:52:55.444 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:52:55.461 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:52:55.461 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:52:55.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:55.513 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:55.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.638 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:55.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:55.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4913/156972 objects degraded (3.130%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:55.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/880259041' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:55.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:55.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3573460529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:55.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4913/156972 objects degraded (3.130%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:55.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/880259041' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:55.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:55.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3573460529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:55.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4913/156972 objects degraded (3.130%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-13T14:52:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/880259041' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:55 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:52:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3573460529' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:52:55.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:55.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:55.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:55.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:55.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:56.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:56.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:56.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:56.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:56.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:56.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:56.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.521 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.523 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-09-13T14:52:52.999306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3431995389,"num_objects":52324,"num_object_clones":0,"num_object_copies":156972,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":4913,"num_objects_misplaced":2399,"num_objects_unfound":0,"num_objects_dirty":52324,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":120375,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39724,"num_bytes_recovered":2497924229,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110406,"ondisk_log_size":110406,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12617828,"kb_used_data":10792876,"kb_used_omap":32,"kb_used_meta":1824735,"kb_avail":643598236,"statfs":{"total":671965249536,"available":659044593664,"internally_reserved":0,"allocated":11051905024,"data_stored":11042801553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":1868529200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,1,3,2,2,5,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58851328,"num_objects":-898,"num_object_clones":0,"num_object_copies":-2694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2253,"num_objects_misplaced":1222,"num_objects_unfound":0,"num_objects_dirty":-898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":898,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":954,"num_bytes_recovered":62521344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000950"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":357,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.437887+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:52:49.437887+0000","last_peered":"2024-09-13T14:52:49.437887+0000","last_clean":"2024-09-13T14:52:49.437887+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:52:49.437887+0000","last_undegraded":"2024-09-13T14:52:49.437887+0000","last_fullsized":"2024-09-13T14:52:49.437887+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":103,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.377441+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:52:35.377441+0000","last_peered":"2024-09-13T14:52:35.377441+0000","last_clean":"2024-09-13T14:52:35.377441+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:52:35.377441+0000","last_undegraded":"2024-09-13T14:52:35.377441+0000","last_fullsized":"2024-09-13T14:52:35.377441+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":73,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.383046+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:52:35.383046+0000","last_peered":"2024-09-13T14:52:35.383046+0000","last_clean":"2024-09-13T14:52:35.383046+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:52:35.383046+0000","last_undegraded":"2024-09-13T14:52:35.383046+0000","last_fullsized":"2024-09-13T14:52:35.383046+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":345,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.389793+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:52:35.389793+0000","last_peered":"2024-09-13T14:52:35.389793+0000","last_clean":"2024-09-13T14:52:35.389793+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:52:35.389793+0000","last_undegraded":"2024-09-13T14:52:35.389793+0000","last_fullsized":"2024-09-13T14:52:35.389793+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":345,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390221+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:52:35.390221+0000","last_peered":"2024-09-13T14:52:35.390221+0000","last_clean":"2024-09-13T14:52:35.390221+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:52:35.390221+0000","last_undegraded":"2024-09-13T14:52:35.390221+0000","last_fullsized":"2024-09-13T14:52:35.390221+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":173,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.377206+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:52:35.377206+0000","last_peered":"2024-09-13T14:52:35.377206+0000","last_clean":"2024-09-13T14:52:35.377206+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:52:35.377206+0000","last_undegraded":"2024-09-13T14:52:35.377206+0000","last_fullsized":"2024-09-13T14:52:35.377206+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":73,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.418278+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:52:49.418278+0000","last_peered":"2024-09-13T14:52:49.418278+0000","last_clean":"2024-09-13T14:52:49.418278+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:52:49.418278+0000","last_undegraded":"2024-09-13T14:52:49.418278+0000","last_fullsized":"2024-09-13T14:52:49.418278+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":149,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.353196+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:52:32.353196+0000","last_peered":"2024-09-13T14:52:32.353196+0000","last_clean":"2024-09-13T14:52:32.353196+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:52:32.353196+0000","last_undegraded":"2024-09-13T14:52:32.353196+0000","last_fullsized":"2024-09-13T14:52:32.353196+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"237'7588","reported_seq":8420,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.411550+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:52:49.411550+0000","last_peered":"2024-09-13T14:52:49.411550+0000","last_clean":"2024-09-13T14:52:49.411550+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:52:49.411550+0000","last_undegraded":"2024-09-13T14:52:49.411550+0000","last_fullsized":"2024-09-13T14:52:49.411550+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7588,"log_dups_size":0,"ondisk_log_size":7588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216793088,"num_objects":3308,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7588,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":280,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.382984+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:52:35.382984+0000","last_peered":"2024-09-13T14:52:35.382984+0000","last_clean":"2024-09-13T14:52:35.382984+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:52:35.382984+0000","last_undegraded":"2024-09-13T14:52:35.382984+0000","last_fullsized":"2024-09-13T14:52:35.382984+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":101,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:44.593867+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:52:44.593867+0000","last_peered":"2024-09-13T14:52:44.593867+0000","last_clean":"2024-09-13T14:52:44.593867+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:52:44.593867+0000","last_undegraded":"2024-09-13T14:52:44.593867+0000","last_fullsized":"2024-09-13T14:52:44.593867+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"237'7352","reported_seq":8957,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.417177+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:52:49.417177+0000","last_peered":"2024-09-13T14:52:49.417177+0000","last_clean":"2024-09-13T14:52:49.417177+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:52:49.417177+0000","last_undegraded":"2024-09-13T14:52:49.417177+0000","last_fullsized":"2024-09-13T14:52:49.417177+0000","mapping_epoch":170,"log_start":"168'900","ondisk_log_start":"168'900","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211419136,"num_objects":3226,"num_object_clones":0,"num_object_copies":9678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7352,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":101,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:34.169513+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:52:34.169513+0000","last_peered":"2024-09-13T14:52:34.169513+0000","last_clean":"2024-09-13T14:52:34.169513+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:52:34.169513+0000","last_undegraded":"2024-09-13T14:52:34.169513+0000","last_fullsized":"2024-09-13T14:52:34.169513+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":253,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387562+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:52:32.387562+0000","last_peered":"2024-09-13T14:52:32.387562+0000","last_clean":"2024-09-13T14:52:32.387562+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:52:32.387562+0000","last_undegraded":"2024-09-13T14:52:32.387562+0000","last_fullsized":"2024-09-13T14:52:32.387562+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"237'7516","reported_seq":9212,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.042910+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:52:49.042910+0000","last_peered":"2024-09-13T14:52:49.042910+0000","last_clean":"2024-09-13T14:52:49.042910+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:52:49.042910+0000","last_undegraded":"2024-09-13T14:52:49.042910+0000","last_fullsized":"2024-09-13T14:52:49.042910+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7516,"log_dups_size":0,"ondisk_log_size":7516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7516,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":77,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.383130+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:52:35.383130+0000","last_peered":"2024-09-13T14:52:35.383130+0000","last_clean":"2024-09-13T14:52:35.383130+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:52:35.383130+0000","last_undegraded":"2024-09-13T14:52:35.383130+0000","last_fullsized":"2024-09-13T14:52:35.383130+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":33,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.395047+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:52:35.395047+0000","last_peered":"2024-09-13T14:52:35.395047+0000","last_clean":"2024-09-13T14:52:35.395047+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:52:35.395047+0000","last_undegraded":"2024-09-13T14:52:35.395047+0000","last_fullsized":"2024-09-13T14:52:35.395047+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"237'7320","reported_seq":13017,"reported_epoch":237,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:52:48.004839+0000","last_change":"2024-09-13T14:52:31.266342+0000","last_active":"2024-09-13T14:52:48.004839+0000","last_peered":"2024-09-13T14:52:48.004839+0000","last_clean":"2024-09-13T14:48:36.212253+0000","last_became_active":"2024-09-13T14:50:07.833455+0000","last_became_peered":"2024-09-13T14:50:07.833455+0000","last_unstale":"2024-09-13T14:52:48.004839+0000","last_undegraded":"2024-09-13T14:52:48.004839+0000","last_fullsized":"2024-09-13T14:50:07.547080+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7320,"log_dups_size":0,"ondisk_log_size":7320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208928768,"num_objects":3188,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":131,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7320,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2755,"num_bytes_recovered":180551680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":3057},{"shards":"0,3,6","objects":131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":89,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.383059+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:52:35.383059+0000","last_peered":"2024-09-13T14:52:35.383059+0000","last_clean":"2024-09-13T14:52:35.383059+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:52:35.383059+0000","last_undegraded":"2024-09-13T14:52:35.383059+0000","last_fullsized":"2024-09-13T14:52:35.383059+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":324,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390137+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:52:35.390137+0000","last_peered":"2024-09-13T14:52:35.390137+0000","last_clean":"2024-09-13T14:52:35.390137+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:52:35.390137+0000","last_undegraded":"2024-09-13T14:52:35.390137+0000","last_fullsized":"2024-09-13T14:52:35.390137+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"237'7436","reported_seq":24685,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.154109+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:52:49.154109+0000","last_peered":"2024-09-13T14:52:49.154109+0000","last_clean":"2024-09-13T14:52:49.154109+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:52:49.154109+0000","last_undegraded":"2024-09-13T14:52:49.154109+0000","last_fullsized":"2024-09-13T14:52:49.154109+0000","mapping_epoch":232,"log_start":"168'900","ondisk_log_start":"168'900","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217448448,"num_objects":3318,"num_object_clones":0,"num_object_copies":9954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7436,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":335,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.417436+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:52:49.417436+0000","last_peered":"2024-09-13T14:52:49.417436+0000","last_clean":"2024-09-13T14:52:49.417436+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:52:49.417436+0000","last_undegraded":"2024-09-13T14:52:49.417436+0000","last_fullsized":"2024-09-13T14:52:49.417436+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":219,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387828+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:52:32.387828+0000","last_peered":"2024-09-13T14:52:32.387828+0000","last_clean":"2024-09-13T14:52:32.387828+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:52:32.387828+0000","last_undegraded":"2024-09-13T14:52:32.387828+0000","last_fullsized":"2024-09-13T14:52:32.387828+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"237'7407","reported_seq":7788,"reported_epoch":237,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:49.073495+0000","last_change":"2024-09-13T14:50:20.844850+0000","last_active":"2024-09-13T14:52:49.073495+0000","last_peered":"2024-09-13T14:52:49.073495+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:52:49.073495+0000","last_undegraded":"2024-09-13T14:50:20.836232+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"167'800","ondisk_log_start":"167'800","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211877888,"num_objects":3233,"num_object_clones":0,"num_object_copies":9699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4"],"object_location_counts":[{"shards":"3,4","objects":1053},{"shards":"3,4,6","objects":2169},{"shards":"4,5,6","objects":11}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":120,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.353324+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:52:32.353324+0000","last_peered":"2024-09-13T14:52:32.353324+0000","last_clean":"2024-09-13T14:52:32.353324+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:52:32.353324+0000","last_undegraded":"2024-09-13T14:52:32.353324+0000","last_fullsized":"2024-09-13T14:52:32.353324+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":216,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.388095+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:52:32.388095+0000","last_peered":"2024-09-13T14:52:32.388095+0000","last_clean":"2024-09-13T14:52:32.388095+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:52:32.388095+0000","last_undegraded":"2024-09-13T14:52:32.388095+0000","last_fullsized":"2024-09-13T14:52:32.388095+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"238'7537","reported_seq":11841,"reported_epoch":238,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-13T14:52:49.663426+0000","last_change":"2024-09-13T14:50:50.437830+0000","last_active":"2024-09-13T14:52:49.663426+0000","last_peered":"2024-09-13T14:52:49.663426+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:52:49.663426+0000","last_undegraded":"2024-09-13T14:50:42.477738+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"169'1100","ondisk_log_start":"169'1100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215285760,"num_objects":3285,"num_object_clones":0,"num_object_copies":9855,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":2570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7537,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2197,"num_bytes_recovered":143917056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"1,3,6","objects":169},{"shards":"1,4,5","objects":9},{"shards":"3,5","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":212,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387592+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:52:32.387592+0000","last_peered":"2024-09-13T14:52:32.387592+0000","last_clean":"2024-09-13T14:52:32.387592+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:52:32.387592+0000","last_undegraded":"2024-09-13T14:52:32.387592+0000","last_fullsized":"2024-09-13T14:52:32.387592+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":246,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.388245+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:52:32.388245+0000","last_peered":"2024-09-13T14:52:32.388245+0000","last_clean":"2024-09-13T14:52:32.388245+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:52:32.388245+0000","last_undegraded":"2024-09-13T14:52:32.388245+0000","last_fullsized":"2024-09-13T14:52:32.388245+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"237'7275","reported_seq":11700,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.390075+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:52:49.390075+0000","last_peered":"2024-09-13T14:52:49.390075+0000","last_clean":"2024-09-13T14:52:49.390075+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:52:49.390075+0000","last_undegraded":"2024-09-13T14:52:49.390075+0000","last_fullsized":"2024-09-13T14:52:49.390075+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7275,"log_dups_size":0,"ondisk_log_size":7275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211681304,"num_objects":3231,"num_object_clones":0,"num_object_copies":9693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7275,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":73,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.417714+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:52:49.417714+0000","last_peered":"2024-09-13T14:52:49.417714+0000","last_clean":"2024-09-13T14:52:49.417714+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:52:49.417714+0000","last_undegraded":"2024-09-13T14:52:49.417714+0000","last_fullsized":"2024-09-13T14:52:49.417714+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":180,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.383094+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:52:35.383094+0000","last_peered":"2024-09-13T14:52:35.383094+0000","last_clean":"2024-09-13T14:52:35.383094+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:52:35.383094+0000","last_undegraded":"2024-09-13T14:52:35.383094+0000","last_fullsized":"2024-09-13T14:52:35.383094+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"237'7407","reported_seq":8654,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.402243+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:52:49.402243+0000","last_peered":"2024-09-13T14:52:49.402243+0000","last_clean":"2024-09-13T14:52:49.402243+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:52:49.402243+0000","last_undegraded":"2024-09-13T14:52:49.402243+0000","last_fullsized":"2024-09-13T14:52:49.402243+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218693632,"num_objects":3337,"num_object_clones":0,"num_object_copies":10011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":527,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.389698+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:52:35.389698+0000","last_peered":"2024-09-13T14:52:35.389698+0000","last_clean":"2024-09-13T14:52:35.389698+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:52:35.389698+0000","last_undegraded":"2024-09-13T14:52:35.389698+0000","last_fullsized":"2024-09-13T14:52:35.389698+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":804,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.382833+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:52:35.382833+0000","last_peered":"2024-09-13T14:52:35.382833+0000","last_clean":"2024-09-13T14:52:35.382833+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:52:35.382833+0000","last_undegraded":"2024-09-13T14:52:35.382833+0000","last_fullsized":"2024-09-13T14:52:35.382833+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":293,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.418294+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:52:49.418294+0000","last_peered":"2024-09-13T14:52:49.418294+0000","last_clean":"2024-09-13T14:52:49.418294+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:52:49.418294+0000","last_undegraded":"2024-09-13T14:52:49.418294+0000","last_fullsized":"2024-09-13T14:52:49.418294+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":33,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.395404+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:52:35.395404+0000","last_peered":"2024-09-13T14:52:35.395404+0000","last_clean":"2024-09-13T14:52:35.395404+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:52:35.395404+0000","last_undegraded":"2024-09-13T14:52:35.395404+0000","last_fullsized":"2024-09-13T14:52:35.395404+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"238'7566","reported_seq":12286,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.690608+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:52:49.690608+0000","last_peered":"2024-09-13T14:52:49.690608+0000","last_clean":"2024-09-13T14:52:49.690608+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:52:49.690608+0000","last_undegraded":"2024-09-13T14:52:49.690608+0000","last_fullsized":"2024-09-13T14:52:49.690608+0000","mapping_epoch":223,"log_start":"171'1718","ondisk_log_start":"171'1718","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5848,"log_dups_size":0,"ondisk_log_size":5848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215089152,"num_objects":3282,"num_object_clones":0,"num_object_copies":9846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":374,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.421433+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:52:49.421433+0000","last_peered":"2024-09-13T14:52:49.421433+0000","last_clean":"2024-09-13T14:52:49.421433+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:52:49.421433+0000","last_undegraded":"2024-09-13T14:52:49.421433+0000","last_fullsized":"2024-09-13T14:52:49.421433+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":339,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390633+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:52:35.390633+0000","last_peered":"2024-09-13T14:52:35.390633+0000","last_clean":"2024-09-13T14:52:35.390633+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:52:35.390633+0000","last_undegraded":"2024-09-13T14:52:35.390633+0000","last_fullsized":"2024-09-13T14:52:35.390633+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":349,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390393+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:52:35.390393+0000","last_peered":"2024-09-13T14:52:35.390393+0000","last_clean":"2024-09-13T14:52:35.390393+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:52:35.390393+0000","last_undegraded":"2024-09-13T14:52:35.390393+0000","last_fullsized":"2024-09-13T14:52:35.390393+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"237'7415","reported_seq":17728,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:48.793969+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:52:48.793969+0000","last_peered":"2024-09-13T14:52:48.793969+0000","last_clean":"2024-09-13T14:52:48.793969+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:52:48.793969+0000","last_undegraded":"2024-09-13T14:52:48.793969+0000","last_fullsized":"2024-09-13T14:52:48.793969+0000","mapping_epoch":235,"log_start":"164'700","ondisk_log_start":"164'700","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6715,"log_dups_size":0,"ondisk_log_size":6715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208470016,"num_objects":3181,"num_object_clones":0,"num_object_copies":9543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7415,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":875,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.384119+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:52:35.384119+0000","last_peered":"2024-09-13T14:52:35.384119+0000","last_clean":"2024-09-13T14:52:35.384119+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:52:35.384119+0000","last_undegraded":"2024-09-13T14:52:35.384119+0000","last_fullsized":"2024-09-13T14:52:35.384119+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":332,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.421736+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:52:49.421736+0000","last_peered":"2024-09-13T14:52:49.421736+0000","last_clean":"2024-09-13T14:52:49.421736+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:52:49.421736+0000","last_undegraded":"2024-09-13T14:52:49.421736+0000","last_fullsized":"2024-09-13T14:52:49.421736+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":85,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.423469+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:52:49.423469+0000","last_peered":"2024-09-13T14:52:49.423469+0000","last_clean":"2024-09-13T14:52:49.423469+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:52:49.423469+0000","last_undegraded":"2024-09-13T14:52:49.423469+0000","last_fullsized":"2024-09-13T14:52:49.423469+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"238'7486","reported_seq":16187,"reported_epoch":238,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:52:49.712143+0000","last_change":"2024-09-13T14:52:31.275276+0000","last_active":"2024-09-13T14:52:49.712143+0000","last_peered":"2024-09-13T14:52:49.712143+0000","last_clean":"2024-09-13T14:48:36.214823+0000","last_became_active":"2024-09-13T14:50:41.217309+0000","last_became_peered":"2024-09-13T14:50:41.217309+0000","last_unstale":"2024-09-13T14:52:49.712143+0000","last_undegraded":"2024-09-13T14:52:49.712143+0000","last_fullsized":"2024-09-13T14:50:38.959939+0000","mapping_epoch":216,"log_start":"169'1100","ondisk_log_start":"169'1100","created":160,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6386,"log_dups_size":0,"ondisk_log_size":6386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211943424,"num_objects":3234,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":196,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7486,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4267,"num_bytes_recovered":274399232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4","objects":3038},{"shards":"3,4,5","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":92,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.382951+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:52:35.382951+0000","last_peered":"2024-09-13T14:52:35.382951+0000","last_clean":"2024-09-13T14:52:35.382951+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:52:35.382951+0000","last_undegraded":"2024-09-13T14:52:35.382951+0000","last_fullsized":"2024-09-13T14:52:35.382951+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":221,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387829+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:52:32.387829+0000","last_peered":"2024-09-13T14:52:32.387829+0000","last_clean":"2024-09-13T14:52:32.387829+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:52:32.387829+0000","last_undegraded":"2024-09-13T14:52:32.387829+0000","last_fullsized":"2024-09-13T14:52:32.387829+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":334,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.422609+0000","last_change":"2024-09-13T14:50:22.038026+0000","last_active":"2024-09-13T14:52:49.422609+0000","last_peered":"2024-09-13T14:52:49.422609+0000","last_clean":"2024-09-13T14:52:49.422609+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:52:49.422609+0000","last_undegraded":"2024-09-13T14:52:49.422609+0000","last_fullsized":"2024-09-13T14:52:49.422609+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:13.659469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"237'7513","reported_seq":12094,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:47.982385+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:52:47.982385+0000","last_peered":"2024-09-13T14:52:47.982385+0000","last_clean":"2024-09-13T14:52:47.982385+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:52:47.982385+0000","last_undegraded":"2024-09-13T14:52:47.982385+0000","last_fullsized":"2024-09-13T14:52:47.982385+0000","mapping_epoch":226,"log_start":"164'100","ondisk_log_start":"164'100","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216334336,"num_objects":3301,"num_object_clones":0,"num_object_copies":9903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7513,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":92,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.382952+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:52:35.382952+0000","last_peered":"2024-09-13T14:52:35.382952+0000","last_clean":"2024-09-13T14:52:35.382952+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:52:35.382952+0000","last_undegraded":"2024-09-13T14:52:35.382952+0000","last_fullsized":"2024-09-13T14:52:35.382952+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":886,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.422783+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:52:49.422783+0000","last_peered":"2024-09-13T14:52:49.422783+0000","last_clean":"2024-09-13T14:52:49.422783+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:52:49.422783+0000","last_undegraded":"2024-09-13T14:52:49.422783+0000","last_fullsized":"2024-09-13T14:52:49.422783+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":311,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.422306+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:52:49.422306+0000","last_peered":"2024-09-13T14:52:49.422306+0000","last_clean":"2024-09-13T14:52:49.422306+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:52:49.422306+0000","last_undegraded":"2024-09-13T14:52:49.422306+0000","last_fullsized":"2024-09-13T14:52:49.422306+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"238'7604","reported_seq":12315,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.552870+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:52:49.552870+0000","last_peered":"2024-09-13T14:52:49.552870+0000","last_clean":"2024-09-13T14:52:49.552870+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:52:49.552870+0000","last_undegraded":"2024-09-13T14:52:49.552870+0000","last_fullsized":"2024-09-13T14:52:49.552870+0000","mapping_epoch":218,"log_start":"171'1745","ondisk_log_start":"171'1745","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216530944,"num_objects":3304,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":338,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.422696+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:52:49.422696+0000","last_peered":"2024-09-13T14:52:49.422696+0000","last_clean":"2024-09-13T14:52:49.422696+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:52:49.422696+0000","last_undegraded":"2024-09-13T14:52:49.422696+0000","last_fullsized":"2024-09-13T14:52:49.422696+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":558,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390090+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:52:35.390090+0000","last_peered":"2024-09-13T14:52:35.390090+0000","last_clean":"2024-09-13T14:52:35.390090+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:52:35.390090+0000","last_undegraded":"2024-09-13T14:52:35.390090+0000","last_fullsized":"2024-09-13T14:52:35.390090+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":335,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390060+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:52:35.390060+0000","last_peered":"2024-09-13T14:52:35.390060+0000","last_clean":"2024-09-13T14:52:35.390060+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:52:35.390060+0000","last_undegraded":"2024-09-13T14:52:35.390060+0000","last_fullsized":"2024-09-13T14:52:35.390060+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"237'7332","reported_seq":9573,"reported_epoch":237,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:52:49.169965+0000","last_change":"2024-09-13T14:52:48.998321+0000","last_active":"2024-09-13T14:52:49.169965+0000","last_peered":"2024-09-13T14:52:49.169965+0000","last_clean":"2024-09-13T14:48:36.211439+0000","last_became_active":"2024-09-13T14:50:19.876599+0000","last_became_peered":"2024-09-13T14:50:19.876599+0000","last_unstale":"2024-09-13T14:52:49.169965+0000","last_undegraded":"2024-09-13T14:52:49.169965+0000","last_fullsized":"2024-09-13T14:50:19.782972+0000","mapping_epoch":207,"log_start":"167'800","ondisk_log_start":"167'800","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:16.080703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208273408,"num_objects":3178,"num_object_clones":0,"num_object_copies":9534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2072,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7332,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":69795840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":1106},{"shards":"2,4,5","objects":1},{"shards":"2,4,6","objects":2071}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":357,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.419163+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:52:49.419163+0000","last_peered":"2024-09-13T14:52:49.419163+0000","last_clean":"2024-09-13T14:52:49.419163+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:52:49.419163+0000","last_undegraded":"2024-09-13T14:52:49.419163+0000","last_fullsized":"2024-09-13T14:52:49.419163+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":788,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.423767+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:52:49.423767+0000","last_peered":"2024-09-13T14:52:49.423767+0000","last_clean":"2024-09-13T14:52:49.423767+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:52:49.423767+0000","last_undegraded":"2024-09-13T14:52:49.423767+0000","last_fullsized":"2024-09-13T14:52:49.423767+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":73,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.415320+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:52:49.415320+0000","last_peered":"2024-09-13T14:52:49.415320+0000","last_clean":"2024-09-13T14:52:49.415320+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:52:49.415320+0000","last_undegraded":"2024-09-13T14:52:49.415320+0000","last_fullsized":"2024-09-13T14:52:49.415320+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"237'7498","reported_seq":9069,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:48.701525+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:52:48.701525+0000","last_peered":"2024-09-13T14:52:48.701525+0000","last_clean":"2024-09-13T14:52:48.701525+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:52:48.701525+0000","last_undegraded":"2024-09-13T14:52:48.701525+0000","last_fullsized":"2024-09-13T14:52:48.701525+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220332032,"num_objects":3362,"num_object_clones":0,"num_object_copies":10086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":329,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390195+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:52:35.390195+0000","last_peered":"2024-09-13T14:52:35.390195+0000","last_clean":"2024-09-13T14:52:35.390195+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:52:35.390195+0000","last_undegraded":"2024-09-13T14:52:35.390195+0000","last_fullsized":"2024-09-13T14:52:35.390195+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":789,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.395557+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:52:35.395557+0000","last_peered":"2024-09-13T14:52:35.395557+0000","last_clean":"2024-09-13T14:52:35.395557+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:52:35.395557+0000","last_undegraded":"2024-09-13T14:52:35.395557+0000","last_fullsized":"2024-09-13T14:52:35.395557+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":223,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387479+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:52:32.387479+0000","last_peered":"2024-09-13T14:52:32.387479+0000","last_clean":"2024-09-13T14:52:32.387479+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:52:32.387479+0000","last_undegraded":"2024-09-13T14:52:32.387479+0000","last_fullsized":"2024-09-13T14:52:32.387479+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":219,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387788+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:52:32.387788+0000","last_peered":"2024-09-13T14:52:32.387788+0000","last_clean":"2024-09-13T14:52:32.387788+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:52:32.387788+0000","last_undegraded":"2024-09-13T14:52:32.387788+0000","last_fullsized":"2024-09-13T14:52:32.387788+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":267,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.388033+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:52:32.388033+0000","last_peered":"2024-09-13T14:52:32.388033+0000","last_clean":"2024-09-13T14:52:32.388033+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:52:32.388033+0000","last_undegraded":"2024-09-13T14:52:32.388033+0000","last_fullsized":"2024-09-13T14:52:32.388033+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":336,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.422206+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:52:49.422206+0000","last_peered":"2024-09-13T14:52:49.422206+0000","last_clean":"2024-09-13T14:52:49.422206+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:52:49.422206+0000","last_undegraded":"2024-09-13T14:52:49.422206+0000","last_fullsized":"2024-09-13T14:52:49.422206+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":176,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.417769+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:52:49.417769+0000","last_peered":"2024-09-13T14:52:49.417769+0000","last_clean":"2024-09-13T14:52:49.417769+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:52:49.417769+0000","last_undegraded":"2024-09-13T14:52:49.417769+0000","last_fullsized":"2024-09-13T14:52:49.417769+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":100,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.383083+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:52:35.383083+0000","last_peered":"2024-09-13T14:52:35.383083+0000","last_clean":"2024-09-13T14:52:35.383083+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:52:35.383083+0000","last_undegraded":"2024-09-13T14:52:35.383083+0000","last_fullsized":"2024-09-13T14:52:35.383083+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":313,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.417882+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:52:49.417882+0000","last_peered":"2024-09-13T14:52:49.417882+0000","last_clean":"2024-09-13T14:52:49.417882+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:52:49.417882+0000","last_undegraded":"2024-09-13T14:52:49.417882+0000","last_fullsized":"2024-09-13T14:52:49.417882+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":366,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390426+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:52:35.390426+0000","last_peered":"2024-09-13T14:52:35.390426+0000","last_clean":"2024-09-13T14:52:35.390426+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:52:35.390426+0000","last_undegraded":"2024-09-13T14:52:35.390426+0000","last_fullsized":"2024-09-13T14:52:35.390426+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":364,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.417942+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:52:49.417942+0000","last_peered":"2024-09-13T14:52:49.417942+0000","last_clean":"2024-09-13T14:52:49.417942+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:52:49.417942+0000","last_undegraded":"2024-09-13T14:52:49.417942+0000","last_fullsized":"2024-09-13T14:52:49.417942+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":129,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.353523+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:52:32.353523+0000","last_peered":"2024-09-13T14:52:32.353523+0000","last_clean":"2024-09-13T14:52:32.353523+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:52:32.353523+0000","last_undegraded":"2024-09-13T14:52:32.353523+0000","last_fullsized":"2024-09-13T14:52:32.353523+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":331,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.418090+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:52:49.418090+0000","last_peered":"2024-09-13T14:52:49.418090+0000","last_clean":"2024-09-13T14:52:49.418090+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:52:49.418090+0000","last_undegraded":"2024-09-13T14:52:49.418090+0000","last_fullsized":"2024-09-13T14:52:49.418090+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":346,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390315+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:52:35.390315+0000","last_peered":"2024-09-13T14:52:35.390315+0000","last_clean":"2024-09-13T14:52:35.390315+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:52:35.390315+0000","last_undegraded":"2024-09-13T14:52:35.390315+0000","last_fullsized":"2024-09-13T14:52:35.390315+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":317,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.422016+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:52:49.422016+0000","last_peered":"2024-09-13T14:52:49.422016+0000","last_clean":"2024-09-13T14:52:49.422016+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:52:49.422016+0000","last_undegraded":"2024-09-13T14:52:49.422016+0000","last_fullsized":"2024-09-13T14:52:49.422016+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":180,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.417645+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:52:49.417645+0000","last_peered":"2024-09-13T14:52:49.417645+0000","last_clean":"2024-09-13T14:52:49.417645+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:52:49.417645+0000","last_undegraded":"2024-09-13T14:52:49.417645+0000","last_fullsized":"2024-09-13T14:52:49.417645+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":217,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387785+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:52:32.387785+0000","last_peered":"2024-09-13T14:52:32.387785+0000","last_clean":"2024-09-13T14:52:32.387785+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:52:32.387785+0000","last_undegraded":"2024-09-13T14:52:32.387785+0000","last_fullsized":"2024-09-13T14:52:32.387785+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":74,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.382828+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:52:35.382828+0000","last_peered":"2024-09-13T14:52:35.382828+0000","last_clean":"2024-09-13T14:52:35.382828+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:52:35.382828+0000","last_undegraded":"2024-09-13T14:52:35.382828+0000","last_fullsized":"2024-09-13T14:52:35.382828+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":339,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.389971+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:52:35.389971+0000","last_peered":"2024-09-13T14:52:35.389971+0000","last_clean":"2024-09-13T14:52:35.389971+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:52:35.389971+0000","last_undegraded":"2024-09-13T14:52:35.389971+0000","last_fullsized":"2024-09-13T14:52:35.389971+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":218,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387748+0000","last_change":"2024-09-13T14:43:09.682351+0000","last_active":"2024-09-13T14:52:32.387748+0000","last_peered":"2024-09-13T14:52:32.387748+0000","last_clean":"2024-09-13T14:52:32.387748+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:52:32.387748+0000","last_undegraded":"2024-09-13T14:52:32.387748+0000","last_fullsized":"2024-09-13T14:52:32.387748+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:54.982149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":75,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.382976+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:52:35.382976+0000","last_peered":"2024-09-13T14:52:35.382976+0000","last_clean":"2024-09-13T14:52:35.382976+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:52:35.382976+0000","last_undegraded":"2024-09-13T14:52:35.382976+0000","last_fullsized":"2024-09-13T14:52:35.382976+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":215,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387705+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:52:32.387705+0000","last_peered":"2024-09-13T14:52:32.387705+0000","last_clean":"2024-09-13T14:52:32.387705+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:52:32.387705+0000","last_undegraded":"2024-09-13T14:52:32.387705+0000","last_fullsized":"2024-09-13T14:52:32.387705+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":77,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.383066+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:52:35.383066+0000","last_peered":"2024-09-13T14:52:35.383066+0000","last_clean":"2024-09-13T14:52:35.383066+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:52:35.383066+0000","last_undegraded":"2024-09-13T14:52:35.383066+0000","last_fullsized":"2024-09-13T14:52:35.383066+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":334,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.418188+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:52:49.418188+0000","last_peered":"2024-09-13T14:52:49.418188+0000","last_clean":"2024-09-13T14:52:49.418188+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:52:49.418188+0000","last_undegraded":"2024-09-13T14:52:49.418188+0000","last_fullsized":"2024-09-13T14:52:49.418188+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":351,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-09-13T14:52:49.419384+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:52:49.419384+0000","last_peered":"2024-09-13T14:52:49.419384+0000","last_clean":"2024-09-13T14:52:49.419384+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:52:49.419384+0000","last_undegraded":"2024-09-13T14:52:49.419384+0000","last_fullsized":"2024-09-13T14:52:49.419384+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":246,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-13T14:52:32.387499+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:52:32.387499+0000","last_peered":"2024-09-13T14:52:32.387499+0000","last_clean":"2024-09-13T14:52:32.387499+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:52:32.387499+0000","last_undegraded":"2024-09-13T14:52:32.387499+0000","last_fullsized":"2024-09-13T14:52:32.387499+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":402,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T14:52:35.390324+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:52:35.390324+0000","last_peered":"2024-09-13T14:52:35.390324+0000","last_clean":"2024-09-13T14:52:35.390324+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:52:35.390324+0000","last_undegraded":"2024-09-13T14:52:35.390324+0000","last_fullsized":"2024-09-13T14:52:35.390324+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3427205144,"num_objects":52296,"num_object_clones":0,"num_object_copies":156888,"num_objects_missing_on_primary":87,"num_objects_missing":87,"num_objects_degraded":4913,"num_objects_misplaced":2399,"num_objects_unfound":0,"num_objects_dirty":52296,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119252,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39568,"num_bytes_recovered":2495545344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11079266304,"data_stored":11079254088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109224,"ondisk_log_size":109224,"up":48,"acting":43,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495941,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2198792,"kb_used_data":1955200,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91546360,"statfs":{"total":95995035648,"available":93743472640,"internally_reserved":0,"allocated":2002124800,"data_stored":2000823684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":249428423},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757576,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1513216,"kb_used_data":1333624,"kb_used_omap":1,"kb_used_meta":179582,"kb_avail":92231936,"statfs":{"total":95995035648,"available":94445502464,"internally_reserved":0,"allocated":1365630976,"data_stored":1364333655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":183892423},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019216,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1339656,"kb_used_data":1051776,"kb_used_omap":1,"kb_used_meta":287870,"kb_avail":92405496,"statfs":{"total":95995035648,"available":94623227904,"internally_reserved":0,"allocated":1077018624,"data_stored":1075717569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":294779334},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468101,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2399036,"kb_used_data":2178548,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":91346116,"statfs":{"total":95995035648,"available":93538422784,"internally_reserved":0,"allocated":2230833152,"data_stored":2229539677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":225701028},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,2,2,5,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631630,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2109260,"kb_used_data":1752644,"kb_used_omap":8,"kb_used_meta":356535,"kb_avail":91635892,"statfs":{"total":95995035648,"available":93835153408,"internally_reserved":0,"allocated":1794707456,"data_stored":1793402831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":365092831},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795153,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1791108,"kb_used_data":1534524,"kb_used_omap":7,"kb_used_meta":256568,"kb_avail":91954044,"statfs":{"total":95995035648,"available":94160941056,"internally_reserved":0,"allocated":1571352576,"data_stored":1570051358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":262726258},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958683,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266760,"kb_used_data":986560,"kb_used_omap":7,"kb_used_meta":280184,"kb_avail":92478392,"statfs":{"total":95995035648,"available":94697873408,"internally_reserved":0,"allocated":1010237440,"data_stored":1008932779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":286908903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1000214528,"data_stored":1000210456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1560612864,"data_stored":1560608792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1785790464,"data_stored":1785790464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2222915584,"data_stored":2222915584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1070858240,"data_stored":1070858240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1361842176,"data_stored":1361838104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1998520320,"data_stored":1998520320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":78512128,"data_stored":78512128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:52:56.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:52:56.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:56.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:56.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[102314]: pgmap v716: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 320 op/s; 2343/153444 objects degraded (1.527%); 3617/153444 objects misplaced (2.357%); 23 MiB/s, 394 objects/s recovering 2024-09-13T14:52:56.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:56.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:56.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[102314]: osdmap e242: 8 total, 8 up, 7 in 2024-09-13T14:52:56.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[107038]: pgmap v716: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 320 op/s; 2343/153444 objects degraded (1.527%); 3617/153444 objects misplaced (2.357%); 23 MiB/s, 394 objects/s recovering 2024-09-13T14:52:56.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:56.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:56.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:56 smithi154 ceph-mon[107038]: osdmap e242: 8 total, 8 up, 7 in 2024-09-13T14:52:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:56 smithi172 ceph-mon[100696]: pgmap v716: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 320 op/s; 2343/153444 objects degraded (1.527%); 3617/153444 objects misplaced (2.357%); 23 MiB/s, 394 objects/s recovering 2024-09-13T14:52:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:56 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:52:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:56 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:52:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:56 smithi172 ceph-mon[100696]: osdmap e242: 8 total, 8 up, 7 in 2024-09-13T14:52:56.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:56.769 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:56.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:56.894 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:56.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:56.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:57.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:57.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:57.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:57.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.271 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:57.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:57.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:57.396 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:57.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:57.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:57.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:57.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:57.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:52:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:57.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:52:57.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:57.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:57.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:58.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.024 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:58.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:58.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:58.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:58.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:52:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.282 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:58.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:52:58.383 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:58.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:58.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.539 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:58.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:58.707 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:58 smithi154 ceph-mon[102314]: pgmap v718: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 2343/153444 objects degraded (1.527%); 3617/153444 objects misplaced (2.357%); 12 MiB/s, 212 objects/s recovering 2024-09-13T14:52:58.707 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:52:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:58.707 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:58 smithi154 ceph-mon[107038]: pgmap v718: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 2343/153444 objects degraded (1.527%); 3617/153444 objects misplaced (2.357%); 12 MiB/s, 212 objects/s recovering 2024-09-13T14:52:58.707 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:52:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:58 smithi172 ceph-mon[100696]: pgmap v718: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 195 op/s; 2343/153444 objects degraded (1.527%); 3617/153444 objects misplaced (2.357%); 12 MiB/s, 212 objects/s recovering 2024-09-13T14:52:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:52:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:52:58.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:52:58.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:52:58.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:58.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:58.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:59.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:59.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:59.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:52:59.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:52:59.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:59.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:59.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:52:59.444 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:52:59.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:52:59.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:52:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:52:59.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:59.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:52:59.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:52:59.828 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:59.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:52:59.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:52:59.954 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:52:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:52:59.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:00.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:00.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:00.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:00.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:00.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:00.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:00.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:00.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:00.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:00.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:00.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:00.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:00.710 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:00 smithi154 ceph-mon[102314]: pgmap v719: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 207 op/s; 2343/152256 objects degraded (1.539%); 3617/152256 objects misplaced (2.376%); 9.1 MiB/s, 167 objects/s recovering 2024-09-13T14:53:00.711 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2343/152256 objects degraded (1.539%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:00.711 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:00 smithi154 ceph-mon[107038]: pgmap v719: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 207 op/s; 2343/152256 objects degraded (1.539%); 3617/152256 objects misplaced (2.376%); 9.1 MiB/s, 167 objects/s recovering 2024-09-13T14:53:00.711 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2343/152256 objects degraded (1.539%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:00.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:00 smithi172 ceph-mon[100696]: pgmap v719: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 207 op/s; 2343/152256 objects degraded (1.539%); 3617/152256 objects misplaced (2.376%); 9.1 MiB/s, 167 objects/s recovering 2024-09-13T14:53:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2343/152256 objects degraded (1.539%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:00.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:00.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:00.905 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:00.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:00.997 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:00.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:01.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:01.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:01.182 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:01.183 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:01.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:01.271 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:01.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:01.402 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:01.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:53:01.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:01.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:01.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:01.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:01.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:01.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:01.902 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:01.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:53:01.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:01.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:01.958 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:53:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:53:01] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:53:02.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:02.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:02.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:02.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:02.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:02.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:02.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:02.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:53:02.757 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:02 smithi154 ceph-mon[102314]: pgmap v720: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 328 op/s; 2343/148518 objects degraded (1.578%); 2773/148518 objects misplaced (1.867%); 14 MiB/s, 246 objects/s recovering 2024-09-13T14:53:02.757 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:02 smithi154 ceph-mon[107038]: pgmap v720: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 328 op/s; 2343/148518 objects degraded (1.578%); 2773/148518 objects misplaced (1.867%); 14 MiB/s, 246 objects/s recovering 2024-09-13T14:53:02.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:02 smithi172 ceph-mon[100696]: pgmap v720: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 328 op/s; 2343/148518 objects degraded (1.578%); 2773/148518 objects misplaced (1.867%); 14 MiB/s, 246 objects/s recovering 2024-09-13T14:53:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:02.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:02.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:02.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:02.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:03.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:03.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:03.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:03.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:03.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:03.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:03.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:03.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:03.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:03.568 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:03.659 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:03.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:03.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:03.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:03.843 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:03.945 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:03.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:03.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:04.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:04.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:04.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:04.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.520 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:04.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:04.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:04.647 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:04 smithi154 ceph-mon[102314]: pgmap v721: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 281 op/s; 2343/148518 objects degraded (1.578%); 2773/148518 objects misplaced (1.867%); 12 MiB/s, 211 objects/s recovering 2024-09-13T14:53:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:04 smithi154 ceph-mon[107038]: pgmap v721: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 281 op/s; 2343/148518 objects degraded (1.578%); 2773/148518 objects misplaced (1.867%); 12 MiB/s, 211 objects/s recovering 2024-09-13T14:53:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:04 smithi172 ceph-mon[100696]: pgmap v721: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 86 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 281 op/s; 2343/148518 objects degraded (1.578%); 2773/148518 objects misplaced (1.867%); 12 MiB/s, 211 objects/s recovering 2024-09-13T14:53:04.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:04.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:04.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:04.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:04.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:05.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:05.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:05.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:05.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.246 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:05.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:05.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:05.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:05.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:05.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:05.545 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:05.609 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:53:05.609 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg degraded, 3 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1001,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":244,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238500,"num_in_osds":7,"osd_in_since":1726238900,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":86},{"state_name":"active+undersized+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":46854,"data_bytes":3073513469,"bytes_used":11605401600,"bytes_avail":660359847936,"bytes_total":671965249536,"misplaced_objects":4324,"misplaced_total":140562,"misplaced_ratio":0.030762225921657348,"recovering_objects_per_sec":163,"recovering_bytes_per_sec":10097340,"recovering_keys_per_sec":0,"num_objects_recovered":1565,"num_bytes_recovered":96534528,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":449},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi154.daekzd","status":"up:active","gid":24539}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"}},"servicemap":{"epoch":25,"modified":"2024-09-13T14:50:26.951855+0000","services":{}},"progress_events":{}} 2024-09-13T14:53:05.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:05.671 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:05 smithi172 ceph-mon[100696]: osdmap e243: 8 total, 8 up, 7 in 2024-09-13T14:53:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2343/148518 objects degraded (1.578%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:05 smithi172 ceph-mon[100696]: osdmap e244: 8 total, 8 up, 7 in 2024-09-13T14:53:05.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:05.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:05.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:05.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:05.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:05.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:05 smithi154 ceph-mon[102314]: osdmap e243: 8 total, 8 up, 7 in 2024-09-13T14:53:05.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2343/148518 objects degraded (1.578%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:05.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:05 smithi154 ceph-mon[102314]: osdmap e244: 8 total, 8 up, 7 in 2024-09-13T14:53:05.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:05 smithi154 ceph-mon[107038]: osdmap e243: 8 total, 8 up, 7 in 2024-09-13T14:53:05.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2343/148518 objects degraded (1.578%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:05.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:05 smithi154 ceph-mon[107038]: osdmap e244: 8 total, 8 up, 7 in 2024-09-13T14:53:06.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:06.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:06.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.192 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:06.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:06.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:06.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:06.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:06.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:06.503 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:06.522 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:53:06.523 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:06.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:06.551 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:53:06.552 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:06.552 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:06 smithi154 ceph-mon[102314]: pgmap v723: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 449 op/s; 4324/140562 objects misplaced (3.076%); 9.6 MiB/s, 163 objects/s recovering 2024-09-13T14:53:06.552 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1365744140' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:53:06.553 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:06 smithi154 ceph-mon[107038]: pgmap v723: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 449 op/s; 4324/140562 objects misplaced (3.076%); 9.6 MiB/s, 163 objects/s recovering 2024-09-13T14:53:06.553 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1365744140' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:53:06.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:06.575 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:53:06.575 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:06.601 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:53:06.601 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:06.629 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:53:06.629 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:06.655 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:53:06.656 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:06.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:06.676 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:53:06.677 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:06.692 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:06.705 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:53:06.705 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:06 smithi172 ceph-mon[100696]: pgmap v723: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 449 op/s; 4324/140562 objects misplaced (3.076%); 9.6 MiB/s, 163 objects/s recovering 2024-09-13T14:53:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1365744140' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:53:06.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:06.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:06.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:06.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:06.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:07.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:07.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:07.111 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:53:07.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:53:07.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:07.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:07.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:07.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:07.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:07.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:07.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:07.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:07.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:07.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/864130717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:07.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/864130717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:07.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/864130717' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.759 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:07.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:07.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:07.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:07.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:07.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:08.011 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:08.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:08.137 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:08.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:08.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:08.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:08.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:08.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:08.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:08.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:08.661 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:08 smithi172 ceph-mon[100696]: pgmap v725: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 4324/140562 objects misplaced (3.076%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:53:08.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:08.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:08.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:08.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:08.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:08.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:08 smithi154 ceph-mon[102314]: pgmap v725: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 4324/140562 objects misplaced (3.076%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:53:08.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:08 smithi154 ceph-mon[107038]: pgmap v725: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 4324/140562 objects misplaced (3.076%); 12 MiB/s, 195 objects/s recovering 2024-09-13T14:53:09.024 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.048 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:09.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:09.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:09.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:09.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:09.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:09.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.444 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:09.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:09.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:09.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:09.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:09.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:09.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.841 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:09.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:09.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:09.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:09.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:10.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.094 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:10.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:10.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.239 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:10.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:10.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:10.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:10.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:10.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:10.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:10 smithi154 ceph-mon[102314]: pgmap v726: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 407 op/s; 4324/138747 objects misplaced (3.116%); 4.9 MiB/s, 90 objects/s recovering 2024-09-13T14:53:10.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:10.585 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:10 smithi154 ceph-mon[107038]: pgmap v726: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 407 op/s; 4324/138747 objects misplaced (3.116%); 4.9 MiB/s, 90 objects/s recovering 2024-09-13T14:53:10.585 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:10.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:10.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:10.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:10.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27212 5.a 5:58ee1d02:::benchmark_data_smithi154_144140_object27211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.14810731099999, 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17096689200000001, 2024-09-13T14:53:10.816 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27212 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:53:10.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664197+0000", 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664198+0000", 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9800000000000003e-07 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664220+0000", 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1591999999999999e-05 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664227+0000", 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.712e-06 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664233+0000", 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8289999999999996e-06 2024-09-13T14:53:10.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669218+0000", 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049845829999999999 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669320+0000", 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102304 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669434+0000", 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113581 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674379+0000", 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00494525 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674700+0000", 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032070699999999999 2024-09-13T14:53:10.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.834993+0000", 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160292992 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835121+0000", 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012801800000000001 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835164+0000", 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3726000000000001e-05 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27213 5.a 5:571e06b1:::benchmark_data_smithi154_144140_object27212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.14806327899998, 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.17104000799999999, 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.823 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27213 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664241+0000", 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664242+0000", 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2e-07 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664265+0000", 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3476e-05 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664273+0000", 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2230000000000001e-06 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.664278+0000", 2024-09-13T14:53:10.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1599999999999997e-06 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.670614+0000", 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063359059999999997 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671024+0000", 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041025499999999999 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671138+0000", 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114418 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674735+0000", 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035970080000000001 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674771+0000", 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5506999999999997e-05 2024-09-13T14:53:10.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835177+0000", 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16040633400000001 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835217+0000", 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9641000000000002e-05 2024-09-13T14:53:10.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835281+0000", 2024-09-13T14:53:10.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4659999999999994e-05 2024-09-13T14:53:10.830 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27221 5.a 5:51807044:::benchmark_data_smithi154_144140_object27220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.14390971800003, 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16984976800000001, 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.831 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27221 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668395+0000", 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668396+0000", 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9999999999999997e-07 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668416+0000", 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0516000000000001e-05 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668424+0000", 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6389999999999992e-06 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.668430+0000", 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9939999999999997e-06 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672751+0000", 2024-09-13T14:53:10.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043213569999999996 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673077+0000", 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032614299999999998 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.673198+0000", 2024-09-13T14:53:10.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120492 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833603+0000", 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160404986 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835418+0000", 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018152349999999999 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838152+0000", 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002733989 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838206+0000", 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4165000000000001e-05 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838245+0000", 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8652000000000002e-05 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27226 5.a 5:58f709a9:::benchmark_data_smithi154_144140_object27225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:53:10.838 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.14314900800002, 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.169199034, 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27226 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:53:10.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669156+0000", 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1900000000000001e-07 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669187+0000", 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0571999999999998e-05 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669198+0000", 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0920999999999999e-05 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.669205+0000", 2024-09-13T14:53:10.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6479999999999997e-06 2024-09-13T14:53:10.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674834+0000", 2024-09-13T14:53:10.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056293519999999998 2024-09-13T14:53:10.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674951+0000", 2024-09-13T14:53:10.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011713 2024-09-13T14:53:10.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675069+0000", 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117726 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835380+0000", 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160310762 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835455+0000", 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5813000000000001e-05 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838286+0000", 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028307559999999998 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838325+0000", 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8574e-05 2024-09-13T14:53:10.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838355+0000", 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9961e-05 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27229 5.2 e117/111 5:47a3bf38:::benchmark_data_smithi154_144140_object27228:head v 117'1668, mlcod=117'1665)", 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.14107303000003, 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16704787199999999, 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.849 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671232+0000", 2024-09-13T14:53:10.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5599999999999995e-07 2024-09-13T14:53:10.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671265+0000", 2024-09-13T14:53:10.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3231000000000001e-05 2024-09-13T14:53:10.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671274+0000", 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9479999999999997e-06 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671280+0000", 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672510+0000", 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012303030000000001 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672591+0000", 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1227e-05 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838248+0000", 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16565628499999999 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838280+0000", 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1881999999999997e-05 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27212 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.14097306099995, 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16395237200000001, 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.855 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.856 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:53:10.856 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:53:10.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671332+0000", 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8700000000000001e-07 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671335+0000", 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.858e-06 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671342+0000", 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4189999999999996e-06 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671347+0000", 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9089999999999999e-06 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674368+0000", 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003020371 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674374+0000", 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2829999999999998e-06 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835284+0000", 2024-09-13T14:53:10.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.160910145 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27231 5.d 5:bd1c1b1b:::benchmark_data_smithi154_144140_object27230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.14065406199995, 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16226318100000001, 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.860 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27231 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671651+0000", 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671652+0000", 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0100000000000003e-07 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671671+0000", 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9686000000000001e-05 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671680+0000", 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6989999999999992e-06 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671685+0000", 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2950000000000001e-06 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.671954+0000", 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026847900000000002 2024-09-13T14:53:10.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672066+0000", 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011244 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.672249+0000", 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000182573 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674493+0000", 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022442439999999998 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.674564+0000", 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1538000000000005e-05 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675871+0000", 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013069449999999999 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833838+0000", 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15796623000000001 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833914+0000", 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6150999999999996e-05 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.866 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.13722937399996, 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158535863, 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675075+0000", 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675076+0000", 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5400000000000002e-07 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675079+0000", 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.365e-06 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675087+0000", 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6249999999999998e-06 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675093+0000", 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8300000000000001e-06 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833585+0000", 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158492884 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833596+0000", 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0645999999999999e-05 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833611+0000", 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5058999999999999e-05 2024-09-13T14:53:10.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27237 5.b 5:d9e8c993:::benchmark_data_smithi154_144140_object27236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.137220329, 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16640950199999999, 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27237 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675084+0000", 2024-09-13T14:53:10.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675086+0000", 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.274e-06 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675108+0000", 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2412000000000002e-05 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675120+0000", 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1729000000000001e-05 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675127+0000", 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7680000000000003e-06 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675594+0000", 2024-09-13T14:53:10.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000466992 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675717+0000", 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012363700000000001 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833850+0000", 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158133311 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836021+0000", 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021703880000000001 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.836035+0000", 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3657e-05 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841412+0000", 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053777620000000003 2024-09-13T14:53:10.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841458+0000", 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5235000000000003e-05 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.841494+0000", 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6337000000000001e-05 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27239 5.2 e117/111 5:4246bfbf:::benchmark_data_smithi154_144140_object27238:head v 117'1669, mlcod=117'1665)", 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.136754451, 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16283244499999999, 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675550+0000", 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675551+0000", 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4000000000000001e-07 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675624+0000", 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3522000000000005e-05 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675633+0000", 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0499999999999997e-06 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675639+0000", 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8839999999999999e-06 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833644+0000", 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.158004476 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833663+0000", 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9238000000000001e-05 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838359+0000", 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046954900000000001 2024-09-13T14:53:10.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838383+0000", 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4145e-05 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27231 5.d e117/111 ondisk, result = 0)", 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.13661417000003, 2024-09-13T14:53:10.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15823179200000001, 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:53:10.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675691+0000", 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8500000000000001e-07 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675695+0000", 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9099999999999998e-06 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675705+0000", 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6670000000000008e-06 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675710+0000", 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7810000000000002e-06 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675848+0000", 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137677 2024-09-13T14:53:10.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675855+0000", 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1790000000000002e-06 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833922+0000", 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15806699299999999 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:27241 5.6 5:6b00c174:::benchmark_data_smithi154_144140_object27240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.13656663699999, 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16225515099999999, 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 27241 2024-09-13T14:53:10.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675738+0000", 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675739+0000", 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6499999999999999e-07 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675758+0000", 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8799000000000001e-05 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675766+0000", 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4649999999999994e-06 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675772+0000", 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6470000000000003e-06 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675823+0000", 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1010000000000001e-05 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675943+0000", 2024-09-13T14:53:10.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120389 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833416+0000", 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15747296699999999 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835725+0000", 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023092400000000002 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835762+0000", 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6693000000000003e-05 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837162+0000", 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001400152 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837949+0000", 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000787417 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837993+0000", 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3707e-05 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:27240 5.e e117/111 5:77f483c1:::benchmark_data_smithi154_144140_object27239:head v 117'1698, mlcod=117'1696)", 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.13655510900003, 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16227988300000001, 2024-09-13T14:53:10.893 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675750+0000", 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675755+0000", 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5079999999999996e-06 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675810+0000", 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.452e-05 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675818+0000", 2024-09-13T14:53:10.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8659999999999995e-06 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675824+0000", 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7930000000000003e-06 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675915+0000", 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0556000000000005e-05 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.675932+0000", 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6979000000000001e-05 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.837997+0000", 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16206567699999999 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.838029+0000", 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1983999999999997e-05 2024-09-13T14:53:10.897 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27226 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.13541547600005, 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15849907399999999, 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676889+0000", 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676890+0000", 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7599999999999997e-07 2024-09-13T14:53:10.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676894+0000", 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1590000000000003e-06 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833403+0000", 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.156509065 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833418+0000", 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5142e-05 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835366+0000", 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019481570000000001 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835374+0000", 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5639999999999999e-06 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835388+0000", 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4411e-05 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.34620.0:27221 5.a e117/110 ondisk, result = 0)", 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 583.13534427299999, 2024-09-13T14:53:10.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15846376300000001, 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676960+0000", 2024-09-13T14:53:10.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676961+0000", 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4199999999999995e-07 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.676965+0000", 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1130000000000003e-06 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833821+0000", 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15685534100000001 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.833830+0000", 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1130000000000007e-06 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835407+0000", 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015777499999999999 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835413+0000", 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8749999999999997e-06 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:43:27.835424+0000", 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0829000000000001e-05 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:117172 5.a 5:5df8026a:::benchmark_data_smithi154_144140_object6819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 493.65039115100001, 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15741727699999999, 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 117172 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.907 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161914+0000", 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161915+0000", 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.314e-06 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161919+0000", 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6569999999999996e-06 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161933+0000", 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3055e-05 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.161940+0000", 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.593e-06 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164296+0000", 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023560709999999999 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164406+0000", 2024-09-13T14:53:10.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011003899999999999 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.164510+0000", 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010334400000000001 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.175396+0000", 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010886682 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179579+0000", 2024-09-13T14:53:10.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041824269999999998 2024-09-13T14:53:10.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:10.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319249+0000", 2024-09-13T14:53:10.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13966985800000001 2024-09-13T14:53:10.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319294+0000", 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5141e-05 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319331+0000", 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7095999999999999e-05 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:117176 5.a 5:574be98d:::benchmark_data_smithi154_144140_object6823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 493.65019464599999, 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15802838399999999, 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 117176 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.915 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162110+0000", 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162111+0000", 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5499999999999997e-07 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162115+0000", 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6679999999999999e-06 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162124+0000", 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3700000000000001e-06 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.162130+0000", 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.789e-06 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171194+0000", 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090644290000000006 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171302+0000", 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108194 2024-09-13T14:53:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.171403+0000", 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100468 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179555+0000", 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008152365 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.179585+0000", 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0063e-05 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320052+0000", 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14046650699999999 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320103+0000", 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0924000000000002e-05 2024-09-13T14:53:10.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.320138+0000", 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5751999999999999e-05 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:117185 5.f e132/130 5:f308a48e:::benchmark_data_smithi154_144140_object6832:head v 132'7432, mlcod=125'5196)", 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 493.63749228, 2024-09-13T14:53:10.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.151216923, 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174812+0000", 2024-09-13T14:53:10.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174813+0000", 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7100000000000001e-07 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174820+0000", 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8739999999999997e-06 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174828+0000", 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4130000000000006e-06 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.174834+0000", 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8370000000000002e-06 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319258+0000", 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14442416499999999 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.319282+0000", 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3923000000000001e-05 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.325925+0000", 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066427910000000003 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:57.326029+0000", 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104249 2024-09-13T14:53:10.925 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.34620.0:117851 5.e e133/111 5:700ea90e:::benchmark_data_smithi154_144140_object7498:head v 133'7344, mlcod=133'7342)", 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 492.71567425000001, 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.26475042900000001, 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096630+0000", 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096632+0000", 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.158e-06 2024-09-13T14:53:10.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.096640+0000", 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4800000000000001e-06 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219022+0000", 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.122381897 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219039+0000", 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6616000000000002e-05 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219381+0000", 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000342443 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.219401+0000", 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9698e-05 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.361334+0000", 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.141933647 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:44:58.361381+0000", 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6489999999999997e-05 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:197886 5.d 5:ba09dd65:::benchmark_data_smithi154_144140_object87533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 390.79941721099999, 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15809542200000001, 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 197886 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.931 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012887+0000", 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:10.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012889+0000", 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2810000000000001e-06 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012893+0000", 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.771e-06 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012906+0000", 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.329e-05 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012913+0000", 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3089999999999998e-06 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.012969+0000", 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.013085+0000", 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115708 2024-09-13T14:53:10.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,7", 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.013194+0000", 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010938300000000001 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.018482+0000", 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052877269999999999 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.125895+0000", 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.107413177 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170889+0000", 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.044993742000000003 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170944+0000", 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4966999999999997e-05 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:10.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:40.170983+0000", 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8781e-05 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:10.937 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:53:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:10.975 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:10.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:10 smithi172 ceph-mon[100696]: pgmap v726: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 86 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 407 op/s; 4324/138747 objects misplaced (3.116%); 4.9 MiB/s, 90 objects/s recovering 2024-09-13T14:53:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:11.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:11.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:11.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:11.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:11.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:11.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:11.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:11.505 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:53:11.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:11.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:11.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.665 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:11.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:11.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:11.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:11.878 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:11 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3892971756' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:11.879 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:11 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:11.879 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:11 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3892971756' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:11.879 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:11 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:11.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:11.925 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:11.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:11.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:11 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3892971756' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:11.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:11 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:53:12.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:12.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:12.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:12.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:53:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:53:11] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-13T14:53:12.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:12.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:12.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:12.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:12.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:12.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:12.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:12.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[102314]: pgmap v727: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 3852/133731 objects misplaced (2.880%); 8.5 MiB/s, 155 objects/s recovering 2024-09-13T14:53:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[102314]: osdmap e245: 8 total, 8 up, 7 in 2024-09-13T14:53:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[107038]: pgmap v727: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 3852/133731 objects misplaced (2.880%); 8.5 MiB/s, 155 objects/s recovering 2024-09-13T14:53:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:12 smithi154 ceph-mon[107038]: osdmap e245: 8 total, 8 up, 7 in 2024-09-13T14:53:12.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:12.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:12.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:12.886 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:12.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:53:12.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:12.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:12 smithi172 ceph-mon[100696]: pgmap v727: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 3852/133731 objects misplaced (2.880%); 8.5 MiB/s, 155 objects/s recovering 2024-09-13T14:53:12.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:12 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:12.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:12 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:53:12.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:12 smithi172 ceph-mon[100696]: osdmap e245: 8 total, 8 up, 7 in 2024-09-13T14:53:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:13.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:13.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:13.145 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:13.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:13.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:13.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:13.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:13.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:13.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:13.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:13.655 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]} : dispatch 2024-09-13T14:53:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]} : dispatch 2024-09-13T14:53:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-13T14:53:13.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:13.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]} : dispatch 2024-09-13T14:53:13.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]} : dispatch 2024-09-13T14:53:13.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-13T14:53:13.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:13.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:13.846 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:13.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:13.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:13.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:13.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:13.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]} : dispatch 2024-09-13T14:53:13.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]} : dispatch 2024-09-13T14:53:13.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]} : dispatch 2024-09-13T14:53:14.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:14.037 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:14.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:14.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:14.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:14.300 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:14.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:14.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:14.436 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:14.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:14.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:14.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[102314]: pgmap v729: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 284 op/s; 3852/133731 objects misplaced (2.880%); 3.6 MiB/s, 65 objects/s recovering 2024-09-13T14:53:14.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]': finished 2024-09-13T14:53:14.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:53:14.650 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-13T14:53:14.650 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[102314]: osdmap e246: 8 total, 8 up, 7 in 2024-09-13T14:53:14.650 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[107038]: pgmap v729: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 284 op/s; 3852/133731 objects misplaced (2.880%); 3.6 MiB/s, 65 objects/s recovering 2024-09-13T14:53:14.650 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]': finished 2024-09-13T14:53:14.650 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:53:14.651 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-13T14:53:14.651 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:14 smithi154 ceph-mon[107038]: osdmap e246: 8 total, 8 up, 7 in 2024-09-13T14:53:14.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:14.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:14.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:53:14.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:14.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:14.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:14.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:14 smithi172 ceph-mon[100696]: pgmap v729: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 284 op/s; 3852/133731 objects misplaced (2.880%); 3.6 MiB/s, 65 objects/s recovering 2024-09-13T14:53:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]': finished 2024-09-13T14:53:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-13T14:53:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 0]}]': finished 2024-09-13T14:53:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:14 smithi172 ceph-mon[100696]: osdmap e246: 8 total, 8 up, 7 in 2024-09-13T14:53:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:14.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:15.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:15.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:15.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:15.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:15.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:15.378 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:15.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:15.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.736 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:15 smithi154 ceph-mon[102314]: osdmap e247: 8 total, 8 up, 7 in 2024-09-13T14:53:15.737 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:15.737 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:15 smithi154 ceph-mon[107038]: osdmap e247: 8 total, 8 up, 7 in 2024-09-13T14:53:15.737 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:15.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:15.780 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:15.881 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:15.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:15.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:15.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:15 smithi172 ceph-mon[100696]: osdmap e247: 8 total, 8 up, 7 in 2024-09-13T14:53:15.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-13T14:53:16.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:16.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:16.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:16.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:16.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:16.423 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:16.525 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:16.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:16.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:16.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:16.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.805 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:16.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:16.897 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:16 smithi154 ceph-mon[102314]: pgmap v732: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 3356/125190 objects misplaced (2.681%); 10 MiB/s, 170 objects/s recovering 2024-09-13T14:53:16.897 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:16 smithi154 ceph-mon[102314]: from='client.54997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:16.897 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:16 smithi154 ceph-mon[107038]: pgmap v732: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 3356/125190 objects misplaced (2.681%); 10 MiB/s, 170 objects/s recovering 2024-09-13T14:53:16.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:16 smithi154 ceph-mon[107038]: from='client.54997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:16.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:16.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:16.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:16 smithi172 ceph-mon[100696]: pgmap v732: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 3356/125190 objects misplaced (2.681%); 10 MiB/s, 170 objects/s recovering 2024-09-13T14:53:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:16 smithi172 ceph-mon[100696]: from='client.54997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:17.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:17.061 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:17.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:17.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:17.232 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:53:17.237 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:53:17.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:17.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:17.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:17.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:17.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:17.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:17.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.706 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:17.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:17.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:17.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:17.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:17.965 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:17.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:18.067 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:18.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:18.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:18.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:18.244 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:18.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:18.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:18.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:18.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:18.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:18.617 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:18.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:18.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:18.828 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:18 smithi172 ceph-mon[100696]: pgmap v733: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 3356/125190 objects misplaced (2.681%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:53:18.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:18.870 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:18.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:18 smithi154 ceph-mon[102314]: pgmap v733: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 3356/125190 objects misplaced (2.681%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:53:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:18 smithi154 ceph-mon[107038]: pgmap v733: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 3356/125190 objects misplaced (2.681%); 5.2 MiB/s, 82 objects/s recovering 2024-09-13T14:53:18.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:18.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:18.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:19.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:19.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:19.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:19.238 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2024-09-13T14:53:15.005205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2737707005,"num_objects":41730,"num_object_clones":0,"num_object_copies":125190,"num_objects_missing_on_primary":80,"num_objects_missing":80,"num_objects_degraded":0,"num_objects_misplaced":3356,"num_objects_unfound":0,"num_objects_dirty":41730,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":130969,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43585,"num_bytes_recovered":2729921669,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115900,"ondisk_log_size":115900,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10307672,"kb_used_data":8288520,"kb_used_omap":32,"kb_used_meta":2019039,"kb_avail":645908392,"statfs":{"total":671965249536,"available":661410193408,"internally_reserved":0,"allocated":8487444480,"data_stored":8478099955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2067496496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,7,3,2,3,0,2,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296157184,"num_objects":-4519,"num_object_clones":0,"num_object_copies":-13557,"num_objects_missing_on_primary":-7,"num_objects_missing":-7,"num_objects_degraded":0,"num_objects_misplaced":-968,"num_objects_unfound":0,"num_objects_dirty":-4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4519,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":62980096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001847"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":375,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.575846+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:53:14.575846+0000","last_peered":"2024-09-13T14:53:14.575846+0000","last_clean":"2024-09-13T14:53:14.575846+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:53:14.575846+0000","last_undegraded":"2024-09-13T14:53:14.575846+0000","last_fullsized":"2024-09-13T14:53:14.575846+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":112,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507433+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:53:11.507433+0000","last_peered":"2024-09-13T14:53:11.507433+0000","last_clean":"2024-09-13T14:53:11.507433+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:53:11.507433+0000","last_undegraded":"2024-09-13T14:53:11.507433+0000","last_fullsized":"2024-09-13T14:53:11.507433+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":83,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535558+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:53:14.535558+0000","last_peered":"2024-09-13T14:53:14.535558+0000","last_clean":"2024-09-13T14:53:14.535558+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:53:14.535558+0000","last_undegraded":"2024-09-13T14:53:14.535558+0000","last_fullsized":"2024-09-13T14:53:14.535558+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":361,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497183+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:53:11.497183+0000","last_peered":"2024-09-13T14:53:11.497183+0000","last_clean":"2024-09-13T14:53:11.497183+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:53:11.497183+0000","last_undegraded":"2024-09-13T14:53:11.497183+0000","last_fullsized":"2024-09-13T14:53:11.497183+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":361,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497243+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:53:11.497243+0000","last_peered":"2024-09-13T14:53:11.497243+0000","last_clean":"2024-09-13T14:53:11.497243+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:53:11.497243+0000","last_undegraded":"2024-09-13T14:53:11.497243+0000","last_fullsized":"2024-09-13T14:53:11.497243+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":182,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507151+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:53:11.507151+0000","last_peered":"2024-09-13T14:53:11.507151+0000","last_clean":"2024-09-13T14:53:11.507151+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:53:11.507151+0000","last_undegraded":"2024-09-13T14:53:11.507151+0000","last_fullsized":"2024-09-13T14:53:11.507151+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":91,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.539025+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:53:14.539025+0000","last_peered":"2024-09-13T14:53:14.539025+0000","last_clean":"2024-09-13T14:53:14.539025+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:53:14.539025+0000","last_undegraded":"2024-09-13T14:53:14.539025+0000","last_fullsized":"2024-09-13T14:53:14.539025+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":158,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.509506+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:53:11.509506+0000","last_peered":"2024-09-13T14:53:11.509506+0000","last_clean":"2024-09-13T14:53:11.509506+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:53:11.509506+0000","last_undegraded":"2024-09-13T14:53:11.509506+0000","last_fullsized":"2024-09-13T14:53:11.509506+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"247'8280","reported_seq":9121,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.557001+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:53:14.557001+0000","last_peered":"2024-09-13T14:53:14.557001+0000","last_clean":"2024-09-13T14:53:14.557001+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:53:14.557001+0000","last_undegraded":"2024-09-13T14:53:14.557001+0000","last_fullsized":"2024-09-13T14:53:14.557001+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171442176,"num_objects":2616,"num_object_clones":0,"num_object_copies":7848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":298,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.537536+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:53:13.537536+0000","last_peered":"2024-09-13T14:53:13.537536+0000","last_clean":"2024-09-13T14:53:13.537536+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:53:13.537536+0000","last_undegraded":"2024-09-13T14:53:13.537536+0000","last_fullsized":"2024-09-13T14:53:13.537536+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":110,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.533261+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:53:13.533261+0000","last_peered":"2024-09-13T14:53:13.533261+0000","last_clean":"2024-09-13T14:53:13.533261+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:53:13.533261+0000","last_undegraded":"2024-09-13T14:53:13.533261+0000","last_fullsized":"2024-09-13T14:53:13.533261+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"247'8029","reported_seq":9652,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.808999+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:53:14.808999+0000","last_peered":"2024-09-13T14:53:14.808999+0000","last_clean":"2024-09-13T14:53:14.808999+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:53:14.808999+0000","last_undegraded":"2024-09-13T14:53:14.808999+0000","last_fullsized":"2024-09-13T14:53:14.808999+0000","mapping_epoch":170,"log_start":"171'1600","ondisk_log_start":"171'1600","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6429,"log_dups_size":0,"ondisk_log_size":6429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167051264,"num_objects":2549,"num_object_clones":0,"num_object_copies":7647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8029,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":110,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.511130+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:53:11.511130+0000","last_peered":"2024-09-13T14:53:11.511130+0000","last_clean":"2024-09-13T14:53:11.511130+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:53:11.511130+0000","last_undegraded":"2024-09-13T14:53:11.511130+0000","last_fullsized":"2024-09-13T14:53:11.511130+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":262,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507252+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:53:11.507252+0000","last_peered":"2024-09-13T14:53:11.507252+0000","last_clean":"2024-09-13T14:53:11.507252+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:53:11.507252+0000","last_undegraded":"2024-09-13T14:53:11.507252+0000","last_fullsized":"2024-09-13T14:53:11.507252+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"246'8202","reported_seq":9916,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.220942+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:53:14.220942+0000","last_peered":"2024-09-13T14:53:14.220942+0000","last_clean":"2024-09-13T14:53:14.220942+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:53:14.220942+0000","last_undegraded":"2024-09-13T14:53:14.220942+0000","last_fullsized":"2024-09-13T14:53:14.220942+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":7926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8202,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":87,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.533868+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:53:13.533868+0000","last_peered":"2024-09-13T14:53:13.533868+0000","last_clean":"2024-09-13T14:53:13.533868+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:53:13.533868+0000","last_undegraded":"2024-09-13T14:53:13.533868+0000","last_fullsized":"2024-09-13T14:53:13.533868+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":42,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.509350+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:53:11.509350+0000","last_peered":"2024-09-13T14:53:11.509350+0000","last_clean":"2024-09-13T14:53:11.509350+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:53:11.509350+0000","last_undegraded":"2024-09-13T14:53:11.509350+0000","last_fullsized":"2024-09-13T14:53:11.509350+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"245'7952","reported_seq":14059,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:12.932443+0000","last_change":"2024-09-13T14:52:50.434519+0000","last_active":"2024-09-13T14:53:12.932443+0000","last_peered":"2024-09-13T14:53:12.932443+0000","last_clean":"2024-09-13T14:53:12.932443+0000","last_became_active":"2024-09-13T14:52:50.417699+0000","last_became_peered":"2024-09-13T14:52:50.417699+0000","last_unstale":"2024-09-13T14:53:12.932443+0000","last_undegraded":"2024-09-13T14:53:12.932443+0000","last_fullsized":"2024-09-13T14:53:12.932443+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7952,"log_dups_size":0,"ondisk_log_size":7952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:10.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167510016,"num_objects":2556,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7952,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2948,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":99,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.534019+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:53:13.534019+0000","last_peered":"2024-09-13T14:53:13.534019+0000","last_clean":"2024-09-13T14:53:13.534019+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:53:13.534019+0000","last_undegraded":"2024-09-13T14:53:13.534019+0000","last_fullsized":"2024-09-13T14:53:13.534019+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":340,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497617+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:53:11.497617+0000","last_peered":"2024-09-13T14:53:11.497617+0000","last_clean":"2024-09-13T14:53:11.497617+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:53:11.497617+0000","last_undegraded":"2024-09-13T14:53:11.497617+0000","last_fullsized":"2024-09-13T14:53:11.497617+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"246'8112","reported_seq":25370,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.169084+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:53:14.169084+0000","last_peered":"2024-09-13T14:53:14.169084+0000","last_clean":"2024-09-13T14:53:14.169084+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:53:14.169084+0000","last_undegraded":"2024-09-13T14:53:14.169084+0000","last_fullsized":"2024-09-13T14:53:14.169084+0000","mapping_epoch":232,"log_start":"171'1500","ondisk_log_start":"171'1500","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173146112,"num_objects":2642,"num_object_clones":0,"num_object_copies":7926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8112,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":353,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.548647+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:53:14.548647+0000","last_peered":"2024-09-13T14:53:14.548647+0000","last_clean":"2024-09-13T14:53:14.548647+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:53:14.548647+0000","last_undegraded":"2024-09-13T14:53:14.548647+0000","last_fullsized":"2024-09-13T14:53:14.548647+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":229,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507676+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:53:11.507676+0000","last_peered":"2024-09-13T14:53:11.507676+0000","last_clean":"2024-09-13T14:53:11.507676+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:53:11.507676+0000","last_undegraded":"2024-09-13T14:53:11.507676+0000","last_fullsized":"2024-09-13T14:53:11.507676+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"246'8032","reported_seq":10596,"reported_epoch":246,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:53:14.192189+0000","last_change":"2024-09-13T14:53:04.771859+0000","last_active":"2024-09-13T14:53:14.192189+0000","last_peered":"2024-09-13T14:53:14.192189+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:53:14.192189+0000","last_undegraded":"2024-09-13T14:53:14.192189+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":208,"log_start":"171'1500","ondisk_log_start":"171'1500","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:40.513693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170917888,"num_objects":2608,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1213,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8032,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1120,"num_bytes_recovered":68419584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1395},{"shards":"3,4,6","objects":1213}],"blocked_by":[],"up_primary":2,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":129,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.509530+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:53:11.509530+0000","last_peered":"2024-09-13T14:53:11.509530+0000","last_clean":"2024-09-13T14:53:11.509530+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:53:11.509530+0000","last_undegraded":"2024-09-13T14:53:11.509530+0000","last_fullsized":"2024-09-13T14:53:11.509530+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":226,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507826+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:53:11.507826+0000","last_peered":"2024-09-13T14:53:11.507826+0000","last_clean":"2024-09-13T14:53:11.507826+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:53:11.507826+0000","last_undegraded":"2024-09-13T14:53:11.507826+0000","last_fullsized":"2024-09-13T14:53:11.507826+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"247'8215","reported_seq":12892,"reported_epoch":247,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:53:14.814470+0000","last_change":"2024-09-13T14:52:50.916272+0000","last_active":"2024-09-13T14:53:14.814470+0000","last_peered":"2024-09-13T14:53:14.814470+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:53:14.814470+0000","last_undegraded":"2024-09-13T14:53:14.814470+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"171'1800","ondisk_log_start":"171'1800","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":7821,"num_objects_missing_on_primary":80,"num_objects_missing":80,"num_objects_degraded":0,"num_objects_misplaced":2143,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155320320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"1,3,4,5","objects":3},{"shards":"1,3,5","objects":2136},{"shards":"1,4,5","objects":2},{"shards":"3,5","objects":466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":221,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507323+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:53:11.507323+0000","last_peered":"2024-09-13T14:53:11.507323+0000","last_clean":"2024-09-13T14:53:11.507323+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:53:11.507323+0000","last_undegraded":"2024-09-13T14:53:11.507323+0000","last_fullsized":"2024-09-13T14:53:11.507323+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":256,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.511177+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:53:11.511177+0000","last_peered":"2024-09-13T14:53:11.511177+0000","last_clean":"2024-09-13T14:53:11.511177+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:53:11.511177+0000","last_undegraded":"2024-09-13T14:53:11.511177+0000","last_fullsized":"2024-09-13T14:53:11.511177+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"247'7928","reported_seq":12362,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.582130+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:53:14.582130+0000","last_peered":"2024-09-13T14:53:14.582130+0000","last_clean":"2024-09-13T14:53:14.582130+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:53:14.582130+0000","last_undegraded":"2024-09-13T14:53:14.582130+0000","last_fullsized":"2024-09-13T14:53:14.582130+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7928,"log_dups_size":0,"ondisk_log_size":7928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168886296,"num_objects":2578,"num_object_clones":0,"num_object_copies":7734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7928,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":91,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.539849+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:53:14.539849+0000","last_peered":"2024-09-13T14:53:14.539849+0000","last_clean":"2024-09-13T14:53:14.539849+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:53:14.539849+0000","last_undegraded":"2024-09-13T14:53:14.539849+0000","last_fullsized":"2024-09-13T14:53:14.539849+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":198,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.537641+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:53:13.537641+0000","last_peered":"2024-09-13T14:53:13.537641+0000","last_clean":"2024-09-13T14:53:13.537641+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:53:13.537641+0000","last_undegraded":"2024-09-13T14:53:13.537641+0000","last_fullsized":"2024-09-13T14:53:13.537641+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"247'8125","reported_seq":9381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.541328+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:53:14.541328+0000","last_peered":"2024-09-13T14:53:14.541328+0000","last_clean":"2024-09-13T14:53:14.541328+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:53:14.541328+0000","last_undegraded":"2024-09-13T14:53:14.541328+0000","last_fullsized":"2024-09-13T14:53:14.541328+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171638784,"num_objects":2619,"num_object_clones":0,"num_object_copies":7857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":543,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.499232+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:53:11.499232+0000","last_peered":"2024-09-13T14:53:11.499232+0000","last_clean":"2024-09-13T14:53:11.499232+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:53:11.499232+0000","last_undegraded":"2024-09-13T14:53:11.499232+0000","last_fullsized":"2024-09-13T14:53:11.499232+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":814,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.533591+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:53:13.533591+0000","last_peered":"2024-09-13T14:53:13.533591+0000","last_clean":"2024-09-13T14:53:13.533591+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:53:13.533591+0000","last_undegraded":"2024-09-13T14:53:13.533591+0000","last_fullsized":"2024-09-13T14:53:13.533591+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":311,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.538880+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:53:14.538880+0000","last_peered":"2024-09-13T14:53:14.538880+0000","last_clean":"2024-09-13T14:53:14.538880+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:53:14.538880+0000","last_undegraded":"2024-09-13T14:53:14.538880+0000","last_fullsized":"2024-09-13T14:53:14.538880+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":42,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.509700+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:53:11.509700+0000","last_peered":"2024-09-13T14:53:11.509700+0000","last_clean":"2024-09-13T14:53:11.509700+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:53:11.509700+0000","last_undegraded":"2024-09-13T14:53:11.509700+0000","last_fullsized":"2024-09-13T14:53:11.509700+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"247'8198","reported_seq":12936,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.684898+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:53:14.684898+0000","last_peered":"2024-09-13T14:53:14.684898+0000","last_clean":"2024-09-13T14:53:14.684898+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:53:14.684898+0000","last_undegraded":"2024-09-13T14:53:14.684898+0000","last_fullsized":"2024-09-13T14:53:14.684898+0000","mapping_epoch":223,"log_start":"173'2118","ondisk_log_start":"173'2118","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":392,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.532844+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:53:14.532844+0000","last_peered":"2024-09-13T14:53:14.532844+0000","last_clean":"2024-09-13T14:53:14.532844+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:53:14.532844+0000","last_undegraded":"2024-09-13T14:53:14.532844+0000","last_fullsized":"2024-09-13T14:53:14.532844+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":355,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.498185+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:53:11.498185+0000","last_peered":"2024-09-13T14:53:11.498185+0000","last_clean":"2024-09-13T14:53:11.498185+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:53:11.498185+0000","last_undegraded":"2024-09-13T14:53:11.498185+0000","last_fullsized":"2024-09-13T14:53:11.498185+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":366,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497476+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:53:11.497476+0000","last_peered":"2024-09-13T14:53:11.497476+0000","last_clean":"2024-09-13T14:53:11.497476+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:53:11.497476+0000","last_undegraded":"2024-09-13T14:53:11.497476+0000","last_fullsized":"2024-09-13T14:53:11.497476+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"246'8049","reported_seq":18371,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.860809+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:53:13.860809+0000","last_peered":"2024-09-13T14:53:13.860809+0000","last_clean":"2024-09-13T14:53:13.860809+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:53:13.860809+0000","last_undegraded":"2024-09-13T14:53:13.860809+0000","last_fullsized":"2024-09-13T14:53:13.860809+0000","mapping_epoch":235,"log_start":"164'700","ondisk_log_start":"164'700","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7349,"log_dups_size":0,"ondisk_log_size":7349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":7641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8049,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":893,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.536831+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:53:13.536831+0000","last_peered":"2024-09-13T14:53:13.536831+0000","last_clean":"2024-09-13T14:53:13.536831+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:53:13.536831+0000","last_undegraded":"2024-09-13T14:53:13.536831+0000","last_fullsized":"2024-09-13T14:53:13.536831+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":350,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.533106+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:53:14.533106+0000","last_peered":"2024-09-13T14:53:14.533106+0000","last_clean":"2024-09-13T14:53:14.533106+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:53:14.533106+0000","last_undegraded":"2024-09-13T14:53:14.533106+0000","last_fullsized":"2024-09-13T14:53:14.533106+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":103,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.533327+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:53:14.533327+0000","last_peered":"2024-09-13T14:53:14.533327+0000","last_clean":"2024-09-13T14:53:14.533327+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:53:14.533327+0000","last_undegraded":"2024-09-13T14:53:14.533327+0000","last_fullsized":"2024-09-13T14:53:14.533327+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"247'8137","reported_seq":17468,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.859550+0000","last_change":"2024-09-13T14:52:52.443442+0000","last_active":"2024-09-13T14:53:14.859550+0000","last_peered":"2024-09-13T14:53:14.859550+0000","last_clean":"2024-09-13T14:53:14.859550+0000","last_became_active":"2024-09-13T14:52:52.421083+0000","last_became_peered":"2024-09-13T14:52:52.421083+0000","last_unstale":"2024-09-13T14:53:14.859550+0000","last_undegraded":"2024-09-13T14:53:14.859550+0000","last_fullsized":"2024-09-13T14:53:14.859550+0000","mapping_epoch":240,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6437,"log_dups_size":0,"ondisk_log_size":6437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:14.887813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169279488,"num_objects":2583,"num_object_clones":0,"num_object_copies":7749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4569,"num_bytes_recovered":287244288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":101,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.533663+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:53:13.533663+0000","last_peered":"2024-09-13T14:53:13.533663+0000","last_clean":"2024-09-13T14:53:13.533663+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:53:13.533663+0000","last_undegraded":"2024-09-13T14:53:13.533663+0000","last_fullsized":"2024-09-13T14:53:13.533663+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":230,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507579+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:53:11.507579+0000","last_peered":"2024-09-13T14:53:11.507579+0000","last_clean":"2024-09-13T14:53:11.507579+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:53:11.507579+0000","last_undegraded":"2024-09-13T14:53:11.507579+0000","last_fullsized":"2024-09-13T14:53:11.507579+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":5,"reported_epoch":247,"state":"activating","last_fresh":"2024-09-13T14:53:14.556639+0000","last_change":"2024-09-13T14:53:14.535151+0000","last_active":"2024-09-13T14:53:14.534980+0000","last_peered":"2024-09-13T14:53:11.509118+0000","last_clean":"2024-09-13T14:53:11.509118+0000","last_became_active":"2024-09-13T14:50:22.037671+0000","last_became_peered":"2024-09-13T14:50:22.037671+0000","last_unstale":"2024-09-13T14:53:14.556639+0000","last_undegraded":"2024-09-13T14:53:14.556639+0000","last_fullsized":"2024-09-13T14:53:14.556639+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":["5","1","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"245'8187","reported_seq":12784,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:12.989444+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:53:12.989444+0000","last_peered":"2024-09-13T14:53:12.989444+0000","last_clean":"2024-09-13T14:53:12.989444+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:53:12.989444+0000","last_undegraded":"2024-09-13T14:53:12.989444+0000","last_fullsized":"2024-09-13T14:53:12.989444+0000","mapping_epoch":226,"log_start":"164'200","ondisk_log_start":"164'200","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":7881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":101,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.533883+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:53:13.533883+0000","last_peered":"2024-09-13T14:53:13.533883+0000","last_clean":"2024-09-13T14:53:13.533883+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:53:13.533883+0000","last_undegraded":"2024-09-13T14:53:13.533883+0000","last_fullsized":"2024-09-13T14:53:13.533883+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":905,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.533917+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:53:14.533917+0000","last_peered":"2024-09-13T14:53:14.533917+0000","last_clean":"2024-09-13T14:53:14.533917+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:53:14.533917+0000","last_undegraded":"2024-09-13T14:53:14.533917+0000","last_fullsized":"2024-09-13T14:53:14.533917+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":324,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-13T14:53:05.494763+0000","last_change":"2024-09-13T14:49:58.520460+0000","last_active":"2024-09-13T14:53:05.494763+0000","last_peered":"2024-09-13T14:53:05.494763+0000","last_clean":"2024-09-13T14:53:05.494763+0000","last_became_active":"2024-09-13T14:49:58.520134+0000","last_became_peered":"2024-09-13T14:49:58.520134+0000","last_unstale":"2024-09-13T14:53:05.494763+0000","last_undegraded":"2024-09-13T14:53:05.494763+0000","last_fullsized":"2024-09-13T14:53:05.494763+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:58.821547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"247'8285","reported_seq":13014,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.699868+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:53:14.699868+0000","last_peered":"2024-09-13T14:53:14.699868+0000","last_clean":"2024-09-13T14:53:14.699868+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:53:14.699868+0000","last_undegraded":"2024-09-13T14:53:14.699868+0000","last_fullsized":"2024-09-13T14:53:14.699868+0000","mapping_epoch":218,"log_start":"174'2245","ondisk_log_start":"174'2245","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171900928,"num_objects":2623,"num_object_clones":0,"num_object_copies":7869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8285,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":357,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.533819+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:53:14.533819+0000","last_peered":"2024-09-13T14:53:14.533819+0000","last_clean":"2024-09-13T14:53:14.533819+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:53:14.533819+0000","last_undegraded":"2024-09-13T14:53:14.533819+0000","last_fullsized":"2024-09-13T14:53:14.533819+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":574,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497957+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:53:11.497957+0000","last_peered":"2024-09-13T14:53:11.497957+0000","last_clean":"2024-09-13T14:53:11.497957+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:53:11.497957+0000","last_undegraded":"2024-09-13T14:53:11.497957+0000","last_fullsized":"2024-09-13T14:53:11.497957+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":351,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497536+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:53:11.497536+0000","last_peered":"2024-09-13T14:53:11.497536+0000","last_clean":"2024-09-13T14:53:11.497536+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:53:11.497536+0000","last_undegraded":"2024-09-13T14:53:11.497536+0000","last_fullsized":"2024-09-13T14:53:11.497536+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"246'7949","reported_seq":14507,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.120227+0000","last_change":"2024-09-13T14:53:05.481759+0000","last_active":"2024-09-13T14:53:14.120227+0000","last_peered":"2024-09-13T14:53:14.120227+0000","last_clean":"2024-09-13T14:53:14.120227+0000","last_became_active":"2024-09-13T14:53:05.465653+0000","last_became_peered":"2024-09-13T14:53:05.465653+0000","last_unstale":"2024-09-13T14:53:14.120227+0000","last_undegraded":"2024-09-13T14:53:14.120227+0000","last_fullsized":"2024-09-13T14:53:14.120227+0000","mapping_epoch":243,"log_start":"169'1400","ondisk_log_start":"169'1400","created":160,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6549,"log_dups_size":0,"ondisk_log_size":6549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:28.220387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167837696,"num_objects":2561,"num_object_clones":0,"num_object_copies":7683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7949,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3214,"num_bytes_recovered":205586432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":375,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.540979+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:53:14.540979+0000","last_peered":"2024-09-13T14:53:14.540979+0000","last_clean":"2024-09-13T14:53:14.540979+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:53:14.540979+0000","last_undegraded":"2024-09-13T14:53:14.540979+0000","last_fullsized":"2024-09-13T14:53:14.540979+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":806,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.536101+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:53:14.536101+0000","last_peered":"2024-09-13T14:53:14.536101+0000","last_clean":"2024-09-13T14:53:14.536101+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:53:14.536101+0000","last_undegraded":"2024-09-13T14:53:14.536101+0000","last_fullsized":"2024-09-13T14:53:14.536101+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":91,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535280+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:53:14.535280+0000","last_peered":"2024-09-13T14:53:14.535280+0000","last_clean":"2024-09-13T14:53:14.535280+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:53:14.535280+0000","last_undegraded":"2024-09-13T14:53:14.535280+0000","last_fullsized":"2024-09-13T14:53:14.535280+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"246'8166","reported_seq":9746,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.851634+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:53:13.851634+0000","last_peered":"2024-09-13T14:53:13.851634+0000","last_clean":"2024-09-13T14:53:13.851634+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:53:13.851634+0000","last_undegraded":"2024-09-13T14:53:13.851634+0000","last_fullsized":"2024-09-13T14:53:13.851634+0000","mapping_epoch":188,"log_start":"164'200","ondisk_log_start":"164'200","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7966,"log_dups_size":0,"ondisk_log_size":7966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176553984,"num_objects":2694,"num_object_clones":0,"num_object_copies":8082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8166,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":346,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497194+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:53:11.497194+0000","last_peered":"2024-09-13T14:53:11.497194+0000","last_clean":"2024-09-13T14:53:11.497194+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:53:11.497194+0000","last_undegraded":"2024-09-13T14:53:11.497194+0000","last_fullsized":"2024-09-13T14:53:11.497194+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":798,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.509859+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:53:11.509859+0000","last_peered":"2024-09-13T14:53:11.509859+0000","last_clean":"2024-09-13T14:53:11.509859+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:53:11.509859+0000","last_undegraded":"2024-09-13T14:53:11.509859+0000","last_fullsized":"2024-09-13T14:53:11.509859+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":233,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.529969+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:53:13.529969+0000","last_peered":"2024-09-13T14:53:13.529969+0000","last_clean":"2024-09-13T14:53:13.529969+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:53:13.529969+0000","last_undegraded":"2024-09-13T14:53:13.529969+0000","last_fullsized":"2024-09-13T14:53:13.529969+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":228,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507507+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:53:11.507507+0000","last_peered":"2024-09-13T14:53:11.507507+0000","last_clean":"2024-09-13T14:53:11.507507+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:53:11.507507+0000","last_undegraded":"2024-09-13T14:53:11.507507+0000","last_fullsized":"2024-09-13T14:53:11.507507+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":276,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507816+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:53:11.507816+0000","last_peered":"2024-09-13T14:53:11.507816+0000","last_clean":"2024-09-13T14:53:11.507816+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:53:11.507816+0000","last_undegraded":"2024-09-13T14:53:11.507816+0000","last_fullsized":"2024-09-13T14:53:11.507816+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":354,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.533363+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:53:14.533363+0000","last_peered":"2024-09-13T14:53:14.533363+0000","last_clean":"2024-09-13T14:53:14.533363+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:53:14.533363+0000","last_undegraded":"2024-09-13T14:53:14.533363+0000","last_fullsized":"2024-09-13T14:53:14.533363+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":195,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.539757+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:53:14.539757+0000","last_peered":"2024-09-13T14:53:14.539757+0000","last_clean":"2024-09-13T14:53:14.539757+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:53:14.539757+0000","last_undegraded":"2024-09-13T14:53:14.539757+0000","last_fullsized":"2024-09-13T14:53:14.539757+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":110,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535093+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:53:14.535093+0000","last_peered":"2024-09-13T14:53:14.535093+0000","last_clean":"2024-09-13T14:53:14.535093+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:53:14.535093+0000","last_undegraded":"2024-09-13T14:53:14.535093+0000","last_fullsized":"2024-09-13T14:53:14.535093+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":331,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.541322+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:53:14.541322+0000","last_peered":"2024-09-13T14:53:14.541322+0000","last_clean":"2024-09-13T14:53:14.541322+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:53:14.541322+0000","last_undegraded":"2024-09-13T14:53:14.541322+0000","last_fullsized":"2024-09-13T14:53:14.541322+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":382,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497316+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:53:11.497316+0000","last_peered":"2024-09-13T14:53:11.497316+0000","last_clean":"2024-09-13T14:53:11.497316+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:53:11.497316+0000","last_undegraded":"2024-09-13T14:53:11.497316+0000","last_fullsized":"2024-09-13T14:53:11.497316+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":382,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.538778+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:53:14.538778+0000","last_peered":"2024-09-13T14:53:14.538778+0000","last_clean":"2024-09-13T14:53:14.538778+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:53:14.538778+0000","last_undegraded":"2024-09-13T14:53:14.538778+0000","last_fullsized":"2024-09-13T14:53:14.538778+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":138,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.509786+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:53:11.509786+0000","last_peered":"2024-09-13T14:53:11.509786+0000","last_clean":"2024-09-13T14:53:11.509786+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:53:11.509786+0000","last_undegraded":"2024-09-13T14:53:11.509786+0000","last_fullsized":"2024-09-13T14:53:11.509786+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":349,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.539395+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:53:14.539395+0000","last_peered":"2024-09-13T14:53:14.539395+0000","last_clean":"2024-09-13T14:53:14.539395+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:53:14.539395+0000","last_undegraded":"2024-09-13T14:53:14.539395+0000","last_fullsized":"2024-09-13T14:53:14.539395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":362,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497397+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:53:11.497397+0000","last_peered":"2024-09-13T14:53:11.497397+0000","last_clean":"2024-09-13T14:53:11.497397+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:53:11.497397+0000","last_undegraded":"2024-09-13T14:53:11.497397+0000","last_fullsized":"2024-09-13T14:53:11.497397+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":335,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.532708+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:53:14.532708+0000","last_peered":"2024-09-13T14:53:14.532708+0000","last_clean":"2024-09-13T14:53:14.532708+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:53:14.532708+0000","last_undegraded":"2024-09-13T14:53:14.532708+0000","last_fullsized":"2024-09-13T14:53:14.532708+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":198,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.539638+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:53:14.539638+0000","last_peered":"2024-09-13T14:53:14.539638+0000","last_clean":"2024-09-13T14:53:14.539638+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:53:14.539638+0000","last_undegraded":"2024-09-13T14:53:14.539638+0000","last_fullsized":"2024-09-13T14:53:14.539638+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":227,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.530357+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:53:13.530357+0000","last_peered":"2024-09-13T14:53:13.530357+0000","last_clean":"2024-09-13T14:53:13.530357+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:53:13.530357+0000","last_undegraded":"2024-09-13T14:53:13.530357+0000","last_fullsized":"2024-09-13T14:53:13.530357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":92,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.536982+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:53:13.536982+0000","last_peered":"2024-09-13T14:53:13.536982+0000","last_clean":"2024-09-13T14:53:13.536982+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:53:13.536982+0000","last_undegraded":"2024-09-13T14:53:13.536982+0000","last_fullsized":"2024-09-13T14:53:13.536982+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":355,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497447+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:53:11.497447+0000","last_peered":"2024-09-13T14:53:11.497447+0000","last_clean":"2024-09-13T14:53:11.497447+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:53:11.497447+0000","last_undegraded":"2024-09-13T14:53:11.497447+0000","last_fullsized":"2024-09-13T14:53:11.497447+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":228,"reported_epoch":246,"state":"peering","last_fresh":"2024-09-13T14:53:13.530177+0000","last_change":"2024-09-13T14:53:13.530337+0000","last_active":"2024-09-13T14:53:13.530177+0000","last_peered":"2024-09-13T14:53:11.507591+0000","last_clean":"2024-09-13T14:53:11.507591+0000","last_became_active":"2024-09-13T14:43:09.681835+0000","last_became_peered":"2024-09-13T14:43:09.681835+0000","last_unstale":"2024-09-13T14:53:13.530177+0000","last_undegraded":"2024-09-13T14:53:13.530177+0000","last_fullsized":"2024-09-13T14:53:13.530177+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:03.169168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":84,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.533947+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:53:13.533947+0000","last_peered":"2024-09-13T14:53:13.533947+0000","last_clean":"2024-09-13T14:53:13.533947+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:53:13.533947+0000","last_undegraded":"2024-09-13T14:53:13.533947+0000","last_fullsized":"2024-09-13T14:53:13.533947+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":225,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.530262+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:53:13.530262+0000","last_peered":"2024-09-13T14:53:13.530262+0000","last_clean":"2024-09-13T14:53:13.530262+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:53:13.530262+0000","last_undegraded":"2024-09-13T14:53:13.530262+0000","last_fullsized":"2024-09-13T14:53:13.530262+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":86,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-13T14:53:13.533781+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:53:13.533781+0000","last_peered":"2024-09-13T14:53:13.533781+0000","last_clean":"2024-09-13T14:53:13.533781+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:53:13.533781+0000","last_undegraded":"2024-09-13T14:53:13.533781+0000","last_fullsized":"2024-09-13T14:53:13.533781+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":352,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.541715+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:53:14.541715+0000","last_peered":"2024-09-13T14:53:14.541715+0000","last_clean":"2024-09-13T14:53:14.541715+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:53:14.541715+0000","last_undegraded":"2024-09-13T14:53:14.541715+0000","last_fullsized":"2024-09-13T14:53:14.541715+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":369,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.542480+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:53:14.542480+0000","last_peered":"2024-09-13T14:53:14.542480+0000","last_clean":"2024-09-13T14:53:14.542480+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:53:14.542480+0000","last_undegraded":"2024-09-13T14:53:14.542480+0000","last_fullsized":"2024-09-13T14:53:14.542480+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":255,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.507048+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:53:11.507048+0000","last_peered":"2024-09-13T14:53:11.507048+0000","last_clean":"2024-09-13T14:53:11.507048+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:53:11.507048+0000","last_undegraded":"2024-09-13T14:53:11.507048+0000","last_fullsized":"2024-09-13T14:53:11.507048+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":418,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-13T14:53:11.497194+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:53:11.497194+0000","last_peered":"2024-09-13T14:53:11.497194+0000","last_clean":"2024-09-13T14:53:11.497194+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:53:11.497194+0000","last_undegraded":"2024-09-13T14:53:11.497194+0000","last_fullsized":"2024-09-13T14:53:11.497194+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2732916760,"num_objects":41702,"num_object_clones":0,"num_object_copies":125106,"num_objects_missing_on_primary":80,"num_objects_missing":80,"num_objects_degraded":0,"num_objects_misplaced":3356,"num_objects_unfound":0,"num_objects_dirty":41702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129846,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43429,"num_bytes_recovered":2727542784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8425385984,"data_stored":8425373768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114718,"ondisk_log_size":114718,"up":48,"acting":46,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495946,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1667948,"kb_used_data":1380064,"kb_used_omap":1,"kb_used_meta":287870,"kb_avail":92077204,"statfs":{"total":95995035648,"available":94287056896,"internally_reserved":0,"allocated":1413185536,"data_stored":1411852718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294779335},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757581,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058668,"kb_used_data":847076,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":92686484,"statfs":{"total":95995035648,"available":94910959616,"internally_reserved":0,"allocated":867405824,"data_stored":866072973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216660423},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019221,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1153468,"kb_used_data":847532,"kb_used_omap":1,"kb_used_meta":305918,"kb_avail":92591684,"statfs":{"total":95995035648,"available":94813884416,"internally_reserved":0,"allocated":867872768,"data_stored":866536183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":313260486},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468106,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2129072,"kb_used_data":1848928,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":91616080,"statfs":{"total":95995035648,"available":93814865920,"internally_reserved":0,"allocated":1893302272,"data_stored":1891973267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":286846116},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,2,2,0,2,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631635,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1784124,"kb_used_data":1431728,"kb_used_omap":8,"kb_used_meta":352375,"kb_avail":91961028,"statfs":{"total":95995035648,"available":94168092672,"internally_reserved":0,"allocated":1466089472,"data_stored":1464749317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":360832991},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795158,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1545392,"kb_used_data":1252712,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":92199760,"statfs":{"total":95995035648,"available":94412554240,"internally_reserved":0,"allocated":1282777088,"data_stored":1281440340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":299688562},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958688,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969000,"kb_used_data":680480,"kb_used_omap":7,"kb_used_meta":288504,"kb_avail":92776152,"statfs":{"total":95995035648,"available":95002779648,"internally_reserved":0,"allocated":696811520,"data_stored":695475157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":295428583},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":688459776,"data_stored":688455704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1266094080,"data_stored":1266090008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1452736512,"data_stored":1452736512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1857945600,"data_stored":1857945600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":857407488,"data_stored":857407488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":858066944,"data_stored":858062872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1392705536,"data_stored":1392705536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":51970048,"data_stored":51970048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:53:19.243 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:53:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=cleanup t=2024-09-13T14:53:18.826556869Z level=info msg="Completed cleanup jobs" duration=17.362043ms 2024-09-13T14:53:19.243 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:53:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=plugins.update.checker t=2024-09-13T14:53:18.945387149Z level=info msg="Update check succeeded" duration=59.479894ms 2024-09-13T14:53:19.243 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 14:53:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana.update.checker t=2024-09-13T14:53:18.96150178Z level=info msg="Update check succeeded" duration=64.830449ms 2024-09-13T14:53:19.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.256 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:19.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:19.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:19.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:19.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:19.519 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:19.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.655 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:19.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:19.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:19.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:19.824 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4127616918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:19.824 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4127616918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:53:19.825 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:19.826 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:19.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:19.907 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:19.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:19.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4127616918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:20.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:20.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:20.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:20.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:20.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:20.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:20.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:20.426 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:20.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:20.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:20.654 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:20.678 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:20.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:20.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:20.891 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:20 smithi154 ceph-mon[102314]: pgmap v734: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 465 op/s; 3356/123246 objects misplaced (2.723%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:53:20.892 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:20 smithi154 ceph-mon[107038]: pgmap v734: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 465 op/s; 3356/123246 objects misplaced (2.723%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:53:20.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:20.919 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:20.919 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:20.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:20.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:20.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:20 smithi172 ceph-mon[100696]: pgmap v734: 89 pgs: 1 activating, 1 peering, 2 active+recovering+undersized+remapped, 85 active+clean; 2.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 465 op/s; 3356/123246 objects misplaced (2.723%); 4.1 MiB/s, 66 objects/s recovering 2024-09-13T14:53:21.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.077 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.202 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:21.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:21.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:21.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:21.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:21.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:21.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:21.721 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:21.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:21.846 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:21.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:21.983 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:21.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:22.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:22.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:22.189 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:53:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:53:21] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-13T14:53:22.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:22.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:22.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:53:22.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:22.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:22.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:22.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:22.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:22.640 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:22.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:22.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:22.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:22.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:22.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:22.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:22 smithi154 ceph-mon[102314]: pgmap v735: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 2722/117690 objects misplaced (2.313%); 8.8 MiB/s, 141 objects/s recovering 2024-09-13T14:53:22.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:22 smithi154 ceph-mon[107038]: pgmap v735: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 2722/117690 objects misplaced (2.313%); 8.8 MiB/s, 141 objects/s recovering 2024-09-13T14:53:22.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:22 smithi172 ceph-mon[100696]: pgmap v735: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 2722/117690 objects misplaced (2.313%); 8.8 MiB/s, 141 objects/s recovering 2024-09-13T14:53:22.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:23.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:23.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:53:23.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:23.169 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:23.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:23.396 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:23.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:23.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:23.545 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:23.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:23.748 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:23.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:23.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:23.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:23.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:24.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:24.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:24.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:24.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:24.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:24.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:24.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:24.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:24.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:24.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:24.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:24.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:53:24.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:24.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:24.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:24.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:24.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:24.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:24.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:24.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:24 smithi154 ceph-mon[102314]: pgmap v736: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 529 op/s; 2722/117690 objects misplaced (2.313%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:53:24.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:24 smithi154 ceph-mon[102314]: osdmap e248: 8 total, 8 up, 7 in 2024-09-13T14:53:24.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:24 smithi154 ceph-mon[107038]: pgmap v736: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 529 op/s; 2722/117690 objects misplaced (2.313%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:53:24.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:24 smithi154 ceph-mon[107038]: osdmap e248: 8 total, 8 up, 7 in 2024-09-13T14:53:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:24 smithi172 ceph-mon[100696]: pgmap v736: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 529 op/s; 2722/117690 objects misplaced (2.313%); 7.4 MiB/s, 119 objects/s recovering 2024-09-13T14:53:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:24 smithi172 ceph-mon[100696]: osdmap e248: 8 total, 8 up, 7 in 2024-09-13T14:53:25.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:25.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:25.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:25.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:25.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:25.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:25.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:25.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:25.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:25.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:25.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:25.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:25.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:25.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:25.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:25.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:25.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:25 smithi154 ceph-mon[102314]: osdmap e249: 8 total, 8 up, 7 in 2024-09-13T14:53:25.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:25 smithi154 ceph-mon[107038]: osdmap e249: 8 total, 8 up, 7 in 2024-09-13T14:53:25.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:25.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:25 smithi172 ceph-mon[100696]: osdmap e249: 8 total, 8 up, 7 in 2024-09-13T14:53:26.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:26.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:26.188 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:26.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:26.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:26.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:26.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:26.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:26.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:26.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:26.621 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:53:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:26.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:26.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:26.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:26.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:26.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:26.945 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:26 smithi154 ceph-mon[102314]: pgmap v739: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 2137/110301 objects misplaced (1.937%); 9.5 MiB/s, 153 objects/s recovering 2024-09-13T14:53:26.945 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:26 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:26.946 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/277636577' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:26.946 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:26 smithi154 ceph-mon[107038]: pgmap v739: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 2137/110301 objects misplaced (1.937%); 9.5 MiB/s, 153 objects/s recovering 2024-09-13T14:53:26.946 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:26 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:26.946 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/277636577' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:26.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:26 smithi172 ceph-mon[100696]: pgmap v739: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 620 op/s; 2137/110301 objects misplaced (1.937%); 9.5 MiB/s, 153 objects/s recovering 2024-09-13T14:53:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:26 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/277636577' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:27.006 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:27.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:27.134 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:27.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:27.259 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:27.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:27.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:27.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:27.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:27.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:27.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:27.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:27.779 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:27.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:27.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:27.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:27.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:27 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:27.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/277636577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:27.973 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:27 smithi154 ceph-mon[102314]: osdmap e250: 8 total, 8 up, 7 in 2024-09-13T14:53:27.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:27 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:27.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/277636577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:27.974 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:27 smithi154 ceph-mon[107038]: osdmap e250: 8 total, 8 up, 7 in 2024-09-13T14:53:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:27 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:53:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/277636577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:27 smithi172 ceph-mon[100696]: osdmap e250: 8 total, 8 up, 7 in 2024-09-13T14:53:28.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:28.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:28.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:28.162 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:28.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:28.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:28.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:28.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:28.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:28.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:28.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:28.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:28.746 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:53:28.748 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:53:28.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:28.798 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:28.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:28.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:28.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:28.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[107038]: pgmap v741: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 2137/110301 objects misplaced (1.937%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:53:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[107038]: from='client.45230 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:28.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:28.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[102314]: pgmap v741: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 2137/110301 objects misplaced (1.937%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:53:28.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[102314]: from='client.45230 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:28.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:28.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:28 smithi172 ceph-mon[100696]: pgmap v741: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 410 op/s; 2137/110301 objects misplaced (1.937%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:53:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:28 smithi172 ceph-mon[100696]: from='client.45230 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:29.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:29.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:29.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:29.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:29.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:29.533 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:29.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:29.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:29.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:29.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:29.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:29.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:29.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:29.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:53:30.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:30.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:30.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:30.209 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":741,"stamp":"2024-09-13T14:53:27.008206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2412451837,"num_objects":36767,"num_object_clones":0,"num_object_copies":110301,"num_objects_missing_on_primary":67,"num_objects_missing":67,"num_objects_degraded":0,"num_objects_misplaced":2137,"num_objects_unfound":0,"num_objects_dirty":36767,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":135932,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44813,"num_bytes_recovered":2809678981,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118863,"ondisk_log_size":118863,"up":267,"acting":266,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":278,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9404344,"kb_used_data":7362992,"kb_used_omap":32,"kb_used_meta":2041183,"kb_avail":646811720,"statfs":{"total":671965249536,"available":662335201280,"internally_reserved":0,"allocated":7539703808,"data_stored":7530304794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2090171952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-161415168,"num_objects":-2463,"num_object_clones":0,"num_object_copies":-7389,"num_objects_missing_on_primary":-13,"num_objects_missing":-13,"num_objects_degraded":0,"num_objects_misplaced":-585,"num_objects_unfound":0,"num_objects_dirty":-2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":38207488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001626"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":380,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581168+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:53:24.581168+0000","last_peered":"2024-09-13T14:53:24.581168+0000","last_clean":"2024-09-13T14:53:24.581168+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:53:24.581168+0000","last_undegraded":"2024-09-13T14:53:24.581168+0000","last_fullsized":"2024-09-13T14:53:24.581168+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":114,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534436+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:53:14.534436+0000","last_peered":"2024-09-13T14:53:14.534436+0000","last_clean":"2024-09-13T14:53:14.534436+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:53:14.534436+0000","last_undegraded":"2024-09-13T14:53:14.534436+0000","last_fullsized":"2024-09-13T14:53:14.534436+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":84,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.597129+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:53:14.597129+0000","last_peered":"2024-09-13T14:53:14.597129+0000","last_clean":"2024-09-13T14:53:14.597129+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:53:14.597129+0000","last_undegraded":"2024-09-13T14:53:14.597129+0000","last_fullsized":"2024-09-13T14:53:14.597129+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":365,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.545979+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:53:14.545979+0000","last_peered":"2024-09-13T14:53:14.545979+0000","last_clean":"2024-09-13T14:53:14.545979+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:53:14.545979+0000","last_undegraded":"2024-09-13T14:53:14.545979+0000","last_fullsized":"2024-09-13T14:53:14.545979+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":365,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.536815+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:53:14.536815+0000","last_peered":"2024-09-13T14:53:14.536815+0000","last_clean":"2024-09-13T14:53:14.536815+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:53:14.536815+0000","last_undegraded":"2024-09-13T14:53:14.536815+0000","last_fullsized":"2024-09-13T14:53:14.536815+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":184,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534129+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:53:14.534129+0000","last_peered":"2024-09-13T14:53:14.534129+0000","last_clean":"2024-09-13T14:53:14.534129+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:53:14.534129+0000","last_undegraded":"2024-09-13T14:53:14.534129+0000","last_fullsized":"2024-09-13T14:53:14.534129+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":95,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.580706+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:53:24.580706+0000","last_peered":"2024-09-13T14:53:24.580706+0000","last_clean":"2024-09-13T14:53:24.580706+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:53:24.580706+0000","last_undegraded":"2024-09-13T14:53:24.580706+0000","last_fullsized":"2024-09-13T14:53:24.580706+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":160,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537932+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:53:14.537932+0000","last_peered":"2024-09-13T14:53:14.537932+0000","last_clean":"2024-09-13T14:53:14.537932+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:53:14.537932+0000","last_undegraded":"2024-09-13T14:53:14.537932+0000","last_fullsized":"2024-09-13T14:53:14.537932+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"248'8578","reported_seq":9420,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.215598+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:53:24.215598+0000","last_peered":"2024-09-13T14:53:24.215598+0000","last_clean":"2024-09-13T14:53:24.215598+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:53:24.215598+0000","last_undegraded":"2024-09-13T14:53:24.215598+0000","last_fullsized":"2024-09-13T14:53:24.215598+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8578,"log_dups_size":0,"ondisk_log_size":8578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151912448,"num_objects":2318,"num_object_clones":0,"num_object_copies":6954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":302,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.584493+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:53:23.584493+0000","last_peered":"2024-09-13T14:53:23.584493+0000","last_clean":"2024-09-13T14:53:23.584493+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:53:23.584493+0000","last_undegraded":"2024-09-13T14:53:23.584493+0000","last_fullsized":"2024-09-13T14:53:23.584493+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":111,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534779+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:53:14.534779+0000","last_peered":"2024-09-13T14:53:14.534779+0000","last_clean":"2024-09-13T14:53:14.534779+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:53:14.534779+0000","last_undegraded":"2024-09-13T14:53:14.534779+0000","last_fullsized":"2024-09-13T14:53:14.534779+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"249'8322","reported_seq":9949,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.934101+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:53:24.934101+0000","last_peered":"2024-09-13T14:53:24.934101+0000","last_clean":"2024-09-13T14:53:24.934101+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:53:24.934101+0000","last_undegraded":"2024-09-13T14:53:24.934101+0000","last_fullsized":"2024-09-13T14:53:24.934101+0000","mapping_epoch":170,"log_start":"173'1900","ondisk_log_start":"173'1900","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147849216,"num_objects":2256,"num_object_clones":0,"num_object_copies":6768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":112,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535196+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:53:14.535196+0000","last_peered":"2024-09-13T14:53:14.535196+0000","last_clean":"2024-09-13T14:53:14.535196+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:53:14.535196+0000","last_undegraded":"2024-09-13T14:53:14.535196+0000","last_fullsized":"2024-09-13T14:53:14.535196+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":264,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534247+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:53:14.534247+0000","last_peered":"2024-09-13T14:53:14.534247+0000","last_clean":"2024-09-13T14:53:14.534247+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:53:14.534247+0000","last_undegraded":"2024-09-13T14:53:14.534247+0000","last_fullsized":"2024-09-13T14:53:14.534247+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"248'8500","reported_seq":10218,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.186131+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:53:24.186131+0000","last_peered":"2024-09-13T14:53:24.186131+0000","last_clean":"2024-09-13T14:53:24.186131+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:53:24.186131+0000","last_undegraded":"2024-09-13T14:53:24.186131+0000","last_fullsized":"2024-09-13T14:53:24.186131+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8500,"log_dups_size":0,"ondisk_log_size":8500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153616384,"num_objects":2344,"num_object_clones":0,"num_object_copies":7032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8500,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":88,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535638+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:53:14.535638+0000","last_peered":"2024-09-13T14:53:14.535638+0000","last_clean":"2024-09-13T14:53:14.535638+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:53:14.535638+0000","last_undegraded":"2024-09-13T14:53:14.535638+0000","last_fullsized":"2024-09-13T14:53:14.535638+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":44,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537791+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:53:14.537791+0000","last_peered":"2024-09-13T14:53:14.537791+0000","last_clean":"2024-09-13T14:53:14.537791+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:53:14.537791+0000","last_undegraded":"2024-09-13T14:53:14.537791+0000","last_fullsized":"2024-09-13T14:53:14.537791+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"247'8244","reported_seq":14355,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:22.981542+0000","last_change":"2024-09-13T14:52:50.434519+0000","last_active":"2024-09-13T14:53:22.981542+0000","last_peered":"2024-09-13T14:53:22.981542+0000","last_clean":"2024-09-13T14:53:22.981542+0000","last_became_active":"2024-09-13T14:52:50.417699+0000","last_became_peered":"2024-09-13T14:52:50.417699+0000","last_unstale":"2024-09-13T14:53:22.981542+0000","last_undegraded":"2024-09-13T14:53:22.981542+0000","last_fullsized":"2024-09-13T14:53:22.981542+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:10.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148373504,"num_objects":2264,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2948,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":100,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535576+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:53:14.535576+0000","last_peered":"2024-09-13T14:53:14.535576+0000","last_clean":"2024-09-13T14:53:14.535576+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:53:14.535576+0000","last_undegraded":"2024-09-13T14:53:14.535576+0000","last_fullsized":"2024-09-13T14:53:14.535576+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":344,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537602+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:53:14.537602+0000","last_peered":"2024-09-13T14:53:14.537602+0000","last_clean":"2024-09-13T14:53:14.537602+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:53:14.537602+0000","last_undegraded":"2024-09-13T14:53:14.537602+0000","last_fullsized":"2024-09-13T14:53:14.537602+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"248'8428","reported_seq":25688,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.147182+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:53:24.147182+0000","last_peered":"2024-09-13T14:53:24.147182+0000","last_clean":"2024-09-13T14:53:24.147182+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:53:24.147182+0000","last_undegraded":"2024-09-13T14:53:24.147182+0000","last_fullsized":"2024-09-13T14:53:24.147182+0000","mapping_epoch":232,"log_start":"172'1800","ondisk_log_start":"172'1800","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6628,"log_dups_size":0,"ondisk_log_size":6628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152436736,"num_objects":2326,"num_object_clones":0,"num_object_copies":6978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":357,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581602+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:53:24.581602+0000","last_peered":"2024-09-13T14:53:24.581602+0000","last_clean":"2024-09-13T14:53:24.581602+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:53:24.581602+0000","last_undegraded":"2024-09-13T14:53:24.581602+0000","last_fullsized":"2024-09-13T14:53:24.581602+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":231,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534397+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:53:14.534397+0000","last_peered":"2024-09-13T14:53:14.534397+0000","last_clean":"2024-09-13T14:53:14.534397+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:53:14.534397+0000","last_undegraded":"2024-09-13T14:53:14.534397+0000","last_fullsized":"2024-09-13T14:53:14.534397+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"247'8312","reported_seq":13308,"reported_epoch":248,"state":"peering","last_fresh":"2024-09-13T14:53:23.555681+0000","last_change":"2024-09-13T14:53:24.299915+0000","last_active":"2024-09-13T14:53:24.299915+0000","last_peered":"2024-09-13T14:53:23.555681+0000","last_clean":"2024-09-13T14:48:36.208446+0000","last_became_active":"2024-09-13T14:50:20.839039+0000","last_became_peered":"2024-09-13T14:50:20.839039+0000","last_unstale":"2024-09-13T14:53:23.555681+0000","last_undegraded":"2024-09-13T14:53:23.555681+0000","last_fullsized":"2024-09-13T14:50:20.783547+0000","mapping_epoch":248,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152567808,"num_objects":2328,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8312,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":147914752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":131,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537969+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:53:14.537969+0000","last_peered":"2024-09-13T14:53:14.537969+0000","last_clean":"2024-09-13T14:53:14.537969+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:53:14.537969+0000","last_undegraded":"2024-09-13T14:53:14.537969+0000","last_fullsized":"2024-09-13T14:53:14.537969+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":228,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535319+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:53:14.535319+0000","last_peered":"2024-09-13T14:53:14.535319+0000","last_clean":"2024-09-13T14:53:14.535319+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:53:14.535319+0000","last_undegraded":"2024-09-13T14:53:14.535319+0000","last_fullsized":"2024-09-13T14:53:14.535319+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"249'8536","reported_seq":13261,"reported_epoch":249,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-13T14:53:24.957781+0000","last_change":"2024-09-13T14:52:50.916272+0000","last_active":"2024-09-13T14:53:24.957781+0000","last_peered":"2024-09-13T14:53:24.957781+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:53:24.957781+0000","last_undegraded":"2024-09-13T14:53:24.957781+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"173'2100","ondisk_log_start":"173'2100","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149815296,"num_objects":2286,"num_object_clones":0,"num_object_copies":6858,"num_objects_missing_on_primary":67,"num_objects_missing":67,"num_objects_degraded":0,"num_objects_misplaced":2137,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":155582464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":2136},{"shards":"3,5","objects":149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":223,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534334+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:53:14.534334+0000","last_peered":"2024-09-13T14:53:14.534334+0000","last_clean":"2024-09-13T14:53:14.534334+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:53:14.534334+0000","last_undegraded":"2024-09-13T14:53:14.534334+0000","last_fullsized":"2024-09-13T14:53:14.534334+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":258,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535252+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:53:14.535252+0000","last_peered":"2024-09-13T14:53:14.535252+0000","last_clean":"2024-09-13T14:53:14.535252+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:53:14.535252+0000","last_undegraded":"2024-09-13T14:53:14.535252+0000","last_fullsized":"2024-09-13T14:53:14.535252+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"248'8242","reported_seq":12678,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.181267+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:53:24.181267+0000","last_peered":"2024-09-13T14:53:24.181267+0000","last_clean":"2024-09-13T14:53:24.181267+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:53:24.181267+0000","last_undegraded":"2024-09-13T14:53:24.181267+0000","last_fullsized":"2024-09-13T14:53:24.181267+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148307992,"num_objects":2264,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8242,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":95,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.580610+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:53:24.580610+0000","last_peered":"2024-09-13T14:53:24.580610+0000","last_clean":"2024-09-13T14:53:24.580610+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:53:24.580610+0000","last_undegraded":"2024-09-13T14:53:24.580610+0000","last_fullsized":"2024-09-13T14:53:24.580610+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":202,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.584615+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:53:23.584615+0000","last_peered":"2024-09-13T14:53:23.584615+0000","last_clean":"2024-09-13T14:53:23.584615+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:53:23.584615+0000","last_undegraded":"2024-09-13T14:53:23.584615+0000","last_fullsized":"2024-09-13T14:53:23.584615+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"248'8445","reported_seq":9702,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.248480+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:53:24.248480+0000","last_peered":"2024-09-13T14:53:24.248480+0000","last_clean":"2024-09-13T14:53:24.248480+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:53:24.248480+0000","last_undegraded":"2024-09-13T14:53:24.248480+0000","last_fullsized":"2024-09-13T14:53:24.248480+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150667264,"num_objects":2299,"num_object_clones":0,"num_object_copies":6897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8445,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":547,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.551698+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:53:14.551698+0000","last_peered":"2024-09-13T14:53:14.551698+0000","last_clean":"2024-09-13T14:53:14.551698+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:53:14.551698+0000","last_undegraded":"2024-09-13T14:53:14.551698+0000","last_fullsized":"2024-09-13T14:53:14.551698+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":815,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535029+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:53:14.535029+0000","last_peered":"2024-09-13T14:53:14.535029+0000","last_clean":"2024-09-13T14:53:14.535029+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:53:14.535029+0000","last_undegraded":"2024-09-13T14:53:14.535029+0000","last_fullsized":"2024-09-13T14:53:14.535029+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":315,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581933+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:53:24.581933+0000","last_peered":"2024-09-13T14:53:24.581933+0000","last_clean":"2024-09-13T14:53:24.581933+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:53:24.581933+0000","last_undegraded":"2024-09-13T14:53:24.581933+0000","last_fullsized":"2024-09-13T14:53:24.581933+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":44,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.538250+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:53:14.538250+0000","last_peered":"2024-09-13T14:53:14.538250+0000","last_clean":"2024-09-13T14:53:14.538250+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:53:14.538250+0000","last_undegraded":"2024-09-13T14:53:14.538250+0000","last_fullsized":"2024-09-13T14:53:14.538250+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"249'8491","reported_seq":13232,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.643492+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:53:24.643492+0000","last_peered":"2024-09-13T14:53:24.643492+0000","last_clean":"2024-09-13T14:53:24.643492+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:53:24.643492+0000","last_undegraded":"2024-09-13T14:53:24.643492+0000","last_fullsized":"2024-09-13T14:53:24.643492+0000","mapping_epoch":223,"log_start":"173'2118","ondisk_log_start":"173'2118","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6373,"log_dups_size":0,"ondisk_log_size":6373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154468352,"num_objects":2357,"num_object_clones":0,"num_object_copies":7071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8491,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":396,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.642270+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:53:24.642270+0000","last_peered":"2024-09-13T14:53:24.642270+0000","last_clean":"2024-09-13T14:53:24.642270+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:53:24.642270+0000","last_undegraded":"2024-09-13T14:53:24.642270+0000","last_fullsized":"2024-09-13T14:53:24.642270+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":359,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.536240+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:53:14.536240+0000","last_peered":"2024-09-13T14:53:14.536240+0000","last_clean":"2024-09-13T14:53:14.536240+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:53:14.536240+0000","last_undegraded":"2024-09-13T14:53:14.536240+0000","last_fullsized":"2024-09-13T14:53:14.536240+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.536987+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:53:14.536987+0000","last_peered":"2024-09-13T14:53:14.536987+0000","last_clean":"2024-09-13T14:53:14.536987+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:53:14.536987+0000","last_undegraded":"2024-09-13T14:53:14.536987+0000","last_fullsized":"2024-09-13T14:53:14.536987+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"248'8405","reported_seq":18729,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.841745+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:53:23.841745+0000","last_peered":"2024-09-13T14:53:23.841745+0000","last_clean":"2024-09-13T14:53:23.841745+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:53:23.841745+0000","last_undegraded":"2024-09-13T14:53:23.841745+0000","last_fullsized":"2024-09-13T14:53:23.841745+0000","mapping_epoch":235,"log_start":"164'700","ondisk_log_start":"164'700","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7705,"log_dups_size":0,"ondisk_log_size":7705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":6573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8405,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":897,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.584507+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:53:23.584507+0000","last_peered":"2024-09-13T14:53:23.584507+0000","last_clean":"2024-09-13T14:53:23.584507+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:53:23.584507+0000","last_undegraded":"2024-09-13T14:53:23.584507+0000","last_fullsized":"2024-09-13T14:53:23.584507+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":354,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.642552+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:53:24.642552+0000","last_peered":"2024-09-13T14:53:24.642552+0000","last_clean":"2024-09-13T14:53:24.642552+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:53:24.642552+0000","last_undegraded":"2024-09-13T14:53:24.642552+0000","last_fullsized":"2024-09-13T14:53:24.642552+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":107,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.660464+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:53:24.660464+0000","last_peered":"2024-09-13T14:53:24.660464+0000","last_clean":"2024-09-13T14:53:24.660464+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:53:24.660464+0000","last_undegraded":"2024-09-13T14:53:24.660464+0000","last_fullsized":"2024-09-13T14:53:24.660464+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"249'8467","reported_seq":17802,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.926687+0000","last_change":"2024-09-13T14:52:52.443442+0000","last_active":"2024-09-13T14:53:24.926687+0000","last_peered":"2024-09-13T14:53:24.926687+0000","last_clean":"2024-09-13T14:53:24.926687+0000","last_became_active":"2024-09-13T14:52:52.421083+0000","last_became_peered":"2024-09-13T14:52:52.421083+0000","last_unstale":"2024-09-13T14:53:24.926687+0000","last_undegraded":"2024-09-13T14:53:24.926687+0000","last_fullsized":"2024-09-13T14:53:24.926687+0000","mapping_epoch":240,"log_start":"173'2000","ondisk_log_start":"173'2000","created":160,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6467,"log_dups_size":0,"ondisk_log_size":6467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:14.887813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652608,"num_objects":2253,"num_object_clones":0,"num_object_copies":6759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4569,"num_bytes_recovered":287244288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":102,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535055+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:53:14.535055+0000","last_peered":"2024-09-13T14:53:14.535055+0000","last_clean":"2024-09-13T14:53:14.535055+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:53:14.535055+0000","last_undegraded":"2024-09-13T14:53:14.535055+0000","last_fullsized":"2024-09-13T14:53:14.535055+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":233,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:19.201910+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:53:19.201910+0000","last_peered":"2024-09-13T14:53:19.201910+0000","last_clean":"2024-09-13T14:53:19.201910+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:53:19.201910+0000","last_undegraded":"2024-09-13T14:53:19.201910+0000","last_fullsized":"2024-09-13T14:53:19.201910+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.599585+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:53:14.599585+0000","last_peered":"2024-09-13T14:53:14.599585+0000","last_clean":"2024-09-13T14:53:14.599585+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:53:14.599585+0000","last_undegraded":"2024-09-13T14:53:14.599585+0000","last_fullsized":"2024-09-13T14:53:14.599585+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"247'8529","reported_seq":13130,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:22.986528+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:53:22.986528+0000","last_peered":"2024-09-13T14:53:22.986528+0000","last_clean":"2024-09-13T14:53:22.986528+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:53:22.986528+0000","last_undegraded":"2024-09-13T14:53:22.986528+0000","last_fullsized":"2024-09-13T14:53:22.986528+0000","mapping_epoch":226,"log_start":"164'200","ondisk_log_start":"164'200","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8329,"log_dups_size":0,"ondisk_log_size":8329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149749760,"num_objects":2285,"num_object_clones":0,"num_object_copies":6855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8529,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":102,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.535467+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:53:14.535467+0000","last_peered":"2024-09-13T14:53:14.535467+0000","last_clean":"2024-09-13T14:53:14.535467+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:53:14.535467+0000","last_undegraded":"2024-09-13T14:53:14.535467+0000","last_fullsized":"2024-09-13T14:53:14.535467+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":909,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.644122+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:53:24.644122+0000","last_peered":"2024-09-13T14:53:24.644122+0000","last_clean":"2024-09-13T14:53:24.644122+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:53:24.644122+0000","last_undegraded":"2024-09-13T14:53:24.644122+0000","last_fullsized":"2024-09-13T14:53:24.644122+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.555030+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:53:14.555030+0000","last_peered":"2024-09-13T14:53:14.555030+0000","last_clean":"2024-09-13T14:53:14.555030+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:53:14.555030+0000","last_undegraded":"2024-09-13T14:53:14.555030+0000","last_fullsized":"2024-09-13T14:53:14.555030+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"249'8546","reported_seq":13279,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.728285+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:53:24.728285+0000","last_peered":"2024-09-13T14:53:24.728285+0000","last_clean":"2024-09-13T14:53:24.728285+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:53:24.728285+0000","last_undegraded":"2024-09-13T14:53:24.728285+0000","last_fullsized":"2024-09-13T14:53:24.728285+0000","mapping_epoch":218,"log_start":"174'2245","ondisk_log_start":"174'2245","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154796032,"num_objects":2362,"num_object_clones":0,"num_object_copies":7086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":361,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.643628+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:53:24.643628+0000","last_peered":"2024-09-13T14:53:24.643628+0000","last_clean":"2024-09-13T14:53:24.643628+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:53:24.643628+0000","last_undegraded":"2024-09-13T14:53:24.643628+0000","last_fullsized":"2024-09-13T14:53:24.643628+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":578,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.544630+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:53:14.544630+0000","last_peered":"2024-09-13T14:53:14.544630+0000","last_clean":"2024-09-13T14:53:14.544630+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:53:14.544630+0000","last_undegraded":"2024-09-13T14:53:14.544630+0000","last_fullsized":"2024-09-13T14:53:14.544630+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":355,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537527+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:53:14.537527+0000","last_peered":"2024-09-13T14:53:14.537527+0000","last_clean":"2024-09-13T14:53:14.537527+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:53:14.537527+0000","last_undegraded":"2024-09-13T14:53:14.537527+0000","last_fullsized":"2024-09-13T14:53:14.537527+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"248'8285","reported_seq":14845,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.135645+0000","last_change":"2024-09-13T14:53:05.481759+0000","last_active":"2024-09-13T14:53:24.135645+0000","last_peered":"2024-09-13T14:53:24.135645+0000","last_clean":"2024-09-13T14:53:24.135645+0000","last_became_active":"2024-09-13T14:53:05.465653+0000","last_became_peered":"2024-09-13T14:53:05.465653+0000","last_unstale":"2024-09-13T14:53:24.135645+0000","last_undegraded":"2024-09-13T14:53:24.135645+0000","last_fullsized":"2024-09-13T14:53:24.135645+0000","mapping_epoch":243,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:28.220387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145817600,"num_objects":2225,"num_object_clones":0,"num_object_copies":6675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8285,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3214,"num_bytes_recovered":205586432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":379,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581282+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:53:24.581282+0000","last_peered":"2024-09-13T14:53:24.581282+0000","last_clean":"2024-09-13T14:53:24.581282+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:53:24.581282+0000","last_undegraded":"2024-09-13T14:53:24.581282+0000","last_fullsized":"2024-09-13T14:53:24.581282+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":810,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.645961+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:53:24.645961+0000","last_peered":"2024-09-13T14:53:24.645961+0000","last_clean":"2024-09-13T14:53:24.645961+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:53:24.645961+0000","last_undegraded":"2024-09-13T14:53:24.645961+0000","last_fullsized":"2024-09-13T14:53:24.645961+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":95,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.580466+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:53:24.580466+0000","last_peered":"2024-09-13T14:53:24.580466+0000","last_clean":"2024-09-13T14:53:24.580466+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:53:24.580466+0000","last_undegraded":"2024-09-13T14:53:24.580466+0000","last_fullsized":"2024-09-13T14:53:24.580466+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"248'8479","reported_seq":10061,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.860422+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:53:23.860422+0000","last_peered":"2024-09-13T14:53:23.860422+0000","last_clean":"2024-09-13T14:53:23.860422+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:53:23.860422+0000","last_undegraded":"2024-09-13T14:53:23.860422+0000","last_fullsized":"2024-09-13T14:53:23.860422+0000","mapping_epoch":188,"log_start":"164'500","ondisk_log_start":"164'500","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7979,"log_dups_size":0,"ondisk_log_size":7979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156041216,"num_objects":2381,"num_object_clones":0,"num_object_copies":7143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":350,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.536715+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:53:14.536715+0000","last_peered":"2024-09-13T14:53:14.536715+0000","last_clean":"2024-09-13T14:53:14.536715+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:53:14.536715+0000","last_undegraded":"2024-09-13T14:53:14.536715+0000","last_fullsized":"2024-09-13T14:53:14.536715+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":800,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.538236+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:53:14.538236+0000","last_peered":"2024-09-13T14:53:14.538236+0000","last_clean":"2024-09-13T14:53:14.538236+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:53:14.538236+0000","last_undegraded":"2024-09-13T14:53:14.538236+0000","last_fullsized":"2024-09-13T14:53:14.538236+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":235,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534154+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:53:14.534154+0000","last_peered":"2024-09-13T14:53:14.534154+0000","last_clean":"2024-09-13T14:53:14.534154+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:53:14.534154+0000","last_undegraded":"2024-09-13T14:53:14.534154+0000","last_fullsized":"2024-09-13T14:53:14.534154+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":230,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534911+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:53:14.534911+0000","last_peered":"2024-09-13T14:53:14.534911+0000","last_clean":"2024-09-13T14:53:14.534911+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:53:14.534911+0000","last_undegraded":"2024-09-13T14:53:14.534911+0000","last_fullsized":"2024-09-13T14:53:14.534911+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":278,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534488+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:53:14.534488+0000","last_peered":"2024-09-13T14:53:14.534488+0000","last_clean":"2024-09-13T14:53:14.534488+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:53:14.534488+0000","last_undegraded":"2024-09-13T14:53:14.534488+0000","last_fullsized":"2024-09-13T14:53:14.534488+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":358,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.642997+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:53:24.642997+0000","last_peered":"2024-09-13T14:53:24.642997+0000","last_clean":"2024-09-13T14:53:24.642997+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:53:24.642997+0000","last_undegraded":"2024-09-13T14:53:24.642997+0000","last_fullsized":"2024-09-13T14:53:24.642997+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":199,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581655+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:53:24.581655+0000","last_peered":"2024-09-13T14:53:24.581655+0000","last_clean":"2024-09-13T14:53:24.581655+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:53:24.581655+0000","last_undegraded":"2024-09-13T14:53:24.581655+0000","last_fullsized":"2024-09-13T14:53:24.581655+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":111,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.597148+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:53:14.597148+0000","last_peered":"2024-09-13T14:53:14.597148+0000","last_clean":"2024-09-13T14:53:14.597148+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:53:14.597148+0000","last_undegraded":"2024-09-13T14:53:14.597148+0000","last_fullsized":"2024-09-13T14:53:14.597148+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":336,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.580757+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:53:24.580757+0000","last_peered":"2024-09-13T14:53:24.580757+0000","last_clean":"2024-09-13T14:53:24.580757+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:53:24.580757+0000","last_undegraded":"2024-09-13T14:53:24.580757+0000","last_fullsized":"2024-09-13T14:53:24.580757+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":386,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537074+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:53:14.537074+0000","last_peered":"2024-09-13T14:53:14.537074+0000","last_clean":"2024-09-13T14:53:14.537074+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:53:14.537074+0000","last_undegraded":"2024-09-13T14:53:14.537074+0000","last_fullsized":"2024-09-13T14:53:14.537074+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":387,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.580835+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:53:24.580835+0000","last_peered":"2024-09-13T14:53:24.580835+0000","last_clean":"2024-09-13T14:53:24.580835+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:53:24.580835+0000","last_undegraded":"2024-09-13T14:53:24.580835+0000","last_fullsized":"2024-09-13T14:53:24.580835+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":141,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:18.887716+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:53:18.887716+0000","last_peered":"2024-09-13T14:53:18.887716+0000","last_clean":"2024-09-13T14:53:18.887716+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:53:18.887716+0000","last_undegraded":"2024-09-13T14:53:18.887716+0000","last_fullsized":"2024-09-13T14:53:18.887716+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":353,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.580977+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:53:24.580977+0000","last_peered":"2024-09-13T14:53:24.580977+0000","last_clean":"2024-09-13T14:53:24.580977+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:53:24.580977+0000","last_undegraded":"2024-09-13T14:53:24.580977+0000","last_fullsized":"2024-09-13T14:53:24.580977+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":366,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537698+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:53:14.537698+0000","last_peered":"2024-09-13T14:53:14.537698+0000","last_clean":"2024-09-13T14:53:14.537698+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:53:14.537698+0000","last_undegraded":"2024-09-13T14:53:14.537698+0000","last_fullsized":"2024-09-13T14:53:14.537698+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":339,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.642551+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:53:24.642551+0000","last_peered":"2024-09-13T14:53:24.642551+0000","last_clean":"2024-09-13T14:53:24.642551+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:53:24.642551+0000","last_undegraded":"2024-09-13T14:53:24.642551+0000","last_fullsized":"2024-09-13T14:53:24.642551+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":202,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581507+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:53:24.581507+0000","last_peered":"2024-09-13T14:53:24.581507+0000","last_clean":"2024-09-13T14:53:24.581507+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:53:24.581507+0000","last_undegraded":"2024-09-13T14:53:24.581507+0000","last_fullsized":"2024-09-13T14:53:24.581507+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":229,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534639+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:53:14.534639+0000","last_peered":"2024-09-13T14:53:14.534639+0000","last_clean":"2024-09-13T14:53:14.534639+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:53:14.534639+0000","last_undegraded":"2024-09-13T14:53:14.534639+0000","last_fullsized":"2024-09-13T14:53:14.534639+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":96,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.583909+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:53:23.583909+0000","last_peered":"2024-09-13T14:53:23.583909+0000","last_clean":"2024-09-13T14:53:23.583909+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:53:23.583909+0000","last_undegraded":"2024-09-13T14:53:23.583909+0000","last_fullsized":"2024-09-13T14:53:23.583909+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":359,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.537428+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:53:14.537428+0000","last_peered":"2024-09-13T14:53:14.537428+0000","last_clean":"2024-09-13T14:53:14.537428+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:53:14.537428+0000","last_undegraded":"2024-09-13T14:53:14.537428+0000","last_fullsized":"2024-09-13T14:53:14.537428+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":239,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:19.201884+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:53:19.201884+0000","last_peered":"2024-09-13T14:53:19.201884+0000","last_clean":"2024-09-13T14:53:19.201884+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:53:19.201884+0000","last_undegraded":"2024-09-13T14:53:19.201884+0000","last_fullsized":"2024-09-13T14:53:19.201884+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":86,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.590164+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:53:23.590164+0000","last_peered":"2024-09-13T14:53:23.590164+0000","last_clean":"2024-09-13T14:53:23.590164+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:53:23.590164+0000","last_undegraded":"2024-09-13T14:53:23.590164+0000","last_fullsized":"2024-09-13T14:53:23.590164+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":227,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534559+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:53:14.534559+0000","last_peered":"2024-09-13T14:53:14.534559+0000","last_clean":"2024-09-13T14:53:14.534559+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:53:14.534559+0000","last_undegraded":"2024-09-13T14:53:14.534559+0000","last_fullsized":"2024-09-13T14:53:14.534559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":88,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-13T14:53:23.590552+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:53:23.590552+0000","last_peered":"2024-09-13T14:53:23.590552+0000","last_clean":"2024-09-13T14:53:23.590552+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:53:23.590552+0000","last_undegraded":"2024-09-13T14:53:23.590552+0000","last_fullsized":"2024-09-13T14:53:23.590552+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":356,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581234+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:53:24.581234+0000","last_peered":"2024-09-13T14:53:24.581234+0000","last_clean":"2024-09-13T14:53:24.581234+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:53:24.581234+0000","last_undegraded":"2024-09-13T14:53:24.581234+0000","last_fullsized":"2024-09-13T14:53:24.581234+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":373,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-13T14:53:24.581905+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:53:24.581905+0000","last_peered":"2024-09-13T14:53:24.581905+0000","last_clean":"2024-09-13T14:53:24.581905+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:53:24.581905+0000","last_undegraded":"2024-09-13T14:53:24.581905+0000","last_fullsized":"2024-09-13T14:53:24.581905+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":257,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.534162+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:53:14.534162+0000","last_peered":"2024-09-13T14:53:14.534162+0000","last_clean":"2024-09-13T14:53:14.534162+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:53:14.534162+0000","last_undegraded":"2024-09-13T14:53:14.534162+0000","last_fullsized":"2024-09-13T14:53:14.534162+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":422,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-13T14:53:14.536982+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:53:14.536982+0000","last_peered":"2024-09-13T14:53:14.536982+0000","last_clean":"2024-09-13T14:53:14.536982+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:53:14.536982+0000","last_undegraded":"2024-09-13T14:53:14.536982+0000","last_fullsized":"2024-09-13T14:53:14.536982+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2407661592,"num_objects":36739,"num_object_clones":0,"num_object_copies":110217,"num_objects_missing_on_primary":67,"num_objects_missing":67,"num_objects_degraded":0,"num_objects_misplaced":2137,"num_objects_unfound":0,"num_objects_dirty":36739,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134809,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44657,"num_bytes_recovered":2807300096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7529639936,"data_stored":7529627720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117681,"ondisk_log_size":117681,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495948,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491840,"kb_used_data":1212152,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92253312,"statfs":{"total":95995035648,"available":94467391488,"internally_reserved":0,"allocated":1241243648,"data_stored":1239903111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":286390727},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757583,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963784,"kb_used_data":743996,"kb_used_omap":1,"kb_used_meta":219774,"kb_avail":92781368,"statfs":{"total":95995035648,"available":95008120832,"internally_reserved":0,"allocated":761851904,"data_stored":760511247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":225049031},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019223,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1051776,"kb_used_data":741752,"kb_used_omap":1,"kb_used_meta":310014,"kb_avail":92693376,"statfs":{"total":95995035648,"available":94918017024,"internally_reserved":0,"allocated":759554048,"data_stored":758213572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":317454790},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468108,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1902868,"kb_used_data":1606020,"kb_used_omap":4,"kb_used_meta":296763,"kb_avail":91842284,"statfs":{"total":95995035648,"available":94046498816,"internally_reserved":0,"allocated":1644564480,"data_stored":1643223733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":303885476},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631637,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680656,"kb_used_data":1328264,"kb_used_omap":8,"kb_used_meta":352375,"kb_avail":92064496,"statfs":{"total":95995035648,"available":94274043904,"internally_reserved":0,"allocated":1360142336,"data_stored":1358794375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":360832991},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795160,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1435788,"kb_used_data":1133376,"kb_used_omap":7,"kb_used_meta":302392,"kb_avail":92309364,"statfs":{"total":95995035648,"available":94524788736,"internally_reserved":0,"allocated":1160577024,"data_stored":1159232470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":309650034},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958690,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":877632,"kb_used_data":597432,"kb_used_omap":7,"kb_used_meta":280184,"kb_avail":92867520,"statfs":{"total":95995035648,"available":95096340480,"internally_reserved":0,"allocated":611770368,"data_stored":610426286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":286908903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":603197440,"data_stored":603193368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1152323584,"data_stored":1152319512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1351548928,"data_stored":1351548928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1635909632,"data_stored":1635909632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":752353280,"data_stored":752353280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":756551680,"data_stored":756547608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1225785344,"data_stored":1225785344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":51970048,"data_stored":51970048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:53:30.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:53:30.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:30.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:30.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:30.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:30.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:30.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:30.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:30.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:30.752 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:30 smithi154 ceph-mon[102314]: pgmap v742: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 509 op/s; 2137/108519 objects misplaced (1.969%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:53:30.752 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:30 smithi154 ceph-mon[107038]: pgmap v742: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 509 op/s; 2137/108519 objects misplaced (1.969%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:53:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:30.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:30.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:30.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:30.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:30 smithi172 ceph-mon[100696]: pgmap v742: 89 pgs: 1 active+recovering+undersized+remapped, 1 peering, 87 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 509 op/s; 2137/108519 objects misplaced (1.969%); 6.1 MiB/s, 98 objects/s recovering 2024-09-13T14:53:31.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:31.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:31.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:31.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:31.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:31.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:31.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:31.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:31.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:31.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:31.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:31.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.721 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:31.723 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:31.723 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:31.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:31.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:31.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:31.948 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:31.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:31.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:31.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:31 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 31/102441 objects degraded (0.030%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:31.972 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:53:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:53:31] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:53:31.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:31 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 31/102441 objects degraded (0.030%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:31.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:31 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 31/102441 objects degraded (0.030%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:32.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:32.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:32.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:32.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:32.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:32.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:32.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:32.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:32.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:32.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:32.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:53:32.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:32.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:32.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:32.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:32.913 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:32 smithi154 ceph-mon[102314]: pgmap v743: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 31/102441 objects degraded (0.030%); 2106/102441 objects misplaced (2.056%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:53:32.914 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:32 smithi154 ceph-mon[107038]: pgmap v743: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 31/102441 objects degraded (0.030%); 2106/102441 objects misplaced (2.056%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:53:32.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:32 smithi172 ceph-mon[100696]: pgmap v743: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 31/102441 objects degraded (0.030%); 2106/102441 objects misplaced (2.056%); 4.9 MiB/s, 79 objects/s recovering 2024-09-13T14:53:32.996 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:32.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:33.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:33.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:33.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:33.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:33.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:33.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:33.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:33.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:33.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:33.632 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:33.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:33.763 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:33.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:33.903 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:33.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:34.027 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:34.050 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:34.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:34.173 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:34.202 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:53:34.202 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:34.229 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:53:34.229 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:34.259 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:53:34.260 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:34.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:34.284 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:53:34.284 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:34.298 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:34.309 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:53:34.310 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:34.337 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:53:34.337 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:34.365 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:53:34.365 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:34.383 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:53:34.383 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:34.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:34.421 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:34.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:34.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:34.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:34.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:34.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:34.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:34.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:34.931 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:34.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:34.952 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:34 smithi154 ceph-mon[102314]: pgmap v744: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 453 op/s; 31/102441 objects degraded (0.030%); 2106/102441 objects misplaced (2.056%); 30 KiB/s, 1 objects/s recovering 2024-09-13T14:53:34.952 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:34 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/19098257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:34.952 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:34 smithi154 ceph-mon[107038]: pgmap v744: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 453 op/s; 31/102441 objects degraded (0.030%); 2106/102441 objects misplaced (2.056%); 30 KiB/s, 1 objects/s recovering 2024-09-13T14:53:34.952 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:34 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/19098257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:34 smithi172 ceph-mon[100696]: pgmap v744: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 453 op/s; 31/102441 objects degraded (0.030%); 2106/102441 objects misplaced (2.056%); 30 KiB/s, 1 objects/s recovering 2024-09-13T14:53:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:34 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/19098257' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:35.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:35.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:35.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:35.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:35.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:35.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:35.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:35.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:35.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:35.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:35.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:35.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:35.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:35.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:35.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:35.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:35.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:36.064 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:36.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:36.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:36.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:36.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:36.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:36.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:36.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:36.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:36.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:36.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:36.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:36.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:36.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:36.843 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:36 smithi154 ceph-mon[102314]: pgmap v745: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 31/95706 objects degraded (0.032%); 2106/95706 objects misplaced (2.200%) 2024-09-13T14:53:36.843 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/455509517' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:36.844 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:36 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:36.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:36 smithi154 ceph-mon[107038]: pgmap v745: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 31/95706 objects degraded (0.032%); 2106/95706 objects misplaced (2.200%) 2024-09-13T14:53:36.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/455509517' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:36.844 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:36 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:36.844 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:53:36.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:36.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:36.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:36.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:37.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:37.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:37.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:36 smithi172 ceph-mon[100696]: pgmap v745: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 31/95706 objects degraded (0.032%); 2106/95706 objects misplaced (2.200%) 2024-09-13T14:53:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/455509517' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:36 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:53:37.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:37.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:37.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:37.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:37.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:37.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:37.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:37.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:37.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:37.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:37.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:37.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:37.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:37.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:38.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:38.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:38.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:38.174 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:37 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:38.174 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:37 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:38.175 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:37 smithi154 ceph-mon[102314]: osdmap e251: 8 total, 8 up, 7 in 2024-09-13T14:53:38.175 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:37 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:38.175 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:37 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:38.175 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:37 smithi154 ceph-mon[107038]: osdmap e251: 8 total, 8 up, 7 in 2024-09-13T14:53:38.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:37 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:37 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:53:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:37 smithi172 ceph-mon[100696]: osdmap e251: 8 total, 8 up, 7 in 2024-09-13T14:53:38.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:38.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:38.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:38.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:38.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:38.591 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:38.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:38.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:38.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:38.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:38.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:38.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:38.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:39.053 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:38 smithi154 ceph-mon[102314]: pgmap v747: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 192/94227 objects degraded (0.204%); 1945/94227 objects misplaced (2.064%) 2024-09-13T14:53:39.053 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:38 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 192/94227 objects degraded (0.204%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:39.053 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:38 smithi154 ceph-mon[107038]: pgmap v747: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 192/94227 objects degraded (0.204%); 1945/94227 objects misplaced (2.064%) 2024-09-13T14:53:39.053 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:38 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 192/94227 objects degraded (0.204%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:39.075 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:39.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:39.086 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:39.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:39.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:38 smithi172 ceph-mon[100696]: pgmap v747: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 192/94227 objects degraded (0.204%); 1945/94227 objects misplaced (2.064%) 2024-09-13T14:53:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:38 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 192/94227 objects degraded (0.204%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:39.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:39.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:39.379 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:39.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:39.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:53:39.510 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 192/94227 objects degraded (0.204%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1035,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":251,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238500,"num_in_osds":7,"osd_in_since":1726238900,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":88},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":31409,"data_bytes":2061309949,"bytes_used":8530665472,"bytes_avail":663434584064,"bytes_total":671965249536,"degraded_objects":192,"degraded_total":94227,"degraded_ratio":0.0020376325257091916,"misplaced_objects":1945,"misplaced_total":94227,"misplaced_ratio":0.020641641992210302,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":535},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi154.daekzd","status":"up:active","gid":24539}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"}},"servicemap":{"epoch":25,"modified":"2024-09-13T14:50:26.951855+0000","services":{}},"progress_events":{}} 2024-09-13T14:53:39.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:39.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:39.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:39.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:39.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:39.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:39.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:39.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:39.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:39 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2296804409' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:53:39.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:39 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2296804409' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:53:40.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:40.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:40.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:40.141 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:39 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2296804409' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:53:40.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:40.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:40.279 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:40.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:40.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:40.403 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:40.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:40.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:40.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:40.655 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:40.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:40.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:40.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:53:40.881 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:40.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:40.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:40.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:41.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:41.212 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:40 smithi154 ceph-mon[102314]: pgmap v748: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 540 op/s; 192/92307 objects degraded (0.208%); 1945/92307 objects misplaced (2.107%) 2024-09-13T14:53:41.213 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:40 smithi154 ceph-mon[107038]: pgmap v748: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 540 op/s; 192/92307 objects degraded (0.208%); 1945/92307 objects misplaced (2.107%) 2024-09-13T14:53:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:40 smithi172 ceph-mon[100696]: pgmap v748: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 540 op/s; 192/92307 objects degraded (0.208%); 1945/92307 objects misplaced (2.107%) 2024-09-13T14:53:41.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:41.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:41.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:41.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:41.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:41.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:41.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:41.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:41.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:41.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:41.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:41.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:41.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:41.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:42.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:42.084 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:53:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:53:41] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:53:42.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:42.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:42.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:42.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:42.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:42.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:42.473 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:42.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:42.599 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:42.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:42.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:42.797 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T14:53:42.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:53:42.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:42.864 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:42.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:42.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:42 smithi154 ceph-mon[102314]: pgmap v749: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 365/86172 objects degraded (0.424%); 1772/86172 objects misplaced (2.056%) 2024-09-13T14:53:42.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:42 smithi154 ceph-mon[107038]: pgmap v749: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 365/86172 objects degraded (0.424%); 1772/86172 objects misplaced (2.056%) 2024-09-13T14:53:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:42.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:42.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:43.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:43.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:43.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:43.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:43.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:42 smithi172 ceph-mon[100696]: pgmap v749: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 365/86172 objects degraded (0.424%); 1772/86172 objects misplaced (2.056%) 2024-09-13T14:53:43.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:43.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86887 6.c 6:302ee6f9:::benchmark_data_smithi154_175640_object1111:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e211)", 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 200.628067289, 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8528283699999992, 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.422 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86887 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789253+0000", 2024-09-13T14:53:43.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789254+0000", 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6649999999999999e-06 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789262+0000", 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.542e-06 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789278+0000", 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6351e-05 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789288+0000", 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0358e-05 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789368+0000", 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9726000000000002e-05 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.789377+0000", 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1609999999999992e-06 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.796583+0000", 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072051260000000001 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.796594+0000", 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1331e-05 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.513336+0000", 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7167424520000001 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.519370+0000", 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060334999999999998 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.527233+0000", 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078633379999999992 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.538182+0000", 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010949257 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.547496+0000", 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093131220000000001 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.631104+0000", 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.083608314000000003 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.633486+0000", 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002382463 2024-09-13T14:53:43.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.642081+0000", 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085946640000000001 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:103741 6.c 6:3fa73a7d:::benchmark_data_smithi154_175640_object17965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:44.677700+0000", 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 118.73962005999999, 2024-09-13T14:53:43.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1458053749999999, 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 103741 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.677700+0000", 2024-09-13T14:53:43.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.677700+0000", 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.677702+0000", 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.677709+0000", 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0099999999999998e-06 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.677723+0000", 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4463e-05 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.677732+0000", 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8759999999999994e-06 2024-09-13T14:53:43.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.628337+0000", 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.95060455799999999 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.639315+0000", 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010978058000000001 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.685260+0000", 2024-09-13T14:53:43.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.045945332999999998 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.691009+0000", 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057491210000000003 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.698788+0000", 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077785579999999997 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.751435+0000", 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052647571999999997 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.807480+0000", 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.056044255000000001 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.809211+0000", 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017311519999999999 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.812914+0000", 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037030240000000001 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.823505+0000", 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010591425 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:103747 6.c 6:3d40f75d:::benchmark_data_smithi154_175640_object17971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:44.702701+0000", 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 118.714618556, 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.136628231, 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 103747 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.439 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.702701+0000", 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.702701+0000", 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.702703+0000", 2024-09-13T14:53:43.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8610000000000001e-06 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.702708+0000", 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1610000000000002e-06 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.702720+0000", 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1674999999999999e-05 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:44.702725+0000", 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0939999999999998e-06 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.639895+0000", 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.93716959 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.652289+0000", 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012394433999999999 2024-09-13T14:53:43.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.726718+0000", 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.074428581999999993 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.731588+0000", 2024-09-13T14:53:43.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048704519999999999 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.741838+0000", 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010249688999999999 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.806088+0000", 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.064250031999999999 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.825288+0000", 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019200423000000001 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.826990+0000", 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001702114 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.829423+0000", 2024-09-13T14:53:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024327659999999998 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:45.839330+0000", 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099063580000000005 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:103964 6.c 6:325c1b5d:::benchmark_data_smithi154_175640_object18188:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:46.030158+0000", 2024-09-13T14:53:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 117.387162326, 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2251706149999999, 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 103964 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.030158+0000", 2024-09-13T14:53:43.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.030158+0000", 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.030159+0000", 2024-09-13T14:53:43.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.248e-06 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.030164+0000", 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4210000000000003e-06 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.030176+0000", 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2035000000000001e-05 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.030182+0000", 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8619999999999999e-06 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.065623+0000", 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.03544065 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.081234+0000", 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015611389 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.139125+0000", 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.057891142 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.144010+0000", 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048849109999999996 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.153118+0000", 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091073379999999995 2024-09-13T14:53:43.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.222380+0000", 2024-09-13T14:53:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069262260000000006 2024-09-13T14:53:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.241063+0000", 2024-09-13T14:53:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018683153000000001 2024-09-13T14:53:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.243098+0000", 2024-09-13T14:53:43.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020349589999999998 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.245849+0000", 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027511359999999999 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.255328+0000", 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0094791110000000001 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.454 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:104009 6.c 6:385b5a9f:::benchmark_data_smithi154_175640_object18233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:46.479884+0000", 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 116.93743593799999, 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.15352374, 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.455 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 104009 2024-09-13T14:53:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.479884+0000", 2024-09-13T14:53:43.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.479884+0000", 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.479886+0000", 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5770000000000001e-06 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.479892+0000", 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4420000000000003e-06 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.479903+0000", 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1381e-05 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:46.479910+0000", 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0609999999999998e-06 2024-09-13T14:53:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.489952+0000", 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.010041162 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.501770+0000", 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011818696 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.545249+0000", 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.043478676000000001 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.548871+0000", 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036216669999999999 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.556164+0000", 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072932689999999998 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.581791+0000", 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025627469999999999 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.620244+0000", 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038453099999999997 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.621921+0000", 2024-09-13T14:53:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016762960000000001 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.624612+0000", 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002691433 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.633408+0000", 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0087955099999999994 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:104116 6.c 6:39e0dc97:::benchmark_data_smithi154_175640_object18340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:47.085819+0000", 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 116.331500834, 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.158346036, 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 104116 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.085819+0000", 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.085819+0000", 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.085821+0000", 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.72e-06 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.085827+0000", 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7100000000000001e-06 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.085845+0000", 2024-09-13T14:53:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7469000000000002e-05 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.085853+0000", 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1980000000000001e-06 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.065250+0000", 2024-09-13T14:53:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97939699700000005 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.076552+0000", 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011301821 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.123408+0000", 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046856285999999997 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.127917+0000", 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045090190000000004 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.135354+0000", 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074364260000000003 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.209025+0000", 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.073671756000000005 2024-09-13T14:53:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.228166+0000", 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019140071000000002 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.229864+0000", 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016989240000000001 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.233282+0000", 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034176499999999999 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.244165+0000", 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010882988999999999 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:104152 6.c 6:3d30e299:::benchmark_data_smithi154_175640_object18376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:47.314719+0000", 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 116.102600396, 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1373569779999999, 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 104152 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.314719+0000", 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.314719+0000", 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.314721+0000", 2024-09-13T14:53:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7740000000000001e-06 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.314727+0000", 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1759999999999998e-06 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.314740+0000", 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2816000000000001e-05 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.314748+0000", 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2169999999999994e-06 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.281226+0000", 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96647711000000003 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.296219+0000", 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014993723 2024-09-13T14:53:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.359588+0000", 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.063368808999999998 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.363257+0000", 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036688900000000002 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.370572+0000", 2024-09-13T14:53:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073149799999999996 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.416165+0000", 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.045592780999999999 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.438537+0000", 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022372291999999998 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.440282+0000", 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00174472 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.443035+0000", 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027536520000000001 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.452076+0000", 2024-09-13T14:53:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090410379999999995 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:104155 6.c 6:325ec7da:::benchmark_data_smithi154_175640_object18379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:47.328131+0000", 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 116.089188918, 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1396112270000001, 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 104155 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.328131+0000", 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.328131+0000", 2024-09-13T14:53:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.328133+0000", 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6500000000000001e-06 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.328138+0000", 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7139999999999998e-06 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.328150+0000", 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2140999999999999e-05 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:47.328158+0000", 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5580000000000003e-06 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.297162+0000", 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96900430500000001 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.315579+0000", 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018416713000000001 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.398808+0000", 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.083228794999999994 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.402456+0000", 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036477749999999998 2024-09-13T14:53:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.409668+0000", 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072122269999999999 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.437339+0000", 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027670831 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.453729+0000", 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016389857000000001 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.455472+0000", 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001743608 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.457928+0000", 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024563129999999999 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:48.467742+0000", 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098137399999999996 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:104870 6.c 6:3d5e8690:::benchmark_data_smithi154_175640_object19094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:50.550206+0000", 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 112.867113574, 2024-09-13T14:53:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.149354717, 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 104870 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:50.550206+0000", 2024-09-13T14:53:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:50.550206+0000", 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:50.550208+0000", 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6080000000000001e-06 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:50.550215+0000", 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6800000000000004e-06 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:50.550227+0000", 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2568999999999999e-05 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:50.550236+0000", 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.524e-06 2024-09-13T14:53:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.535499+0000", 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98526320899999997 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.546011+0000", 2024-09-13T14:53:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010511803 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.591656+0000", 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.045644960999999998 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.595291+0000", 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036353510000000002 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.602277+0000", 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069856190000000002 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.663181+0000", 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.060904145999999999 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.684586+0000", 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021404851999999999 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.686807+0000", 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022217399999999998 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.689645+0000", 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028377060000000002 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:51.699561+0000", 2024-09-13T14:53:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099159490000000003 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:105214 6.c 6:3e67ad81:::benchmark_data_smithi154_175640_object19438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:52.311605+0000", 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 111.105715042, 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1484703979999999, 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 105214 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:52.311605+0000", 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:52.311605+0000", 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:52.311606+0000", 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3400000000000001e-06 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:52.311611+0000", 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9899999999999997e-06 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:52.311623+0000", 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2009e-05 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:52.311633+0000", 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6970000000000005e-06 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.250013+0000", 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.93838048100000004 2024-09-13T14:53:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.261901+0000", 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011887986999999999 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.400602+0000", 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13870101100000001 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.404195+0000", 2024-09-13T14:53:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035927390000000002 2024-09-13T14:53:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.411377+0000", 2024-09-13T14:53:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071814380000000001 2024-09-13T14:53:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.428965+0000", 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017588599999999999 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.430895+0000", 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019303160000000001 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.443846+0000", 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012950755 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.446259+0000", 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024123460000000001 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:53.460075+0000", 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013816689 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:105723 6.c 6:35d6d1bc:::benchmark_data_smithi154_175640_object19947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:54.921043+0000", 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 108.49627684399999, 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.144536854, 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 105723 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.921043+0000", 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.921043+0000", 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.921045+0000", 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5820000000000001e-06 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.921050+0000", 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0760000000000002e-06 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.921064+0000", 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4222999999999999e-05 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.921069+0000", 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3689999999999998e-06 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.887900+0000", 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96683035900000003 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.901309+0000", 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013409559 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.947593+0000", 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046283959 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.951268+0000", 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036750160000000001 2024-09-13T14:53:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.961126+0000", 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098580119999999993 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.029106+0000", 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.067980115999999993 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.050979+0000", 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.02187236 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.053134+0000", 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021549070000000002 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.055810+0000", 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002676234 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.065580+0000", 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097700819999999994 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:105730 6.c 6:34b7bbc7:::benchmark_data_smithi154_175640_object19954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:54.944747+0000", 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 108.472573154, 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.136868923, 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 105730 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.944747+0000", 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.944747+0000", 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.944748+0000", 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5090000000000001e-06 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.944754+0000", 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8850000000000004e-06 2024-09-13T14:53:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.944766+0000", 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2140000000000001e-05 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:54.944774+0000", 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3150000000000003e-06 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.901881+0000", 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.95710709100000002 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.912301+0000", 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010420118000000001 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.992310+0000", 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.080009293999999995 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:55.996133+0000", 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038229750000000002 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.004052+0000", 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079194160000000003 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.051766+0000", 2024-09-13T14:53:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047713985 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.067206+0000", 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015439612 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.068794+0000", 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015879170000000001 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.071221+0000", 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024270110000000002 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:56.081616+0000", 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010394654999999999 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110473 6.c 6:3a47cf6b:::benchmark_data_smithi154_175640_object24697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:15.677399+0000", 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.739920995999995, 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.227982321, 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110473 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.677399+0000", 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.677399+0000", 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.677400+0000", 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3600000000000002e-07 2024-09-13T14:53:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.677404+0000", 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4909999999999999e-06 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.677418+0000", 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3662999999999999e-05 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.677425+0000", 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.007e-06 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.626600+0000", 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.94917499599999999 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.635654+0000", 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090545150000000008 2024-09-13T14:53:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.701274+0000", 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.065619073 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.705190+0000", 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039162010000000002 2024-09-13T14:53:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.712267+0000", 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070772129999999997 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.784836+0000", 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.072568638000000005 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.889823+0000", 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.10498717 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.891923+0000", 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021000110000000002 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.895166+0000", 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003243288 2024-09-13T14:53:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.905381+0000", 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010215119 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110476 6.c 6:3ab2666a:::benchmark_data_smithi154_175640_object24700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:15.688182+0000", 2024-09-13T14:53:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.729137660999996, 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2391212469999999, 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110476 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.688182+0000", 2024-09-13T14:53:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.688182+0000", 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.688184+0000", 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.578e-06 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.688190+0000", 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3729999999999998e-06 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.688203+0000", 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3135e-05 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.688211+0000", 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7139999999999993e-06 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.636061+0000", 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.94785016799999999 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.645198+0000", 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091372039999999995 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.735827+0000", 2024-09-13T14:53:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.090628311000000003 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.738991+0000", 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00316395 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.745300+0000", 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063090450000000001 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.819042+0000", 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.073742354999999996 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.906728+0000", 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.087685660999999998 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.909220+0000", 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024922049999999999 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.912441+0000", 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003221224 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.927303+0000", 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014862324 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110480 6.c 6:38080f4b:::benchmark_data_smithi154_175640_object24704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:15.738037+0000", 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.679282982000004, 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.210701724, 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110480 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.738037+0000", 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.738037+0000", 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.738038+0000", 2024-09-13T14:53:43.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2580000000000001e-06 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.738043+0000", 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2360000000000004e-06 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.738056+0000", 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2284e-05 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.738062+0000", 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4119999999999998e-06 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.646646+0000", 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.90858374200000003 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.655805+0000", 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091595370000000006 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.769451+0000", 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.113645909 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.772600+0000", 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003148264 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.778893+0000", 2024-09-13T14:53:43.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062936950000000002 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.819403+0000", 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.040509528000000003 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.930160+0000", 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.110756781 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.933252+0000", 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030925979999999998 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.936431+0000", 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031791089999999998 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.948739+0000", 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012307370999999999 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.538 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110494 6.c 6:3d0c85ef:::benchmark_data_smithi154_175640_object24718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:15.837436+0000", 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.579883441999996, 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1417330350000001, 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.539 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110494 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.837436+0000", 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.837436+0000", 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.837438+0000", 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7740000000000001e-06 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.837443+0000", 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7700000000000001e-06 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.837459+0000", 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5687000000000001e-05 2024-09-13T14:53:43.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:15.837466+0000", 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7419999999999997e-06 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.656320+0000", 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.81885332899999996 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.665790+0000", 2024-09-13T14:53:43.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0094702339999999992 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.803550+0000", 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.137760046 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.806760+0000", 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032101489999999998 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.813172+0000", 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064119210000000001 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.871194+0000", 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058021715000000001 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.951628+0000", 2024-09-13T14:53:43.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.080433888999999995 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.955959+0000", 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043309100000000003 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.960378+0000", 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044196950000000004 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.979169+0000", 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018791174000000001 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110622 6.c 6:3dfb9b68:::benchmark_data_smithi154_175640_object24846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:16.369783+0000", 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 87.047536944000001, 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1572990059999999, 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.546 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110622 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.369783+0000", 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.369783+0000", 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.369784+0000", 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3379999999999999e-06 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.369790+0000", 2024-09-13T14:53:43.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.541e-06 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.369802+0000", 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1871999999999999e-05 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.369809+0000", 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5560000000000002e-06 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.348030+0000", 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97822052500000001 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.359554+0000", 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011524029999999999 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.411037+0000", 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051483255999999998 2024-09-13T14:53:43.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.415951+0000", 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049137199999999999 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.425115+0000", 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091640460000000003 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.508449+0000", 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.083334288000000006 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.510469+0000", 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020195320000000001 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.512010+0000", 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001541438 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.514549+0000", 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002538818 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.527082+0000", 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012533046000000001 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.553 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110646 6.c 6:3ff31517:::benchmark_data_smithi154_175640_object24870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:16.503993+0000", 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 86.913327343000006, 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2226920670000001, 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110646 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.503993+0000", 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.503993+0000", 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.503994+0000", 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.398e-06 2024-09-13T14:53:43.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.504000+0000", 2024-09-13T14:53:43.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0109999999999997e-06 2024-09-13T14:53:43.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.504012+0000", 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2455e-05 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.504022+0000", 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1360000000000003e-06 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.559262+0000", 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0552400770000001 2024-09-13T14:53:43.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.571348+0000", 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012086556 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.609942+0000", 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038593441999999999 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.614614+0000", 2024-09-13T14:53:43.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004671934 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.623402+0000", 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008788433 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.711614+0000", 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.088212441000000003 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.713131+0000", 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001516332 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.715106+0000", 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019751899999999999 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.718477+0000", 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033706840000000001 2024-09-13T14:53:43.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.726685+0000", 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082079779999999995 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110654 6.c 6:352676dd:::benchmark_data_smithi154_175640_object24878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:16.607395+0000", 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 86.809924835999993, 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1334504000000001, 2024-09-13T14:53:43.562 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110654 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.607395+0000", 2024-09-13T14:53:43.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.607395+0000", 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.607396+0000", 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3489999999999999e-06 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.607401+0000", 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.904e-06 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.607414+0000", 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2598e-05 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.607422+0000", 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5359999999999992e-06 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.571853+0000", 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96443024399999999 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.580755+0000", 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089022900000000002 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.650632+0000", 2024-09-13T14:53:43.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069877481000000005 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.655035+0000", 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044025610000000001 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.663739+0000", 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0087044099999999992 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.727783+0000", 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.064043424000000002 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.729259+0000", 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001476336 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.729968+0000", 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00070924200000000001 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.731691+0000", 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001722513 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.740845+0000", 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091545120000000001 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:110656 6.c 6:30e9e159:::benchmark_data_smithi154_175640_object24880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:16.614353+0000", 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 86.802967193000001, 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.145364813, 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 110656 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.570 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.614353+0000", 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.614353+0000", 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.614354+0000", 2024-09-13T14:53:43.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-06 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.614362+0000", 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3339999999999996e-06 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.614377+0000", 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5381999999999999e-05 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:16.614387+0000", 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0329999999999999e-05 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.581168+0000", 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96678030800000003 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.590348+0000", 2024-09-13T14:53:43.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091799029999999997 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.694881+0000", 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.104533295 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.698045+0000", 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003164423 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T14:53:43.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.705493+0000", 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074477479999999997 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.741287+0000", 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035793606999999998 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.742668+0000", 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001381462 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.744657+0000", 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019892360000000001 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.747864+0000", 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00320645 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:17.759717+0000", 2024-09-13T14:53:43.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011853605 2024-09-13T14:53:43.577 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.577 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.577 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.577 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:53:43.577 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:53:43.577 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:53:43.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:43.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:43.717 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:43.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:43.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:43.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:43.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:43.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:44.053 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:53:44.053 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:53:44.053 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:53:44.053 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:53:44.053 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:53:44.053 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:44.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:44.096 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:44.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[102314]: pgmap v750: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 365/86172 objects degraded (0.424%); 1772/86172 objects misplaced (2.056%) 2024-09-13T14:53:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3682947105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[107038]: pgmap v750: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 365/86172 objects degraded (0.424%); 1772/86172 objects misplaced (2.056%) 2024-09-13T14:53:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3682947105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:44.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:43 smithi172 ceph-mon[100696]: pgmap v750: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 542 op/s; 365/86172 objects degraded (0.424%); 1772/86172 objects misplaced (2.056%) 2024-09-13T14:53:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3682947105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:44.264 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:44.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.393 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:44.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:44.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:44.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:44.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:44.647 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:44.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:44.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:44.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:44.896 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:44.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:44.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:45.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:45.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 365/86172 objects degraded (0.424%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 365/86172 objects degraded (0.424%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 365/86172 objects degraded (0.424%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:45.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:45.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:45.312 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:45.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:53:45.368 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:45.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:45.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:45.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:45.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:45.546 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:45.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:45.581 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:45.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:45.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:45.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:45.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:45.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:45.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:45.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:46.025 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:45 smithi154 ceph-mon[102314]: pgmap v751: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 532 op/s; 365/79722 objects degraded (0.458%); 1772/79722 objects misplaced (2.223%) 2024-09-13T14:53:46.025 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:45 smithi154 ceph-mon[107038]: pgmap v751: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 532 op/s; 365/79722 objects degraded (0.458%); 1772/79722 objects misplaced (2.223%) 2024-09-13T14:53:46.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:46.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:46.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:46.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:45 smithi172 ceph-mon[100696]: pgmap v751: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 532 op/s; 365/79722 objects degraded (0.458%); 1772/79722 objects misplaced (2.223%) 2024-09-13T14:53:46.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:46.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:46.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:46.444 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:46.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:46.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:46.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:46.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:46.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:46.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:46.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:53:46.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:46.964 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:46.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:47.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:47.091 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:47.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:47.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:47.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:47.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:47.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:47.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:47.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:47.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:47.623 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:47.623 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:47.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:47.728 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:47.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:47.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:47.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:47.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:47.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:48.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:48.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:48.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:48 smithi154 ceph-mon[102314]: pgmap v752: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 544/78135 objects degraded (0.696%); 1593/78135 objects misplaced (2.039%) 2024-09-13T14:53:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:48 smithi154 ceph-mon[107038]: pgmap v752: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 544/78135 objects degraded (0.696%); 1593/78135 objects misplaced (2.039%) 2024-09-13T14:53:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:48 smithi172 ceph-mon[100696]: pgmap v752: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 527 op/s; 544/78135 objects degraded (0.696%); 1593/78135 objects misplaced (2.039%) 2024-09-13T14:53:48.243 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:48.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:53:48.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:48.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:48.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:48.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:48.558 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:48.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:48.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:48.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:48.776 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:48.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:48.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:48.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:49.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:49.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:49.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:49.172 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:49.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:49.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:49.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:49.439 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:49.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:49.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:49.648 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:49.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:49.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:49.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:49.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:49.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:49.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:49.992 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:49.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:50.004 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:53:50.005 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:53:50.087 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:53:50.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:50.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:50.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:50.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:50.332 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:50 smithi172 ceph-mon[100696]: pgmap v753: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.5 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 507 op/s; 544/75963 objects degraded (0.716%); 1593/75963 objects misplaced (2.097%) 2024-09-13T14:53:50.333 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3992837238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:50.333 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:50 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 544/75963 objects degraded (0.716%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:50.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:50 smithi154 ceph-mon[102314]: pgmap v753: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.5 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 507 op/s; 544/75963 objects degraded (0.716%); 1593/75963 objects misplaced (2.097%) 2024-09-13T14:53:50.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3992837238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:50.348 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:50 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 544/75963 objects degraded (0.716%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:50.348 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:50 smithi154 ceph-mon[107038]: pgmap v753: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.5 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 507 op/s; 544/75963 objects degraded (0.716%); 1593/75963 objects misplaced (2.097%) 2024-09-13T14:53:50.348 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3992837238' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:53:50.348 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:50 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 544/75963 objects degraded (0.716%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:50.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:50.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:50.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.533 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:50.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:50.561 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:50.561 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:50.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:50.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:50.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:50.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:50.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:50.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:50.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:51.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:51.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:51.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:51.182 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:51 smithi172 ceph-mon[100696]: from='client.45263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:51.182 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:51 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:51.182 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3992837238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:51.183 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:51 smithi172 ceph-mon[100696]: osdmap e252: 8 total, 8 up, 7 in 2024-09-13T14:53:51.183 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.184 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:51.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:51.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:53:51.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:51.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:51.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[102314]: from='client.45263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:51.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:51.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3992837238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:51.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[102314]: osdmap e252: 8 total, 8 up, 7 in 2024-09-13T14:53:51.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:51.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[107038]: from='client.45263 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:53:51.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:53:51.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3992837238' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:53:51.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[107038]: osdmap e252: 8 total, 8 up, 7 in 2024-09-13T14:53:51.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:53:51.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:51.498 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:51.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:51.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:51.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:51.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:51.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:51.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:51.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:51.899 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-09-13T14:53:49.013698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1662326781,"num_objects":25321,"num_object_clones":0,"num_object_copies":75963,"num_objects_missing_on_primary":67,"num_objects_missing":67,"num_objects_degraded":544,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":25321,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":147378,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44813,"num_bytes_recovered":2809678981,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124609,"ondisk_log_size":124609,"up":267,"acting":266,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":269,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7249724,"kb_used_data":5097412,"kb_used_omap":32,"kb_used_meta":2152159,"kb_avail":648966340,"statfs":{"total":671965249536,"available":664541532160,"internally_reserved":0,"allocated":5219749888,"data_stored":5210260297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2203811376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,3,2,0,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-398983168,"num_objects":-6088,"num_object_clones":0,"num_object_copies":-18264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":-352,"num_objects_unfound":0,"num_objects_dirty":-6088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003149"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":384,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.867152+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:53:36.867152+0000","last_peered":"2024-09-13T14:53:36.867152+0000","last_clean":"2024-09-13T14:53:36.867152+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:53:36.867152+0000","last_undegraded":"2024-09-13T14:53:36.867152+0000","last_fullsized":"2024-09-13T14:53:36.867152+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":117,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615871+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:53:26.615871+0000","last_peered":"2024-09-13T14:53:26.615871+0000","last_clean":"2024-09-13T14:53:26.615871+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:53:26.615871+0000","last_undegraded":"2024-09-13T14:53:26.615871+0000","last_fullsized":"2024-09-13T14:53:26.615871+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":87,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640519+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:53:26.640519+0000","last_peered":"2024-09-13T14:53:26.640519+0000","last_clean":"2024-09-13T14:53:26.640519+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:53:26.640519+0000","last_undegraded":"2024-09-13T14:53:26.640519+0000","last_fullsized":"2024-09-13T14:53:26.640519+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":373,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849021+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:53:36.849021+0000","last_peered":"2024-09-13T14:53:36.849021+0000","last_clean":"2024-09-13T14:53:36.849021+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:53:36.849021+0000","last_undegraded":"2024-09-13T14:53:36.849021+0000","last_fullsized":"2024-09-13T14:53:36.849021+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":373,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850201+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:53:36.850201+0000","last_peered":"2024-09-13T14:53:36.850201+0000","last_clean":"2024-09-13T14:53:36.850201+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:53:36.850201+0000","last_undegraded":"2024-09-13T14:53:36.850201+0000","last_fullsized":"2024-09-13T14:53:36.850201+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":187,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615363+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:53:26.615363+0000","last_peered":"2024-09-13T14:53:26.615363+0000","last_clean":"2024-09-13T14:53:26.615363+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:53:26.615363+0000","last_undegraded":"2024-09-13T14:53:26.615363+0000","last_fullsized":"2024-09-13T14:53:26.615363+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":99,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850202+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:53:36.850202+0000","last_peered":"2024-09-13T14:53:36.850202+0000","last_clean":"2024-09-13T14:53:36.850202+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:53:36.850202+0000","last_undegraded":"2024-09-13T14:53:36.850202+0000","last_fullsized":"2024-09-13T14:53:36.850202+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":163,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.636732+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:53:26.636732+0000","last_peered":"2024-09-13T14:53:26.636732+0000","last_clean":"2024-09-13T14:53:26.636732+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:53:26.636732+0000","last_undegraded":"2024-09-13T14:53:26.636732+0000","last_fullsized":"2024-09-13T14:53:26.636732+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"251'9248","reported_seq":10093,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.565070+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:53:44.565070+0000","last_peered":"2024-09-13T14:53:44.565070+0000","last_clean":"2024-09-13T14:53:44.565070+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:53:44.565070+0000","last_undegraded":"2024-09-13T14:53:44.565070+0000","last_fullsized":"2024-09-13T14:53:44.565070+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":9248,"log_dups_size":0,"ondisk_log_size":9248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108003328,"num_objects":1648,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9248,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":308,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.848345+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:53:36.848345+0000","last_peered":"2024-09-13T14:53:36.848345+0000","last_clean":"2024-09-13T14:53:36.848345+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:53:36.848345+0000","last_undegraded":"2024-09-13T14:53:36.848345+0000","last_fullsized":"2024-09-13T14:53:36.848345+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":114,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640152+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:53:26.640152+0000","last_peered":"2024-09-13T14:53:26.640152+0000","last_clean":"2024-09-13T14:53:26.640152+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:53:26.640152+0000","last_undegraded":"2024-09-13T14:53:26.640152+0000","last_fullsized":"2024-09-13T14:53:26.640152+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"251'9009","reported_seq":10640,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:45.073319+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:53:45.073319+0000","last_peered":"2024-09-13T14:53:45.073319+0000","last_clean":"2024-09-13T14:53:45.073319+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:53:45.073319+0000","last_undegraded":"2024-09-13T14:53:45.073319+0000","last_fullsized":"2024-09-13T14:53:45.073319+0000","mapping_epoch":170,"log_start":"180'2600","ondisk_log_start":"180'2600","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102825984,"num_objects":1569,"num_object_clones":0,"num_object_copies":4707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9009,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":115,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615721+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:53:26.615721+0000","last_peered":"2024-09-13T14:53:26.615721+0000","last_clean":"2024-09-13T14:53:26.615721+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:53:26.615721+0000","last_undegraded":"2024-09-13T14:53:26.615721+0000","last_fullsized":"2024-09-13T14:53:26.615721+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":267,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615559+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:53:26.615559+0000","last_peered":"2024-09-13T14:53:26.615559+0000","last_clean":"2024-09-13T14:53:26.615559+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:53:26.615559+0000","last_undegraded":"2024-09-13T14:53:26.615559+0000","last_fullsized":"2024-09-13T14:53:26.615559+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"251'9184","reported_seq":10908,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.277852+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:53:44.277852+0000","last_peered":"2024-09-13T14:53:44.277852+0000","last_clean":"2024-09-13T14:53:44.277852+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:53:44.277852+0000","last_undegraded":"2024-09-13T14:53:44.277852+0000","last_fullsized":"2024-09-13T14:53:44.277852+0000","mapping_epoch":175,"log_start":"164'300","ondisk_log_start":"164'300","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9184,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":91,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640624+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:53:26.640624+0000","last_peered":"2024-09-13T14:53:26.640624+0000","last_clean":"2024-09-13T14:53:26.640624+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:53:26.640624+0000","last_undegraded":"2024-09-13T14:53:26.640624+0000","last_fullsized":"2024-09-13T14:53:26.640624+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":47,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.636696+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:53:26.636696+0000","last_peered":"2024-09-13T14:53:26.636696+0000","last_clean":"2024-09-13T14:53:26.636696+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:53:26.636696+0000","last_undegraded":"2024-09-13T14:53:26.636696+0000","last_fullsized":"2024-09-13T14:53:26.636696+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"251'9051","reported_seq":15170,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:47.945993+0000","last_change":"2024-09-13T14:52:50.434519+0000","last_active":"2024-09-13T14:53:47.945993+0000","last_peered":"2024-09-13T14:53:47.945993+0000","last_clean":"2024-09-13T14:53:47.945993+0000","last_became_active":"2024-09-13T14:52:50.417699+0000","last_became_peered":"2024-09-13T14:52:50.417699+0000","last_unstale":"2024-09-13T14:53:47.945993+0000","last_undegraded":"2024-09-13T14:53:47.945993+0000","last_fullsized":"2024-09-13T14:53:47.945993+0000","mapping_epoch":238,"log_start":"164'700","ondisk_log_start":"164'700","created":160,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:10.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95485952,"num_objects":1457,"num_object_clones":0,"num_object_copies":4371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2948,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":103,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640724+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:53:26.640724+0000","last_peered":"2024-09-13T14:53:26.640724+0000","last_clean":"2024-09-13T14:53:26.640724+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:53:26.640724+0000","last_undegraded":"2024-09-13T14:53:26.640724+0000","last_fullsized":"2024-09-13T14:53:26.640724+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":353,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849870+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:53:36.849870+0000","last_peered":"2024-09-13T14:53:36.849870+0000","last_clean":"2024-09-13T14:53:36.849870+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:53:36.849870+0000","last_undegraded":"2024-09-13T14:53:36.849870+0000","last_fullsized":"2024-09-13T14:53:36.849870+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"251'9095","reported_seq":26358,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.183523+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:53:44.183523+0000","last_peered":"2024-09-13T14:53:44.183523+0000","last_clean":"2024-09-13T14:53:44.183523+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:53:44.183523+0000","last_undegraded":"2024-09-13T14:53:44.183523+0000","last_fullsized":"2024-09-13T14:53:44.183523+0000","mapping_epoch":232,"log_start":"172'1800","ondisk_log_start":"172'1800","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7295,"log_dups_size":0,"ondisk_log_size":7295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108724224,"num_objects":1659,"num_object_clones":0,"num_object_copies":4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":361,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849783+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:53:36.849783+0000","last_peered":"2024-09-13T14:53:36.849783+0000","last_clean":"2024-09-13T14:53:36.849783+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:53:36.849783+0000","last_undegraded":"2024-09-13T14:53:36.849783+0000","last_fullsized":"2024-09-13T14:53:36.849783+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":234,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615425+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:53:26.615425+0000","last_peered":"2024-09-13T14:53:26.615425+0000","last_clean":"2024-09-13T14:53:26.615425+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:53:26.615425+0000","last_undegraded":"2024-09-13T14:53:26.615425+0000","last_fullsized":"2024-09-13T14:53:26.615425+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"251'9023","reported_seq":14033,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.223567+0000","last_change":"2024-09-13T14:53:24.611648+0000","last_active":"2024-09-13T14:53:44.223567+0000","last_peered":"2024-09-13T14:53:44.223567+0000","last_clean":"2024-09-13T14:53:44.223567+0000","last_became_active":"2024-09-13T14:53:24.585428+0000","last_became_peered":"2024-09-13T14:53:24.585428+0000","last_unstale":"2024-09-13T14:53:44.223567+0000","last_undegraded":"2024-09-13T14:53:44.223567+0000","last_fullsized":"2024-09-13T14:53:44.223567+0000","mapping_epoch":248,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7323,"log_dups_size":0,"ondisk_log_size":7323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:41.636547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105971712,"num_objects":1617,"num_object_clones":0,"num_object_copies":4851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9023,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":147914752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":135,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850373+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:53:36.850373+0000","last_peered":"2024-09-13T14:53:36.850373+0000","last_clean":"2024-09-13T14:53:36.850373+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:53:36.850373+0000","last_undegraded":"2024-09-13T14:53:36.850373+0000","last_fullsized":"2024-09-13T14:53:36.850373+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":231,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.616080+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:53:26.616080+0000","last_peered":"2024-09-13T14:53:26.616080+0000","last_clean":"2024-09-13T14:53:26.616080+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:53:26.616080+0000","last_undegraded":"2024-09-13T14:53:26.616080+0000","last_fullsized":"2024-09-13T14:53:26.616080+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"251'9220","reported_seq":13949,"reported_epoch":251,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:53:45.032464+0000","last_change":"2024-09-13T14:53:29.139613+0000","last_active":"2024-09-13T14:53:45.032464+0000","last_peered":"2024-09-13T14:53:45.032464+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:53:45.032464+0000","last_undegraded":"2024-09-13T14:53:29.127369+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"181'2800","ondisk_log_start":"181'2800","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104988672,"num_objects":1602,"num_object_clones":0,"num_object_copies":4806,"num_objects_missing_on_primary":67,"num_objects_missing":67,"num_objects_degraded":544,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":1602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":155582464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":1592},{"shards":"3,5","objects":-535}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":226,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615634+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:53:26.615634+0000","last_peered":"2024-09-13T14:53:26.615634+0000","last_clean":"2024-09-13T14:53:26.615634+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:53:26.615634+0000","last_undegraded":"2024-09-13T14:53:26.615634+0000","last_fullsized":"2024-09-13T14:53:26.615634+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":261,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615783+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:53:26.615783+0000","last_peered":"2024-09-13T14:53:26.615783+0000","last_clean":"2024-09-13T14:53:26.615783+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:53:26.615783+0000","last_undegraded":"2024-09-13T14:53:26.615783+0000","last_fullsized":"2024-09-13T14:53:26.615783+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"251'8907","reported_seq":13346,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.509822+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:53:44.509822+0000","last_peered":"2024-09-13T14:53:44.509822+0000","last_clean":"2024-09-13T14:53:44.509822+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:53:44.509822+0000","last_undegraded":"2024-09-13T14:53:44.509822+0000","last_fullsized":"2024-09-13T14:53:44.509822+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104726552,"num_objects":1599,"num_object_clones":0,"num_object_copies":4797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8907,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":99,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.847287+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:53:36.847287+0000","last_peered":"2024-09-13T14:53:36.847287+0000","last_clean":"2024-09-13T14:53:36.847287+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:53:36.847287+0000","last_undegraded":"2024-09-13T14:53:36.847287+0000","last_fullsized":"2024-09-13T14:53:36.847287+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":208,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.848416+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:53:36.848416+0000","last_peered":"2024-09-13T14:53:36.848416+0000","last_clean":"2024-09-13T14:53:36.848416+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:53:36.848416+0000","last_undegraded":"2024-09-13T14:53:36.848416+0000","last_fullsized":"2024-09-13T14:53:36.848416+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"251'9137","reported_seq":10397,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.478528+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:53:44.478528+0000","last_peered":"2024-09-13T14:53:44.478528+0000","last_clean":"2024-09-13T14:53:44.478528+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:53:44.478528+0000","last_undegraded":"2024-09-13T14:53:44.478528+0000","last_fullsized":"2024-09-13T14:53:44.478528+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105316352,"num_objects":1607,"num_object_clones":0,"num_object_copies":4821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":555,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849205+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:53:36.849205+0000","last_peered":"2024-09-13T14:53:36.849205+0000","last_clean":"2024-09-13T14:53:36.849205+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:53:36.849205+0000","last_undegraded":"2024-09-13T14:53:36.849205+0000","last_fullsized":"2024-09-13T14:53:36.849205+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":818,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640305+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:53:26.640305+0000","last_peered":"2024-09-13T14:53:26.640305+0000","last_clean":"2024-09-13T14:53:26.640305+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:53:26.640305+0000","last_undegraded":"2024-09-13T14:53:26.640305+0000","last_fullsized":"2024-09-13T14:53:26.640305+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":319,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.846194+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:53:36.846194+0000","last_peered":"2024-09-13T14:53:36.846194+0000","last_clean":"2024-09-13T14:53:36.846194+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:53:36.846194+0000","last_undegraded":"2024-09-13T14:53:36.846194+0000","last_fullsized":"2024-09-13T14:53:36.846194+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":47,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.637036+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:53:26.637036+0000","last_peered":"2024-09-13T14:53:26.637036+0000","last_clean":"2024-09-13T14:53:26.637036+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:53:26.637036+0000","last_undegraded":"2024-09-13T14:53:26.637036+0000","last_fullsized":"2024-09-13T14:53:26.637036+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"251'9212","reported_seq":13957,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.645878+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:53:44.645878+0000","last_peered":"2024-09-13T14:53:44.645878+0000","last_clean":"2024-09-13T14:53:44.645878+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:53:44.645878+0000","last_undegraded":"2024-09-13T14:53:44.645878+0000","last_fullsized":"2024-09-13T14:53:44.645878+0000","mapping_epoch":223,"log_start":"180'2618","ondisk_log_start":"180'2618","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":400,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.854378+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:53:36.854378+0000","last_peered":"2024-09-13T14:53:36.854378+0000","last_clean":"2024-09-13T14:53:36.854378+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:53:36.854378+0000","last_undegraded":"2024-09-13T14:53:36.854378+0000","last_fullsized":"2024-09-13T14:53:36.854378+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":367,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849782+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:53:36.849782+0000","last_peered":"2024-09-13T14:53:36.849782+0000","last_clean":"2024-09-13T14:53:36.849782+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:53:36.849782+0000","last_undegraded":"2024-09-13T14:53:36.849782+0000","last_fullsized":"2024-09-13T14:53:36.849782+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":378,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850437+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:53:36.850437+0000","last_peered":"2024-09-13T14:53:36.850437+0000","last_clean":"2024-09-13T14:53:36.850437+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:53:36.850437+0000","last_undegraded":"2024-09-13T14:53:36.850437+0000","last_fullsized":"2024-09-13T14:53:36.850437+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"251'9197","reported_seq":19524,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:48.768128+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:53:48.768128+0000","last_peered":"2024-09-13T14:53:48.768128+0000","last_clean":"2024-09-13T14:53:48.768128+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:53:48.768128+0000","last_undegraded":"2024-09-13T14:53:48.768128+0000","last_fullsized":"2024-09-13T14:53:48.768128+0000","mapping_epoch":235,"log_start":"168'1000","ondisk_log_start":"168'1000","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8197,"log_dups_size":0,"ondisk_log_size":8197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91684864,"num_objects":1399,"num_object_clones":0,"num_object_copies":4197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9197,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":903,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.847286+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:53:36.847286+0000","last_peered":"2024-09-13T14:53:36.847286+0000","last_clean":"2024-09-13T14:53:36.847286+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:53:36.847286+0000","last_undegraded":"2024-09-13T14:53:36.847286+0000","last_fullsized":"2024-09-13T14:53:36.847286+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":358,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.854556+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:53:36.854556+0000","last_peered":"2024-09-13T14:53:36.854556+0000","last_clean":"2024-09-13T14:53:36.854556+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:53:36.854556+0000","last_undegraded":"2024-09-13T14:53:36.854556+0000","last_fullsized":"2024-09-13T14:53:36.854556+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":111,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.854892+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:53:36.854892+0000","last_peered":"2024-09-13T14:53:36.854892+0000","last_clean":"2024-09-13T14:53:36.854892+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:53:36.854892+0000","last_undegraded":"2024-09-13T14:53:36.854892+0000","last_fullsized":"2024-09-13T14:53:36.854892+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"251'9147","reported_seq":18486,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:45.005590+0000","last_change":"2024-09-13T14:52:52.443442+0000","last_active":"2024-09-13T14:53:45.005590+0000","last_peered":"2024-09-13T14:53:45.005590+0000","last_clean":"2024-09-13T14:53:45.005590+0000","last_became_active":"2024-09-13T14:52:52.421083+0000","last_became_peered":"2024-09-13T14:52:52.421083+0000","last_unstale":"2024-09-13T14:53:45.005590+0000","last_undegraded":"2024-09-13T14:53:45.005590+0000","last_fullsized":"2024-09-13T14:53:45.005590+0000","mapping_epoch":240,"log_start":"180'2700","ondisk_log_start":"180'2700","created":160,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:14.887813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103088128,"num_objects":1573,"num_object_clones":0,"num_object_copies":4719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4569,"num_bytes_recovered":287244288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":105,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640293+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:53:26.640293+0000","last_peered":"2024-09-13T14:53:26.640293+0000","last_clean":"2024-09-13T14:53:26.640293+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:53:26.640293+0000","last_undegraded":"2024-09-13T14:53:26.640293+0000","last_fullsized":"2024-09-13T14:53:26.640293+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":236,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.616022+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:53:26.616022+0000","last_peered":"2024-09-13T14:53:26.616022+0000","last_clean":"2024-09-13T14:53:26.616022+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:53:26.616022+0000","last_undegraded":"2024-09-13T14:53:26.616022+0000","last_fullsized":"2024-09-13T14:53:26.616022+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640327+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:53:26.640327+0000","last_peered":"2024-09-13T14:53:26.640327+0000","last_clean":"2024-09-13T14:53:26.640327+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:53:26.640327+0000","last_undegraded":"2024-09-13T14:53:26.640327+0000","last_fullsized":"2024-09-13T14:53:26.640327+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"251'9324","reported_seq":13934,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:47.936248+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:53:47.936248+0000","last_peered":"2024-09-13T14:53:47.936248+0000","last_clean":"2024-09-13T14:53:47.936248+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:53:47.936248+0000","last_undegraded":"2024-09-13T14:53:47.936248+0000","last_fullsized":"2024-09-13T14:53:47.936248+0000","mapping_epoch":226,"log_start":"167'900","ondisk_log_start":"167'900","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97648640,"num_objects":1490,"num_object_clones":0,"num_object_copies":4470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9324,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":105,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640481+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:53:26.640481+0000","last_peered":"2024-09-13T14:53:26.640481+0000","last_clean":"2024-09-13T14:53:26.640481+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:53:26.640481+0000","last_undegraded":"2024-09-13T14:53:26.640481+0000","last_fullsized":"2024-09-13T14:53:26.640481+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":913,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855059+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:53:36.855059+0000","last_peered":"2024-09-13T14:53:36.855059+0000","last_clean":"2024-09-13T14:53:36.855059+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:53:36.855059+0000","last_undegraded":"2024-09-13T14:53:36.855059+0000","last_fullsized":"2024-09-13T14:53:36.855059+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":20,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849987+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:53:36.849987+0000","last_peered":"2024-09-13T14:53:36.849987+0000","last_clean":"2024-09-13T14:53:36.849987+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:53:36.849987+0000","last_undegraded":"2024-09-13T14:53:36.849987+0000","last_fullsized":"2024-09-13T14:53:36.849987+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"251'9236","reported_seq":13973,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.693759+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:53:44.693759+0000","last_peered":"2024-09-13T14:53:44.693759+0000","last_clean":"2024-09-13T14:53:44.693759+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:53:44.693759+0000","last_undegraded":"2024-09-13T14:53:44.693759+0000","last_fullsized":"2024-09-13T14:53:44.693759+0000","mapping_epoch":218,"log_start":"178'2645","ondisk_log_start":"178'2645","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6591,"log_dups_size":0,"ondisk_log_size":6591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9236,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":365,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.854700+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:53:36.854700+0000","last_peered":"2024-09-13T14:53:36.854700+0000","last_clean":"2024-09-13T14:53:36.854700+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:53:36.854700+0000","last_undegraded":"2024-09-13T14:53:36.854700+0000","last_fullsized":"2024-09-13T14:53:36.854700+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":586,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.851007+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:53:36.851007+0000","last_peered":"2024-09-13T14:53:36.851007+0000","last_clean":"2024-09-13T14:53:36.851007+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:53:36.851007+0000","last_undegraded":"2024-09-13T14:53:36.851007+0000","last_fullsized":"2024-09-13T14:53:36.851007+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":363,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849778+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:53:36.849778+0000","last_peered":"2024-09-13T14:53:36.849778+0000","last_clean":"2024-09-13T14:53:36.849778+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:53:36.849778+0000","last_undegraded":"2024-09-13T14:53:36.849778+0000","last_fullsized":"2024-09-13T14:53:36.849778+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"251'8928","reported_seq":15491,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:44.197784+0000","last_change":"2024-09-13T14:53:05.481759+0000","last_active":"2024-09-13T14:53:44.197784+0000","last_peered":"2024-09-13T14:53:44.197784+0000","last_clean":"2024-09-13T14:53:44.197784+0000","last_became_active":"2024-09-13T14:53:05.465653+0000","last_became_peered":"2024-09-13T14:53:05.465653+0000","last_unstale":"2024-09-13T14:53:44.197784+0000","last_undegraded":"2024-09-13T14:53:44.197784+0000","last_fullsized":"2024-09-13T14:53:44.197784+0000","mapping_epoch":243,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7228,"log_dups_size":0,"ondisk_log_size":7228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:28.220387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103677952,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3214,"num_bytes_recovered":205586432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":383,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850945+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:53:36.850945+0000","last_peered":"2024-09-13T14:53:36.850945+0000","last_clean":"2024-09-13T14:53:36.850945+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:53:36.850945+0000","last_undegraded":"2024-09-13T14:53:36.850945+0000","last_fullsized":"2024-09-13T14:53:36.850945+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":815,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855020+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:53:36.855020+0000","last_peered":"2024-09-13T14:53:36.855020+0000","last_clean":"2024-09-13T14:53:36.855020+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:53:36.855020+0000","last_undegraded":"2024-09-13T14:53:36.855020+0000","last_fullsized":"2024-09-13T14:53:36.855020+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":99,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.847201+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:53:36.847201+0000","last_peered":"2024-09-13T14:53:36.847201+0000","last_clean":"2024-09-13T14:53:36.847201+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:53:36.847201+0000","last_undegraded":"2024-09-13T14:53:36.847201+0000","last_fullsized":"2024-09-13T14:53:36.847201+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"251'9337","reported_seq":10922,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:48.885450+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:53:48.885450+0000","last_peered":"2024-09-13T14:53:48.885450+0000","last_clean":"2024-09-13T14:53:48.885450+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:53:48.885450+0000","last_undegraded":"2024-09-13T14:53:48.885450+0000","last_fullsized":"2024-09-13T14:53:48.885450+0000","mapping_epoch":188,"log_start":"169'1200","ondisk_log_start":"169'1200","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99811328,"num_objects":1523,"num_object_clones":0,"num_object_copies":4569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":358,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850099+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:53:36.850099+0000","last_peered":"2024-09-13T14:53:36.850099+0000","last_clean":"2024-09-13T14:53:36.850099+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:53:36.850099+0000","last_undegraded":"2024-09-13T14:53:36.850099+0000","last_fullsized":"2024-09-13T14:53:36.850099+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":803,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.637074+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:53:26.637074+0000","last_peered":"2024-09-13T14:53:26.637074+0000","last_clean":"2024-09-13T14:53:26.637074+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:53:26.637074+0000","last_undegraded":"2024-09-13T14:53:26.637074+0000","last_fullsized":"2024-09-13T14:53:26.637074+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":238,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615434+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:53:26.615434+0000","last_peered":"2024-09-13T14:53:26.615434+0000","last_clean":"2024-09-13T14:53:26.615434+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:53:26.615434+0000","last_undegraded":"2024-09-13T14:53:26.615434+0000","last_fullsized":"2024-09-13T14:53:26.615434+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":233,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615951+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:53:26.615951+0000","last_peered":"2024-09-13T14:53:26.615951+0000","last_clean":"2024-09-13T14:53:26.615951+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:53:26.615951+0000","last_undegraded":"2024-09-13T14:53:26.615951+0000","last_fullsized":"2024-09-13T14:53:26.615951+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":281,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615652+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:53:26.615652+0000","last_peered":"2024-09-13T14:53:26.615652+0000","last_clean":"2024-09-13T14:53:26.615652+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:53:26.615652+0000","last_undegraded":"2024-09-13T14:53:26.615652+0000","last_fullsized":"2024-09-13T14:53:26.615652+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":362,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.854514+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:53:36.854514+0000","last_peered":"2024-09-13T14:53:36.854514+0000","last_clean":"2024-09-13T14:53:36.854514+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:53:36.854514+0000","last_undegraded":"2024-09-13T14:53:36.854514+0000","last_fullsized":"2024-09-13T14:53:36.854514+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":203,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850511+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:53:36.850511+0000","last_peered":"2024-09-13T14:53:36.850511+0000","last_clean":"2024-09-13T14:53:36.850511+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:53:36.850511+0000","last_undegraded":"2024-09-13T14:53:36.850511+0000","last_fullsized":"2024-09-13T14:53:36.850511+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":114,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640433+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:53:26.640433+0000","last_peered":"2024-09-13T14:53:26.640433+0000","last_clean":"2024-09-13T14:53:26.640433+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:53:26.640433+0000","last_undegraded":"2024-09-13T14:53:26.640433+0000","last_fullsized":"2024-09-13T14:53:26.640433+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":340,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.846689+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:53:36.846689+0000","last_peered":"2024-09-13T14:53:36.846689+0000","last_clean":"2024-09-13T14:53:36.846689+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:53:36.846689+0000","last_undegraded":"2024-09-13T14:53:36.846689+0000","last_fullsized":"2024-09-13T14:53:36.846689+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":394,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850644+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:53:36.850644+0000","last_peered":"2024-09-13T14:53:36.850644+0000","last_clean":"2024-09-13T14:53:36.850644+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:53:36.850644+0000","last_undegraded":"2024-09-13T14:53:36.850644+0000","last_fullsized":"2024-09-13T14:53:36.850644+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":391,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.847488+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:53:36.847488+0000","last_peered":"2024-09-13T14:53:36.847488+0000","last_clean":"2024-09-13T14:53:36.847488+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:53:36.847488+0000","last_undegraded":"2024-09-13T14:53:36.847488+0000","last_fullsized":"2024-09-13T14:53:36.847488+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":144,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.637013+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:53:26.637013+0000","last_peered":"2024-09-13T14:53:26.637013+0000","last_clean":"2024-09-13T14:53:26.637013+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:53:26.637013+0000","last_undegraded":"2024-09-13T14:53:26.637013+0000","last_fullsized":"2024-09-13T14:53:26.637013+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":357,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850562+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:53:36.850562+0000","last_peered":"2024-09-13T14:53:36.850562+0000","last_clean":"2024-09-13T14:53:36.850562+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:53:36.850562+0000","last_undegraded":"2024-09-13T14:53:36.850562+0000","last_fullsized":"2024-09-13T14:53:36.850562+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":374,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850356+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:53:36.850356+0000","last_peered":"2024-09-13T14:53:36.850356+0000","last_clean":"2024-09-13T14:53:36.850356+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:53:36.850356+0000","last_undegraded":"2024-09-13T14:53:36.850356+0000","last_fullsized":"2024-09-13T14:53:36.850356+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":343,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855881+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:53:36.855881+0000","last_peered":"2024-09-13T14:53:36.855881+0000","last_clean":"2024-09-13T14:53:36.855881+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:53:36.855881+0000","last_undegraded":"2024-09-13T14:53:36.855881+0000","last_fullsized":"2024-09-13T14:53:36.855881+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":206,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850383+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:53:36.850383+0000","last_peered":"2024-09-13T14:53:36.850383+0000","last_clean":"2024-09-13T14:53:36.850383+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:53:36.850383+0000","last_undegraded":"2024-09-13T14:53:36.850383+0000","last_fullsized":"2024-09-13T14:53:36.850383+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":232,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615707+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:53:26.615707+0000","last_peered":"2024-09-13T14:53:26.615707+0000","last_clean":"2024-09-13T14:53:26.615707+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:53:26.615707+0000","last_undegraded":"2024-09-13T14:53:26.615707+0000","last_fullsized":"2024-09-13T14:53:26.615707+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":102,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.847318+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:53:36.847318+0000","last_peered":"2024-09-13T14:53:36.847318+0000","last_clean":"2024-09-13T14:53:36.847318+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:53:36.847318+0000","last_undegraded":"2024-09-13T14:53:36.847318+0000","last_fullsized":"2024-09-13T14:53:36.847318+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":367,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.849670+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:53:36.849670+0000","last_peered":"2024-09-13T14:53:36.849670+0000","last_clean":"2024-09-13T14:53:36.849670+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:53:36.849670+0000","last_undegraded":"2024-09-13T14:53:36.849670+0000","last_fullsized":"2024-09-13T14:53:36.849670+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":242,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615877+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:53:26.615877+0000","last_peered":"2024-09-13T14:53:26.615877+0000","last_clean":"2024-09-13T14:53:26.615877+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:53:26.615877+0000","last_undegraded":"2024-09-13T14:53:26.615877+0000","last_fullsized":"2024-09-13T14:53:26.615877+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":89,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640645+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:53:26.640645+0000","last_peered":"2024-09-13T14:53:26.640645+0000","last_clean":"2024-09-13T14:53:26.640645+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:53:26.640645+0000","last_undegraded":"2024-09-13T14:53:26.640645+0000","last_fullsized":"2024-09-13T14:53:26.640645+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":230,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615626+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:53:26.615626+0000","last_peered":"2024-09-13T14:53:26.615626+0000","last_clean":"2024-09-13T14:53:26.615626+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:53:26.615626+0000","last_undegraded":"2024-09-13T14:53:26.615626+0000","last_fullsized":"2024-09-13T14:53:26.615626+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":91,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.640557+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:53:26.640557+0000","last_peered":"2024-09-13T14:53:26.640557+0000","last_clean":"2024-09-13T14:53:26.640557+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:53:26.640557+0000","last_undegraded":"2024-09-13T14:53:26.640557+0000","last_fullsized":"2024-09-13T14:53:26.640557+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":360,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.848768+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:53:36.848768+0000","last_peered":"2024-09-13T14:53:36.848768+0000","last_clean":"2024-09-13T14:53:36.848768+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:53:36.848768+0000","last_undegraded":"2024-09-13T14:53:36.848768+0000","last_fullsized":"2024-09-13T14:53:36.848768+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":377,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.851406+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:53:36.851406+0000","last_peered":"2024-09-13T14:53:36.851406+0000","last_clean":"2024-09-13T14:53:36.851406+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:53:36.851406+0000","last_undegraded":"2024-09-13T14:53:36.851406+0000","last_fullsized":"2024-09-13T14:53:36.851406+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":260,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-13T14:53:26.615422+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:53:26.615422+0000","last_peered":"2024-09-13T14:53:26.615422+0000","last_clean":"2024-09-13T14:53:26.615422+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:53:26.615422+0000","last_undegraded":"2024-09-13T14:53:26.615422+0000","last_fullsized":"2024-09-13T14:53:26.615422+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":430,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850543+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:53:36.850543+0000","last_peered":"2024-09-13T14:53:36.850543+0000","last_clean":"2024-09-13T14:53:36.850543+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:53:36.850543+0000","last_undegraded":"2024-09-13T14:53:36.850543+0000","last_fullsized":"2024-09-13T14:53:36.850543+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1657536536,"num_objects":25293,"num_object_clones":0,"num_object_copies":75879,"num_objects_missing_on_primary":67,"num_objects_missing":67,"num_objects_degraded":544,"num_objects_misplaced":1593,"num_objects_unfound":0,"num_objects_dirty":25293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146255,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44657,"num_bytes_recovered":2807300096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5180436480,"data_stored":5180424264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123427,"ondisk_log_size":123427,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234316,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495953,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036208,"kb_used_data":690088,"kb_used_omap":1,"kb_used_meta":346110,"kb_avail":92708944,"statfs":{"total":95995035648,"available":94933958656,"internally_reserved":0,"allocated":706650112,"data_stored":705293836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354417095},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757587,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742888,"kb_used_data":527200,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":93002264,"statfs":{"total":95995035648,"available":95234318336,"internally_reserved":0,"allocated":539852800,"data_stored":538500319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":220854727},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019227,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837616,"kb_used_data":531688,"kb_used_omap":1,"kb_used_meta":305918,"kb_avail":92907536,"statfs":{"total":95995035648,"available":95137316864,"internally_reserved":0,"allocated":544448512,"data_stored":543092297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":313260486},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468112,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1420580,"kb_used_data":1127900,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":92324572,"statfs":{"total":95995035648,"available":94540361728,"internally_reserved":0,"allocated":1154969600,"data_stored":1153620965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":299625636},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,0,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631641,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1342260,"kb_used_data":945452,"kb_used_omap":8,"kb_used_meta":396791,"kb_avail":92402892,"statfs":{"total":95995035648,"available":94620561408,"internally_reserved":0,"allocated":968142848,"data_stored":966783063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":406314975},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795164,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1179756,"kb_used_data":877348,"kb_used_omap":7,"kb_used_meta":302392,"kb_avail":92565396,"statfs":{"total":95995035648,"available":94786965504,"internally_reserved":0,"allocated":898404352,"data_stored":897047974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":309650034},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958695,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":690416,"kb_used_data":397736,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":93054736,"statfs":{"total":95995035648,"available":95288049664,"internally_reserved":0,"allocated":407281664,"data_stored":405921843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":299688423},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":392105984,"data_stored":392101912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":880283648,"data_stored":880279576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":960495616,"data_stored":960495616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1145307136,"data_stored":1145307136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":531693568,"data_stored":531693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":531566592,"data_stored":531562520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":687013888,"data_stored":687013888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":51970048,"data_stored":51970048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:53:51.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:52.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:52.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:52.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:52.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:53:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:53:51] "GET /metrics HTTP/1.1" 200 36518 "" "Prometheus/2.51.0" 2024-09-13T14:53:52.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:52.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:52.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:52.404 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:52.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:52.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:52.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:52.652 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:52 smithi172 ceph-mon[100696]: pgmap v755: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 552 op/s; 729/69606 objects degraded (1.047%); 1408/69606 objects misplaced (2.023%) 2024-09-13T14:53:52.652 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:52.652 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:52.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:52 smithi154 ceph-mon[102314]: pgmap v755: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 552 op/s; 729/69606 objects degraded (1.047%); 1408/69606 objects misplaced (2.023%) 2024-09-13T14:53:52.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:52 smithi154 ceph-mon[107038]: pgmap v755: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 552 op/s; 729/69606 objects degraded (1.047%); 1408/69606 objects misplaced (2.023%) 2024-09-13T14:53:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:52.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.799 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:52.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:52.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:52.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:52.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:53.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:53.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:53.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:53.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:53.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.331 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:53.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:53.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:53.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:53.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:53.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:53.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.734 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:53.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:53.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:53.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:53.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:53.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:53.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:54.088 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:53:54.088 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:53:54.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:54.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:54.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:54.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:54.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:54.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:54.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:54 smithi154 ceph-mon[102314]: pgmap v756: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 552 op/s; 729/69606 objects degraded (1.047%); 1408/69606 objects misplaced (2.023%) 2024-09-13T14:53:54.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:54.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:54 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:54.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:54 smithi154 ceph-mon[107038]: pgmap v756: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 552 op/s; 729/69606 objects degraded (1.047%); 1408/69606 objects misplaced (2.023%) 2024-09-13T14:53:54.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:54.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:54 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:54.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:54 smithi172 ceph-mon[100696]: pgmap v756: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 552 op/s; 729/69606 objects degraded (1.047%); 1408/69606 objects misplaced (2.023%) 2024-09-13T14:53:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:54 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:53:54.499 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:54.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:54.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:54.711 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:53:54.711 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:53:54.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:54.746 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:53:54.747 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:53:54.749 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:54.777 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:53:54.778 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:53:54.807 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:53:54.807 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:53:54.834 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:53:54.834 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:53:54.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:54.864 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:53:54.864 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:53:54.890 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:53:54.891 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:53:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:54.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:54.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:54.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:53:54.920 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:53:54.921 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:53:55.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:55.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:55.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:55.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:55.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.329 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:55.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:55.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:55.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:55.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3682079167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:55.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 729/69606 objects degraded (1.047%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:55.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3682079167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:55.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 729/69606 objects degraded (1.047%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:55.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:55.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:55.658 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3682079167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:53:55.658 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 729/69606 objects degraded (1.047%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:53:55.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:55.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:55.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:55.870 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:55.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:55.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:55.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:53:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:56.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:56.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:56.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:53:56.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:56.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:56.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:56.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.422 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:56.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:56 smithi154 ceph-mon[102314]: pgmap v757: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 571 op/s; 729/62559 objects degraded (1.165%); 1408/62559 objects misplaced (2.251%) 2024-09-13T14:53:56.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:56 smithi154 ceph-mon[107038]: pgmap v757: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 571 op/s; 729/62559 objects degraded (1.165%); 1408/62559 objects misplaced (2.251%) 2024-09-13T14:53:56.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:56.545 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:56.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:56 smithi172 ceph-mon[100696]: pgmap v757: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.3 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 571 op/s; 729/62559 objects degraded (1.165%); 1408/62559 objects misplaced (2.251%) 2024-09-13T14:53:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:56.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:56.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:56.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:56.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:56.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:56.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:56.865 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:53:56.911 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:53:56.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:56.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:56.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:57.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:57.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:57.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:57.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:57.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:57.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:57.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:57.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:57.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:57.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:57.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:57.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:57.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:53:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:57.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:57.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:53:58.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:58.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:58.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:53:58.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:58.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:58.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:58.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:58.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:58.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.435 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:58.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:58 smithi154 ceph-mon[102314]: pgmap v758: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.2 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 576 op/s; 918/60825 objects degraded (1.509%); 1219/60825 objects misplaced (2.004%); 0 B/s, 0 objects/s recovering 2024-09-13T14:53:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:53:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:58 smithi154 ceph-mon[107038]: pgmap v758: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.2 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 576 op/s; 918/60825 objects degraded (1.509%); 1219/60825 objects misplaced (2.004%); 0 B/s, 0 objects/s recovering 2024-09-13T14:53:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:53:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:58.498 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:58 smithi172 ceph-mon[100696]: pgmap v758: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.2 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 576 op/s; 918/60825 objects degraded (1.509%); 1219/60825 objects misplaced (2.004%); 0 B/s, 0 objects/s recovering 2024-09-13T14:53:58.499 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:53:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:53:58.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:58.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:58.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:58.701 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:58.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:53:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:58.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:53:58.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:58.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:58.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:59.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:59.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:59.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:53:59.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:53:59.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:53:59.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:53:59.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:53:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:59.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:53:59.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:53:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:59.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:53:59.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:53:59.752 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:53:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:53:59.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:53:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:53:59.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:53:59.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:53:59.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:00.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:00.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:00.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:00.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:00.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:00.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:00.427 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:00 smithi172 ceph-mon[100696]: pgmap v759: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 918/58704 objects degraded (1.564%); 1219/58704 objects misplaced (2.077%); 0 B/s, 0 objects/s recovering 2024-09-13T14:54:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 918/58704 objects degraded (1.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:00.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:00.557 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:00.574 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:00 smithi154 ceph-mon[102314]: pgmap v759: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 918/58704 objects degraded (1.564%); 1219/58704 objects misplaced (2.077%); 0 B/s, 0 objects/s recovering 2024-09-13T14:54:00.574 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 918/58704 objects degraded (1.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:00.575 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:00 smithi154 ceph-mon[107038]: pgmap v759: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.2 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 918/58704 objects degraded (1.564%); 1219/58704 objects misplaced (2.077%); 0 B/s, 0 objects/s recovering 2024-09-13T14:54:00.575 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 918/58704 objects degraded (1.564%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:00.638 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:155000 6.d 6:b3157578:::benchmark_data_smithi154_175640_object69224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:54:00.628723+0000", 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0081378160000000008, 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081602580000000001, 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:00.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 155000 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628723+0000", 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628723+0000", 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628722+0000", 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628728+0000", 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.271e-06 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628741+0000", 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3371e-05 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628748+0000", 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.435e-06 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628797+0000", 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8253e-05 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628892+0000", 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5816999999999994e-05 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.628981+0000", 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8537999999999998e-05 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.636474+0000", 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074934759999999998 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:155001 6.d 6:b99a293c:::benchmark_data_smithi154_175640_object69225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:54:00.629050+0000", 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0078106460000000001, 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078825820000000008, 2024-09-13T14:54:00.645 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 155001 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.629050+0000", 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:00.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.629050+0000", 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.629050+0000", 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.629054+0000", 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4950000000000002e-06 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.629065+0000", 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0499000000000001e-05 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.629071+0000", 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8410000000000002e-06 2024-09-13T14:54:00.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.633120+0000", 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040483200000000002 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.633237+0000", 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011710900000000001 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:54:00.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.633313+0000", 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5970000000000006e-05 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:54:00.636609+0000", 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032962460000000001 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 2 2024-09-13T14:54:00.650 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:54:00.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:00.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:00.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:00.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:00.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:00.987 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:00.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:01.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:01.111 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:01.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:01.234 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:01.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:01.340 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:54:01.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4146808366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:01 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4146808366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:01 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:01.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:01.507 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:01.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:01.647 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4146808366' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:01 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:01.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:01.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:01.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:01.904 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:01.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:02.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:02.036 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:02.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:02.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:54:02.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:02.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:54:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:54:01] "GET /metrics HTTP/1.1" 200 36515 "" "Prometheus/2.51.0" 2024-09-13T14:54:02.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:02.296 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:02.324 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:54:02.326 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:54:02.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:02.404 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:02.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:02.480 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[102314]: pgmap v760: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 1102/52170 objects degraded (2.112%); 1035/52170 objects misplaced (1.984%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:02.480 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[102314]: osdmap e253: 8 total, 8 up, 7 in 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[107038]: pgmap v760: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 1102/52170 objects degraded (2.112%); 1035/52170 objects misplaced (1.984%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:02.481 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[107038]: osdmap e253: 8 total, 8 up, 7 in 2024-09-13T14:54:02.482 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:02.482 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:02 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:02 smithi172 ceph-mon[100696]: pgmap v760: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 1102/52170 objects degraded (2.112%); 1035/52170 objects misplaced (1.984%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:02 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:02 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:02 smithi172 ceph-mon[100696]: osdmap e253: 8 total, 8 up, 7 in 2024-09-13T14:54:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:02 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:02.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:02.564 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:02.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:02.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:02.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:02.826 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:02.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:02.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:02.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:03.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.077 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:03.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:03.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:03.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:03.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:03.331 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:03.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:03.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:03.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:03.574 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":761,"stamp":"2024-09-13T14:54:01.334467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1142560765,"num_objects":17390,"num_object_clones":0,"num_object_copies":52170,"num_objects_missing_on_primary":54,"num_objects_missing":54,"num_objects_degraded":1102,"num_objects_misplaced":1035,"num_objects_unfound":0,"num_objects_dirty":17390,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":155309,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44826,"num_bytes_recovered":2809678981,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127140,"ondisk_log_size":127140,"up":267,"acting":266,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":269,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5745104,"kb_used_data":3617368,"kb_used_omap":32,"kb_used_meta":2127391,"kb_avail":650470960,"statfs":{"total":671965249536,"available":666082263040,"internally_reserved":0,"allocated":3704184832,"data_stored":3694667633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2178448944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,6,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-380895232,"num_objects":-5812,"num_object_clones":0,"num_object_copies":-17436,"num_objects_missing_on_primary":-13,"num_objects_missing":-13,"num_objects_degraded":373,"num_objects_misplaced":-373,"num_objects_unfound":0,"num_objects_dirty":-5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.320235"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":386,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.142942+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:53:50.142942+0000","last_peered":"2024-09-13T14:53:50.142942+0000","last_clean":"2024-09-13T14:53:50.142942+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:53:50.142942+0000","last_undegraded":"2024-09-13T14:53:50.142942+0000","last_fullsized":"2024-09-13T14:53:50.142942+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":118,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839961+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:53:36.839961+0000","last_peered":"2024-09-13T14:53:36.839961+0000","last_clean":"2024-09-13T14:53:36.839961+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:53:36.839961+0000","last_undegraded":"2024-09-13T14:53:36.839961+0000","last_fullsized":"2024-09-13T14:53:36.839961+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":88,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855630+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:53:36.855630+0000","last_peered":"2024-09-13T14:53:36.855630+0000","last_clean":"2024-09-13T14:53:36.855630+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:53:36.855630+0000","last_undegraded":"2024-09-13T14:53:36.855630+0000","last_fullsized":"2024-09-13T14:53:36.855630+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":375,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098957+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:53:50.098957+0000","last_peered":"2024-09-13T14:53:50.098957+0000","last_clean":"2024-09-13T14:53:50.098957+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:53:50.098957+0000","last_undegraded":"2024-09-13T14:53:50.098957+0000","last_fullsized":"2024-09-13T14:53:50.098957+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":375,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098336+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:53:50.098336+0000","last_peered":"2024-09-13T14:53:50.098336+0000","last_clean":"2024-09-13T14:53:50.098336+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:53:50.098336+0000","last_undegraded":"2024-09-13T14:53:50.098336+0000","last_fullsized":"2024-09-13T14:53:50.098336+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":188,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839689+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:53:36.839689+0000","last_peered":"2024-09-13T14:53:36.839689+0000","last_clean":"2024-09-13T14:53:36.839689+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:53:36.839689+0000","last_undegraded":"2024-09-13T14:53:36.839689+0000","last_fullsized":"2024-09-13T14:53:36.839689+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":101,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095419+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:53:50.095419+0000","last_peered":"2024-09-13T14:53:50.095419+0000","last_clean":"2024-09-13T14:53:50.095419+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:53:50.095419+0000","last_undegraded":"2024-09-13T14:53:50.095419+0000","last_fullsized":"2024-09-13T14:53:50.095419+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":164,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850239+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:53:36.850239+0000","last_peered":"2024-09-13T14:53:36.850239+0000","last_clean":"2024-09-13T14:53:36.850239+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:53:36.850239+0000","last_undegraded":"2024-09-13T14:53:36.850239+0000","last_fullsized":"2024-09-13T14:53:36.850239+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"252'9787","reported_seq":10634,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.579080+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:53:59.579080+0000","last_peered":"2024-09-13T14:53:59.579080+0000","last_clean":"2024-09-13T14:53:59.579080+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:53:59.579080+0000","last_undegraded":"2024-09-13T14:53:59.579080+0000","last_fullsized":"2024-09-13T14:53:59.579080+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":9787,"log_dups_size":0,"ondisk_log_size":9787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72679424,"num_objects":1109,"num_object_clones":0,"num_object_copies":3327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9787,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":310,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.094144+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:53:50.094144+0000","last_peered":"2024-09-13T14:53:50.094144+0000","last_clean":"2024-09-13T14:53:50.094144+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:53:50.094144+0000","last_undegraded":"2024-09-13T14:53:50.094144+0000","last_fullsized":"2024-09-13T14:53:50.094144+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":115,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855345+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:53:36.855345+0000","last_peered":"2024-09-13T14:53:36.855345+0000","last_clean":"2024-09-13T14:53:36.855345+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:53:36.855345+0000","last_undegraded":"2024-09-13T14:53:36.855345+0000","last_fullsized":"2024-09-13T14:53:36.855345+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"252'9549","reported_seq":11182,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:54:00.207454+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:54:00.207454+0000","last_peered":"2024-09-13T14:54:00.207454+0000","last_clean":"2024-09-13T14:54:00.207454+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:54:00.207454+0000","last_undegraded":"2024-09-13T14:54:00.207454+0000","last_fullsized":"2024-09-13T14:54:00.207454+0000","mapping_epoch":170,"log_start":"181'3100","ondisk_log_start":"181'3100","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67436544,"num_objects":1029,"num_object_clones":0,"num_object_copies":3087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":116,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.840126+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:53:36.840126+0000","last_peered":"2024-09-13T14:53:36.840126+0000","last_clean":"2024-09-13T14:53:36.840126+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:53:36.840126+0000","last_undegraded":"2024-09-13T14:53:36.840126+0000","last_fullsized":"2024-09-13T14:53:36.840126+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":268,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.840116+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:53:36.840116+0000","last_peered":"2024-09-13T14:53:36.840116+0000","last_clean":"2024-09-13T14:53:36.840116+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:53:36.840116+0000","last_undegraded":"2024-09-13T14:53:36.840116+0000","last_fullsized":"2024-09-13T14:53:36.840116+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"252'9734","reported_seq":11460,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.279684+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:53:59.279684+0000","last_peered":"2024-09-13T14:53:59.279684+0000","last_clean":"2024-09-13T14:53:59.279684+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:53:59.279684+0000","last_undegraded":"2024-09-13T14:53:59.279684+0000","last_fullsized":"2024-09-13T14:53:59.279684+0000","mapping_epoch":175,"log_start":"168'900","ondisk_log_start":"168'900","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8834,"log_dups_size":0,"ondisk_log_size":8834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9734,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":92,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855734+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:53:36.855734+0000","last_peered":"2024-09-13T14:53:36.855734+0000","last_clean":"2024-09-13T14:53:36.855734+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:53:36.855734+0000","last_undegraded":"2024-09-13T14:53:36.855734+0000","last_fullsized":"2024-09-13T14:53:36.855734+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":48,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850273+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:53:36.850273+0000","last_peered":"2024-09-13T14:53:36.850273+0000","last_clean":"2024-09-13T14:53:36.850273+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:53:36.850273+0000","last_undegraded":"2024-09-13T14:53:36.850273+0000","last_fullsized":"2024-09-13T14:53:36.850273+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"252'9427","reported_seq":15548,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:58.012093+0000","last_change":"2024-09-13T14:52:50.434519+0000","last_active":"2024-09-13T14:53:58.012093+0000","last_peered":"2024-09-13T14:53:58.012093+0000","last_clean":"2024-09-13T14:53:58.012093+0000","last_became_active":"2024-09-13T14:52:50.417699+0000","last_became_peered":"2024-09-13T14:52:50.417699+0000","last_unstale":"2024-09-13T14:53:58.012093+0000","last_undegraded":"2024-09-13T14:53:58.012093+0000","last_fullsized":"2024-09-13T14:53:58.012093+0000","mapping_epoch":238,"log_start":"169'1000","ondisk_log_start":"169'1000","created":160,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8427,"log_dups_size":0,"ondisk_log_size":8427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:10.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70844416,"num_objects":1081,"num_object_clones":0,"num_object_copies":3243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2948,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":104,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855656+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:53:36.855656+0000","last_peered":"2024-09-13T14:53:36.855656+0000","last_clean":"2024-09-13T14:53:36.855656+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:53:36.855656+0000","last_undegraded":"2024-09-13T14:53:36.855656+0000","last_fullsized":"2024-09-13T14:53:36.855656+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":355,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098357+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:53:50.098357+0000","last_peered":"2024-09-13T14:53:50.098357+0000","last_clean":"2024-09-13T14:53:50.098357+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:53:50.098357+0000","last_undegraded":"2024-09-13T14:53:50.098357+0000","last_fullsized":"2024-09-13T14:53:50.098357+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"252'9657","reported_seq":26921,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.175094+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:53:59.175094+0000","last_peered":"2024-09-13T14:53:59.175094+0000","last_clean":"2024-09-13T14:53:59.175094+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:53:59.175094+0000","last_undegraded":"2024-09-13T14:53:59.175094+0000","last_fullsized":"2024-09-13T14:53:59.175094+0000","mapping_epoch":232,"log_start":"174'2100","ondisk_log_start":"174'2100","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71892992,"num_objects":1097,"num_object_clones":0,"num_object_copies":3291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9657,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":363,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095040+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:53:50.095040+0000","last_peered":"2024-09-13T14:53:50.095040+0000","last_clean":"2024-09-13T14:53:50.095040+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:53:50.095040+0000","last_undegraded":"2024-09-13T14:53:50.095040+0000","last_fullsized":"2024-09-13T14:53:50.095040+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":235,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839895+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:53:36.839895+0000","last_peered":"2024-09-13T14:53:36.839895+0000","last_clean":"2024-09-13T14:53:36.839895+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:53:36.839895+0000","last_undegraded":"2024-09-13T14:53:36.839895+0000","last_fullsized":"2024-09-13T14:53:36.839895+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"252'9540","reported_seq":14552,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.204899+0000","last_change":"2024-09-13T14:53:24.611648+0000","last_active":"2024-09-13T14:53:59.204899+0000","last_peered":"2024-09-13T14:53:59.204899+0000","last_clean":"2024-09-13T14:53:59.204899+0000","last_became_active":"2024-09-13T14:53:24.585428+0000","last_became_peered":"2024-09-13T14:53:24.585428+0000","last_unstale":"2024-09-13T14:53:59.204899+0000","last_undegraded":"2024-09-13T14:53:59.204899+0000","last_fullsized":"2024-09-13T14:53:59.204899+0000","mapping_epoch":248,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7840,"log_dups_size":0,"ondisk_log_size":7840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:41.636547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9540,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":147914752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":136,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:38.896097+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:53:38.896097+0000","last_peered":"2024-09-13T14:53:38.896097+0000","last_clean":"2024-09-13T14:53:38.896097+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:53:38.896097+0000","last_undegraded":"2024-09-13T14:53:38.896097+0000","last_fullsized":"2024-09-13T14:53:38.896097+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":232,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.840136+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:53:36.840136+0000","last_peered":"2024-09-13T14:53:36.840136+0000","last_clean":"2024-09-13T14:53:36.840136+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:53:36.840136+0000","last_undegraded":"2024-09-13T14:53:36.840136+0000","last_fullsized":"2024-09-13T14:53:36.840136+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"252'9778","reported_seq":14547,"reported_epoch":252,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:54:00.245263+0000","last_change":"2024-09-13T14:53:29.139613+0000","last_active":"2024-09-13T14:54:00.245263+0000","last_peered":"2024-09-13T14:54:00.245263+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:54:00.245263+0000","last_undegraded":"2024-09-13T14:53:29.127369+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"184'3300","ondisk_log_start":"184'3300","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6478,"log_dups_size":0,"ondisk_log_size":6478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68419584,"num_objects":1044,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":54,"num_objects_missing":54,"num_objects_degraded":1102,"num_objects_misplaced":1035,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9778,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2399,"num_bytes_recovered":155582464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":1034},{"shards":"3,5","objects":-1093}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":227,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839834+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:53:36.839834+0000","last_peered":"2024-09-13T14:53:36.839834+0000","last_clean":"2024-09-13T14:53:36.839834+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:53:36.839834+0000","last_undegraded":"2024-09-13T14:53:36.839834+0000","last_fullsized":"2024-09-13T14:53:36.839834+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":262,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.840212+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:53:36.840212+0000","last_peered":"2024-09-13T14:53:36.840212+0000","last_clean":"2024-09-13T14:53:36.840212+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:53:36.840212+0000","last_undegraded":"2024-09-13T14:53:36.840212+0000","last_fullsized":"2024-09-13T14:53:36.840212+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"252'9419","reported_seq":13859,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.520307+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:53:59.520307+0000","last_peered":"2024-09-13T14:53:59.520307+0000","last_clean":"2024-09-13T14:53:59.520307+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:53:59.520307+0000","last_undegraded":"2024-09-13T14:53:59.520307+0000","last_fullsized":"2024-09-13T14:53:59.520307+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71172120,"num_objects":1087,"num_object_clones":0,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9419,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":101,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095627+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:53:50.095627+0000","last_peered":"2024-09-13T14:53:50.095627+0000","last_clean":"2024-09-13T14:53:50.095627+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:53:50.095627+0000","last_undegraded":"2024-09-13T14:53:50.095627+0000","last_fullsized":"2024-09-13T14:53:50.095627+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":211,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.094271+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:53:50.094271+0000","last_peered":"2024-09-13T14:53:50.094271+0000","last_clean":"2024-09-13T14:53:50.094271+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:53:50.094271+0000","last_undegraded":"2024-09-13T14:53:50.094271+0000","last_fullsized":"2024-09-13T14:53:50.094271+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"252'9660","reported_seq":10921,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.501926+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:53:59.501926+0000","last_peered":"2024-09-13T14:53:59.501926+0000","last_clean":"2024-09-13T14:53:59.501926+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:53:59.501926+0000","last_undegraded":"2024-09-13T14:53:59.501926+0000","last_fullsized":"2024-09-13T14:53:59.501926+0000","mapping_epoch":204,"log_start":"164'165","ondisk_log_start":"164'165","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":9495,"log_dups_size":0,"ondisk_log_size":9495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9660,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":557,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.097510+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:53:50.097510+0000","last_peered":"2024-09-13T14:53:50.097510+0000","last_clean":"2024-09-13T14:53:50.097510+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:53:50.097510+0000","last_undegraded":"2024-09-13T14:53:50.097510+0000","last_fullsized":"2024-09-13T14:53:50.097510+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":819,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855427+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:53:36.855427+0000","last_peered":"2024-09-13T14:53:36.855427+0000","last_clean":"2024-09-13T14:53:36.855427+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:53:36.855427+0000","last_undegraded":"2024-09-13T14:53:36.855427+0000","last_fullsized":"2024-09-13T14:53:36.855427+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":321,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095051+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:53:50.095051+0000","last_peered":"2024-09-13T14:53:50.095051+0000","last_clean":"2024-09-13T14:53:50.095051+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:53:50.095051+0000","last_undegraded":"2024-09-13T14:53:50.095051+0000","last_fullsized":"2024-09-13T14:53:50.095051+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":48,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850610+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:53:36.850610+0000","last_peered":"2024-09-13T14:53:36.850610+0000","last_clean":"2024-09-13T14:53:36.850610+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:53:36.850610+0000","last_undegraded":"2024-09-13T14:53:36.850610+0000","last_fullsized":"2024-09-13T14:53:36.850610+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"252'9745","reported_seq":14493,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.665291+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:53:59.665291+0000","last_peered":"2024-09-13T14:53:59.665291+0000","last_clean":"2024-09-13T14:53:59.665291+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:53:59.665291+0000","last_undegraded":"2024-09-13T14:53:59.665291+0000","last_fullsized":"2024-09-13T14:53:59.665291+0000","mapping_epoch":223,"log_start":"184'3218","ondisk_log_start":"184'3218","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6527,"log_dups_size":0,"ondisk_log_size":6527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72286208,"num_objects":1103,"num_object_clones":0,"num_object_copies":3309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":402,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.099842+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:53:50.099842+0000","last_peered":"2024-09-13T14:53:50.099842+0000","last_clean":"2024-09-13T14:53:50.099842+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:53:50.099842+0000","last_undegraded":"2024-09-13T14:53:50.099842+0000","last_fullsized":"2024-09-13T14:53:50.099842+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":369,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.097931+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:53:50.097931+0000","last_peered":"2024-09-13T14:53:50.097931+0000","last_clean":"2024-09-13T14:53:50.097931+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:53:50.097931+0000","last_undegraded":"2024-09-13T14:53:50.097931+0000","last_fullsized":"2024-09-13T14:53:50.097931+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":380,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098514+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:53:50.098514+0000","last_peered":"2024-09-13T14:53:50.098514+0000","last_clean":"2024-09-13T14:53:50.098514+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:53:50.098514+0000","last_undegraded":"2024-09-13T14:53:50.098514+0000","last_fullsized":"2024-09-13T14:53:50.098514+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"252'9546","reported_seq":19874,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:58.827060+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:53:58.827060+0000","last_peered":"2024-09-13T14:53:58.827060+0000","last_clean":"2024-09-13T14:53:58.827060+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:53:58.827060+0000","last_undegraded":"2024-09-13T14:53:58.827060+0000","last_fullsized":"2024-09-13T14:53:58.827060+0000","mapping_epoch":235,"log_start":"169'1400","ondisk_log_start":"169'1400","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8146,"log_dups_size":0,"ondisk_log_size":8146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68812800,"num_objects":1050,"num_object_clones":0,"num_object_copies":3150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9546,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":905,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.093717+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:53:50.093717+0000","last_peered":"2024-09-13T14:53:50.093717+0000","last_clean":"2024-09-13T14:53:50.093717+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:53:50.093717+0000","last_undegraded":"2024-09-13T14:53:50.093717+0000","last_fullsized":"2024-09-13T14:53:50.093717+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":360,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.100520+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:53:50.100520+0000","last_peered":"2024-09-13T14:53:50.100520+0000","last_clean":"2024-09-13T14:53:50.100520+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:53:50.100520+0000","last_undegraded":"2024-09-13T14:53:50.100520+0000","last_fullsized":"2024-09-13T14:53:50.100520+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":113,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.102191+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:53:50.102191+0000","last_peered":"2024-09-13T14:53:50.102191+0000","last_clean":"2024-09-13T14:53:50.102191+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:53:50.102191+0000","last_undegraded":"2024-09-13T14:53:50.102191+0000","last_fullsized":"2024-09-13T14:53:50.102191+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"252'9703","reported_seq":19044,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:54:00.237101+0000","last_change":"2024-09-13T14:52:52.443442+0000","last_active":"2024-09-13T14:54:00.237101+0000","last_peered":"2024-09-13T14:54:00.237101+0000","last_clean":"2024-09-13T14:54:00.237101+0000","last_became_active":"2024-09-13T14:52:52.421083+0000","last_became_peered":"2024-09-13T14:52:52.421083+0000","last_unstale":"2024-09-13T14:54:00.237101+0000","last_undegraded":"2024-09-13T14:54:00.237101+0000","last_fullsized":"2024-09-13T14:54:00.237101+0000","mapping_epoch":240,"log_start":"184'3300","ondisk_log_start":"184'3300","created":160,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:14.887813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66650112,"num_objects":1017,"num_object_clones":0,"num_object_copies":3051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4569,"num_bytes_recovered":287244288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":106,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855436+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:53:36.855436+0000","last_peered":"2024-09-13T14:53:36.855436+0000","last_clean":"2024-09-13T14:53:36.855436+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:53:36.855436+0000","last_undegraded":"2024-09-13T14:53:36.855436+0000","last_fullsized":"2024-09-13T14:53:36.855436+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":237,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.840150+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:53:36.840150+0000","last_peered":"2024-09-13T14:53:36.840150+0000","last_clean":"2024-09-13T14:53:36.840150+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:53:36.840150+0000","last_undegraded":"2024-09-13T14:53:36.840150+0000","last_fullsized":"2024-09-13T14:53:36.840150+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855403+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:53:36.855403+0000","last_peered":"2024-09-13T14:53:36.855403+0000","last_clean":"2024-09-13T14:53:36.855403+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:53:36.855403+0000","last_undegraded":"2024-09-13T14:53:36.855403+0000","last_fullsized":"2024-09-13T14:53:36.855403+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"252'9677","reported_seq":14289,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:57.814442+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:53:57.814442+0000","last_peered":"2024-09-13T14:53:57.814442+0000","last_clean":"2024-09-13T14:53:57.814442+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:53:57.814442+0000","last_undegraded":"2024-09-13T14:53:57.814442+0000","last_fullsized":"2024-09-13T14:53:57.814442+0000","mapping_epoch":226,"log_start":"169'1300","ondisk_log_start":"169'1300","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8377,"log_dups_size":0,"ondisk_log_size":8377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74514432,"num_objects":1137,"num_object_clones":0,"num_object_copies":3411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9677,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":106,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855529+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:53:36.855529+0000","last_peered":"2024-09-13T14:53:36.855529+0000","last_clean":"2024-09-13T14:53:36.855529+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:53:36.855529+0000","last_undegraded":"2024-09-13T14:53:36.855529+0000","last_fullsized":"2024-09-13T14:53:36.855529+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":915,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.102098+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:53:50.102098+0000","last_peered":"2024-09-13T14:53:50.102098+0000","last_clean":"2024-09-13T14:53:50.102098+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:53:50.102098+0000","last_undegraded":"2024-09-13T14:53:50.102098+0000","last_fullsized":"2024-09-13T14:53:50.102098+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098786+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:53:50.098786+0000","last_peered":"2024-09-13T14:53:50.098786+0000","last_clean":"2024-09-13T14:53:50.098786+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:53:50.098786+0000","last_undegraded":"2024-09-13T14:53:50.098786+0000","last_fullsized":"2024-09-13T14:53:50.098786+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"252'9809","reported_seq":14549,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.680055+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:53:59.680055+0000","last_peered":"2024-09-13T14:53:59.680055+0000","last_clean":"2024-09-13T14:53:59.680055+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:53:59.680055+0000","last_undegraded":"2024-09-13T14:53:59.680055+0000","last_fullsized":"2024-09-13T14:53:59.680055+0000","mapping_epoch":218,"log_start":"181'3245","ondisk_log_start":"181'3245","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6564,"log_dups_size":0,"ondisk_log_size":6564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":367,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.101991+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:53:50.101991+0000","last_peered":"2024-09-13T14:53:50.101991+0000","last_clean":"2024-09-13T14:53:50.101991+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:53:50.101991+0000","last_undegraded":"2024-09-13T14:53:50.101991+0000","last_fullsized":"2024-09-13T14:53:50.101991+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":588,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.101109+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:53:50.101109+0000","last_peered":"2024-09-13T14:53:50.101109+0000","last_clean":"2024-09-13T14:53:50.101109+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:53:50.101109+0000","last_undegraded":"2024-09-13T14:53:50.101109+0000","last_fullsized":"2024-09-13T14:53:50.101109+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":365,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098281+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:53:50.098281+0000","last_peered":"2024-09-13T14:53:50.098281+0000","last_clean":"2024-09-13T14:53:50.098281+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:53:50.098281+0000","last_undegraded":"2024-09-13T14:53:50.098281+0000","last_fullsized":"2024-09-13T14:53:50.098281+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"252'9424","reported_seq":15989,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:59.187960+0000","last_change":"2024-09-13T14:53:05.481759+0000","last_active":"2024-09-13T14:53:59.187960+0000","last_peered":"2024-09-13T14:53:59.187960+0000","last_clean":"2024-09-13T14:53:59.187960+0000","last_became_active":"2024-09-13T14:53:05.465653+0000","last_became_peered":"2024-09-13T14:53:05.465653+0000","last_unstale":"2024-09-13T14:53:59.187960+0000","last_undegraded":"2024-09-13T14:53:59.187960+0000","last_fullsized":"2024-09-13T14:53:59.187960+0000","mapping_epoch":243,"log_start":"173'1900","ondisk_log_start":"173'1900","created":160,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:28.220387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71172096,"num_objects":1086,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9424,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3214,"num_bytes_recovered":205586432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":385,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095783+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:53:50.095783+0000","last_peered":"2024-09-13T14:53:50.095783+0000","last_clean":"2024-09-13T14:53:50.095783+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:53:50.095783+0000","last_undegraded":"2024-09-13T14:53:50.095783+0000","last_fullsized":"2024-09-13T14:53:50.095783+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":817,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.102464+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:53:50.102464+0000","last_peered":"2024-09-13T14:53:50.102464+0000","last_clean":"2024-09-13T14:53:50.102464+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:53:50.102464+0000","last_undegraded":"2024-09-13T14:53:50.102464+0000","last_fullsized":"2024-09-13T14:53:50.102464+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":101,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095539+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:53:50.095539+0000","last_peered":"2024-09-13T14:53:50.095539+0000","last_clean":"2024-09-13T14:53:50.095539+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:53:50.095539+0000","last_undegraded":"2024-09-13T14:53:50.095539+0000","last_fullsized":"2024-09-13T14:53:50.095539+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"252'9731","reported_seq":11317,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:58.829549+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:53:58.829549+0000","last_peered":"2024-09-13T14:53:58.829549+0000","last_clean":"2024-09-13T14:53:58.829549+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:53:58.829549+0000","last_undegraded":"2024-09-13T14:53:58.829549+0000","last_fullsized":"2024-09-13T14:53:58.829549+0000","mapping_epoch":188,"log_start":"171'1600","ondisk_log_start":"171'1600","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8131,"log_dups_size":0,"ondisk_log_size":8131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73990144,"num_objects":1129,"num_object_clones":0,"num_object_copies":3387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9731,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":360,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098233+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:53:50.098233+0000","last_peered":"2024-09-13T14:53:50.098233+0000","last_clean":"2024-09-13T14:53:50.098233+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:53:50.098233+0000","last_undegraded":"2024-09-13T14:53:50.098233+0000","last_fullsized":"2024-09-13T14:53:50.098233+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":804,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850551+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:53:36.850551+0000","last_peered":"2024-09-13T14:53:36.850551+0000","last_clean":"2024-09-13T14:53:36.850551+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:53:36.850551+0000","last_undegraded":"2024-09-13T14:53:36.850551+0000","last_fullsized":"2024-09-13T14:53:36.850551+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":239,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839735+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:53:36.839735+0000","last_peered":"2024-09-13T14:53:36.839735+0000","last_clean":"2024-09-13T14:53:36.839735+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:53:36.839735+0000","last_undegraded":"2024-09-13T14:53:36.839735+0000","last_fullsized":"2024-09-13T14:53:36.839735+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":235,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.090453+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:53:50.090453+0000","last_peered":"2024-09-13T14:53:50.090453+0000","last_clean":"2024-09-13T14:53:50.090453+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:53:50.090453+0000","last_undegraded":"2024-09-13T14:53:50.090453+0000","last_fullsized":"2024-09-13T14:53:50.090453+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":282,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.840008+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:53:36.840008+0000","last_peered":"2024-09-13T14:53:36.840008+0000","last_clean":"2024-09-13T14:53:36.840008+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:53:36.840008+0000","last_undegraded":"2024-09-13T14:53:36.840008+0000","last_fullsized":"2024-09-13T14:53:36.840008+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":364,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.100077+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:53:50.100077+0000","last_peered":"2024-09-13T14:53:50.100077+0000","last_clean":"2024-09-13T14:53:50.100077+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:53:50.100077+0000","last_undegraded":"2024-09-13T14:53:50.100077+0000","last_fullsized":"2024-09-13T14:53:50.100077+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":205,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.097676+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:53:50.097676+0000","last_peered":"2024-09-13T14:53:50.097676+0000","last_clean":"2024-09-13T14:53:50.097676+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:53:50.097676+0000","last_undegraded":"2024-09-13T14:53:50.097676+0000","last_fullsized":"2024-09-13T14:53:50.097676+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":115,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855554+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:53:36.855554+0000","last_peered":"2024-09-13T14:53:36.855554+0000","last_clean":"2024-09-13T14:53:36.855554+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:53:36.855554+0000","last_undegraded":"2024-09-13T14:53:36.855554+0000","last_fullsized":"2024-09-13T14:53:36.855554+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":342,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.099294+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:53:50.099294+0000","last_peered":"2024-09-13T14:53:50.099294+0000","last_clean":"2024-09-13T14:53:50.099294+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:53:50.099294+0000","last_undegraded":"2024-09-13T14:53:50.099294+0000","last_fullsized":"2024-09-13T14:53:50.099294+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":396,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098612+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:53:50.098612+0000","last_peered":"2024-09-13T14:53:50.098612+0000","last_clean":"2024-09-13T14:53:50.098612+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:53:50.098612+0000","last_undegraded":"2024-09-13T14:53:50.098612+0000","last_fullsized":"2024-09-13T14:53:50.098612+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":393,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095817+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:53:50.095817+0000","last_peered":"2024-09-13T14:53:50.095817+0000","last_clean":"2024-09-13T14:53:50.095817+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:53:50.095817+0000","last_undegraded":"2024-09-13T14:53:50.095817+0000","last_fullsized":"2024-09-13T14:53:50.095817+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":145,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.850460+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:53:36.850460+0000","last_peered":"2024-09-13T14:53:36.850460+0000","last_clean":"2024-09-13T14:53:36.850460+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:53:36.850460+0000","last_undegraded":"2024-09-13T14:53:36.850460+0000","last_fullsized":"2024-09-13T14:53:36.850460+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":360,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.096006+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:53:50.096006+0000","last_peered":"2024-09-13T14:53:50.096006+0000","last_clean":"2024-09-13T14:53:50.096006+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:53:50.096006+0000","last_undegraded":"2024-09-13T14:53:50.096006+0000","last_fullsized":"2024-09-13T14:53:50.096006+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":376,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098481+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:53:50.098481+0000","last_peered":"2024-09-13T14:53:50.098481+0000","last_clean":"2024-09-13T14:53:50.098481+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:53:50.098481+0000","last_undegraded":"2024-09-13T14:53:50.098481+0000","last_fullsized":"2024-09-13T14:53:50.098481+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":345,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.103106+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:53:50.103106+0000","last_peered":"2024-09-13T14:53:50.103106+0000","last_clean":"2024-09-13T14:53:50.103106+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:53:50.103106+0000","last_undegraded":"2024-09-13T14:53:50.103106+0000","last_fullsized":"2024-09-13T14:53:50.103106+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":208,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.097524+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:53:50.097524+0000","last_peered":"2024-09-13T14:53:50.097524+0000","last_clean":"2024-09-13T14:53:50.097524+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:53:50.097524+0000","last_undegraded":"2024-09-13T14:53:50.097524+0000","last_fullsized":"2024-09-13T14:53:50.097524+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":233,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839948+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:53:36.839948+0000","last_peered":"2024-09-13T14:53:36.839948+0000","last_clean":"2024-09-13T14:53:36.839948+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:53:36.839948+0000","last_undegraded":"2024-09-13T14:53:36.839948+0000","last_fullsized":"2024-09-13T14:53:36.839948+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":104,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.093428+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:53:50.093428+0000","last_peered":"2024-09-13T14:53:50.093428+0000","last_clean":"2024-09-13T14:53:50.093428+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:53:50.093428+0000","last_undegraded":"2024-09-13T14:53:50.093428+0000","last_fullsized":"2024-09-13T14:53:50.093428+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":369,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098186+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:53:50.098186+0000","last_peered":"2024-09-13T14:53:50.098186+0000","last_clean":"2024-09-13T14:53:50.098186+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:53:50.098186+0000","last_undegraded":"2024-09-13T14:53:50.098186+0000","last_fullsized":"2024-09-13T14:53:50.098186+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":243,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.840050+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:53:36.840050+0000","last_peered":"2024-09-13T14:53:36.840050+0000","last_clean":"2024-09-13T14:53:36.840050+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:53:36.840050+0000","last_undegraded":"2024-09-13T14:53:36.840050+0000","last_fullsized":"2024-09-13T14:53:36.840050+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":90,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855548+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:53:36.855548+0000","last_peered":"2024-09-13T14:53:36.855548+0000","last_clean":"2024-09-13T14:53:36.855548+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:53:36.855548+0000","last_undegraded":"2024-09-13T14:53:36.855548+0000","last_fullsized":"2024-09-13T14:53:36.855548+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":231,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839860+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:53:36.839860+0000","last_peered":"2024-09-13T14:53:36.839860+0000","last_clean":"2024-09-13T14:53:36.839860+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:53:36.839860+0000","last_undegraded":"2024-09-13T14:53:36.839860+0000","last_fullsized":"2024-09-13T14:53:36.839860+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":92,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.855649+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:53:36.855649+0000","last_peered":"2024-09-13T14:53:36.855649+0000","last_clean":"2024-09-13T14:53:36.855649+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:53:36.855649+0000","last_undegraded":"2024-09-13T14:53:36.855649+0000","last_fullsized":"2024-09-13T14:53:36.855649+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":363,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.103209+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:53:50.103209+0000","last_peered":"2024-09-13T14:53:50.103209+0000","last_clean":"2024-09-13T14:53:50.103209+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:53:50.103209+0000","last_undegraded":"2024-09-13T14:53:50.103209+0000","last_fullsized":"2024-09-13T14:53:50.103209+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":379,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.095298+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:53:50.095298+0000","last_peered":"2024-09-13T14:53:50.095298+0000","last_clean":"2024-09-13T14:53:50.095298+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:53:50.095298+0000","last_undegraded":"2024-09-13T14:53:50.095298+0000","last_fullsized":"2024-09-13T14:53:50.095298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":261,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-13T14:53:36.839692+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:53:36.839692+0000","last_peered":"2024-09-13T14:53:36.839692+0000","last_clean":"2024-09-13T14:53:36.839692+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:53:36.839692+0000","last_undegraded":"2024-09-13T14:53:36.839692+0000","last_fullsized":"2024-09-13T14:53:36.839692+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":432,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-13T14:53:50.098559+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:53:50.098559+0000","last_peered":"2024-09-13T14:53:50.098559+0000","last_clean":"2024-09-13T14:53:50.098559+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:53:50.098559+0000","last_undegraded":"2024-09-13T14:53:50.098559+0000","last_fullsized":"2024-09-13T14:53:50.098559+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1137770520,"num_objects":17362,"num_object_clones":0,"num_object_copies":52086,"num_objects_missing_on_primary":54,"num_objects_missing":54,"num_objects_degraded":1102,"num_objects_misplaced":1035,"num_objects_unfound":0,"num_objects_dirty":17362,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154186,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44670,"num_bytes_recovered":2807300096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3635490816,"data_stored":3635478600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125958,"ondisk_log_size":125958,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495955,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":861116,"kb_used_data":519092,"kb_used_omap":1,"kb_used_meta":342014,"kb_avail":92884036,"statfs":{"total":95995035648,"available":95113252864,"internally_reserved":0,"allocated":531550208,"data_stored":530189988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":350222791},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757590,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":572916,"kb_used_data":361324,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":93172236,"statfs":{"total":95995035648,"available":95408369664,"internally_reserved":0,"allocated":369995776,"data_stored":368639351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216660423},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019230,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663228,"kb_used_data":361396,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":93081924,"statfs":{"total":95995035648,"available":95315890176,"internally_reserved":0,"allocated":370069504,"data_stored":368709345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309066182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468115,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1027248,"kb_used_data":747048,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":92717904,"statfs":{"total":95995035648,"available":94943133696,"internally_reserved":0,"allocated":764977152,"data_stored":763624573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":286846116},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631644,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052480,"kb_used_data":655672,"kb_used_omap":8,"kb_used_meta":396791,"kb_avail":92692672,"statfs":{"total":95995035648,"available":94917296128,"internally_reserved":0,"allocated":671408128,"data_stored":670044399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":406314975},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795167,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":983160,"kb_used_data":668144,"kb_used_omap":7,"kb_used_meta":314872,"kb_avail":92761992,"statfs":{"total":95995035648,"available":94988279808,"internally_reserved":0,"allocated":684179456,"data_stored":682819134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":322429554},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958697,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":584956,"kb_used_data":304692,"kb_used_omap":7,"kb_used_meta":280184,"kb_avail":93160196,"statfs":{"total":95995035648,"available":95396040704,"internally_reserved":0,"allocated":312004608,"data_stored":310640843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":286908903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296947712,"data_stored":296943640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":664408064,"data_stored":664403992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":639500288,"data_stored":639500288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":751501312,"data_stored":751501312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":357629952,"data_stored":357629952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":357175296,"data_stored":357171224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":516358144,"data_stored":516358144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":51970048,"data_stored":51970048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:54:03.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:54:03.590 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:03.610 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:03 smithi172 ceph-mon[100696]: from='client.55054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:03.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:03.641 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:03 smithi154 ceph-mon[102314]: from='client.55054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:03.641 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:03 smithi154 ceph-mon[107038]: from='client.55054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:03.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:03.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:03.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:03.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:03.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:04.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:04.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:04.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:04.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.222 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:54:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:54:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:54:04.222 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:54:04.222 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:54:04.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:04.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:04.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:04.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:04.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:04 smithi172 ceph-mon[100696]: pgmap v762: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 580 op/s; 1102/52170 objects degraded (2.112%); 1035/52170 objects misplaced (1.984%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:04.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:04.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:04.652 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:04 smithi154 ceph-mon[102314]: pgmap v762: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 580 op/s; 1102/52170 objects degraded (2.112%); 1035/52170 objects misplaced (1.984%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:04.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:04 smithi154 ceph-mon[107038]: pgmap v762: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 580 op/s; 1102/52170 objects degraded (2.112%); 1035/52170 objects misplaced (1.984%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:04.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:04.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:04.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:04.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:04.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:04.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:54:05.048 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:54:05.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.169 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:05.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:05.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:05.294 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:05.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:05.341 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1102/52170 objects degraded (2.112%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:05.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:05.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:05.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1102/52170 objects degraded (2.112%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:05.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:05.713 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:05.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1102/52170 objects degraded (2.112%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:05.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:05.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:05.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:05.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:05.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:06.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:06.110 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:06.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:06.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:06.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:06.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:06.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:06 smithi172 ceph-mon[100696]: pgmap v763: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 934 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 594 op/s; 1102/44718 objects degraded (2.464%); 1035/44718 objects misplaced (2.315%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:06.500 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:06.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:06.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:06.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:06 smithi154 ceph-mon[102314]: pgmap v763: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 934 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 594 op/s; 1102/44718 objects degraded (2.464%); 1035/44718 objects misplaced (2.315%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:06.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:06 smithi154 ceph-mon[107038]: pgmap v763: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 934 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 594 op/s; 1102/44718 objects degraded (2.464%); 1035/44718 objects misplaced (2.315%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:06.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:06.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:06.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:06.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:06.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:07.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:07.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:07.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:07.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:07.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86391 6.4 6:29721e02:::benchmark_data_smithi154_175640_object615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:52.696080+0000", 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 254.608563871, 2024-09-13T14:54:07.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3464184220000002, 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86391 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.696080+0000", 2024-09-13T14:54:07.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.696080+0000", 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.696079+0000", 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.696086+0000", 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8390000000000001e-06 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.696094+0000", 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5059999999999995e-06 2024-09-13T14:54:07.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.696099+0000", 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0379999999999999e-06 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.765957+0000", 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.069857882999999996 2024-09-13T14:54:07.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.776692+0000", 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010734824 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.806971+0000", 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030278734000000002 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.816144+0000", 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091729429999999994 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.950739+0000", 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.134595606 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.956063+0000", 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053232369999999998 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.965837+0000", 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097743819999999999 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.984928+0000", 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019091144000000001 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.987000+0000", 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020714660000000001 2024-09-13T14:54:07.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.024949+0000", 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037949522999999999 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.029901+0000", 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049514720000000002 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.042499+0000", 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012598251 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86421 6.4 6:29c63d92:::benchmark_data_smithi154_175640_object645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:52.971777+0000", 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 254.33286680699999, 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.254501705, 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86421 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.971777+0000", 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.971777+0000", 2024-09-13T14:54:07.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.971776+0000", 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.971784+0000", 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8099999999999998e-06 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.971799+0000", 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4953000000000001e-05 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:52.971806+0000", 2024-09-13T14:54:07.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7510000000000008e-06 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.117710+0000", 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.145903279 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.127545+0000", 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098357340000000005 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.173277+0000", 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045731357 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.184167+0000", 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010890413 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.129292+0000", 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9451249879999999 2024-09-13T14:54:07.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.134595+0000", 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005302434 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.143586+0000", 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0089917060000000004 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.208378+0000", 2024-09-13T14:54:07.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.064791875999999998 2024-09-13T14:54:07.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.208951+0000", 2024-09-13T14:54:07.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057333899999999997 2024-09-13T14:54:07.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.210678+0000", 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017261920000000001 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.213391+0000", 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027128159999999998 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.226279+0000", 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012888637 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86463 6.4 6:28057c14:::benchmark_data_smithi154_175640_object687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:53.367355+0000", 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 253.937288951, 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3449534280000002, 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.327 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86463 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.367355+0000", 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.367355+0000", 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.367354+0000", 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.367360+0000", 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2660000000000001e-06 2024-09-13T14:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.367369+0000", 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4409999999999999e-06 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.367373+0000", 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5109999999999997e-06 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.425993+0000", 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058619439000000002 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.432300+0000", 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063073920000000002 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.470847+0000", 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038546443 2024-09-13T14:54:07.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.477979+0000", 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007131783 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.654888+0000", 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.176909303 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.660230+0000", 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053422160000000003 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.669230+0000", 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0089997930000000007 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.681552+0000", 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012322082 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.683159+0000", 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001607004 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.697466+0000", 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014307218999999999 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.700361+0000", 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028953160000000002 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.712309+0000", 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011947262 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86474 6.4 6:241181be:::benchmark_data_smithi154_175640_object698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:53.406510+0000", 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 253.89813455699999, 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4082837229999998, 2024-09-13T14:54:07.335 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86474 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.406510+0000", 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.406510+0000", 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.406508+0000", 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.406516+0000", 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1249999999999993e-06 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.406531+0000", 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4528999999999999e-05 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.406538+0000", 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5089999999999996e-06 2024-09-13T14:54:07.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.433022+0000", 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026483818999999999 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.439447+0000", 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064252129999999999 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.501170+0000", 2024-09-13T14:54:07.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061722581999999998 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.507214+0000", 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060446980000000003 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.738163+0000", 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2309488059999998 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.743444+0000", 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052803310000000001 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.752679+0000", 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092351240000000008 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.790910+0000", 2024-09-13T14:54:07.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038231642000000003 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.792718+0000", 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018077869999999999 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.793254+0000", 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053592600000000005 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.796181+0000", 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029266349999999999 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.814793+0000", 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018612701999999998 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86476 6.4 6:2e5cb58b:::benchmark_data_smithi154_175640_object700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:53.414097+0000", 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 253.89054693899999, 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6756402000000001, 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.344 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86476 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.414097+0000", 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.414097+0000", 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.414096+0000", 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.414103+0000", 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4660000000000002e-06 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.414116+0000", 2024-09-13T14:54:07.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2218999999999999e-05 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.414122+0000", 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3400000000000003e-06 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.442071+0000", 2024-09-13T14:54:07.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027949587000000001 2024-09-13T14:54:07.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.448807+0000", 2024-09-13T14:54:07.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067354119999999996 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.529925+0000", 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.081117647000000001 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.536085+0000", 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061608069999999999 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.992888+0000", 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4568024149999999 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.001143+0000", 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082550550000000007 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.013682+0000", 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012538817000000001 2024-09-13T14:54:07.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.025875+0000", 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012193153 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.027731+0000", 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00185625 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.073894+0000", 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.046163026000000003 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.076824+0000", 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029303210000000001 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.089738+0000", 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291311 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.352 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86497 6.4 6:25936c36:::benchmark_data_smithi154_175640_object721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:53.766253+0000", 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 253.538390991, 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.462496072, 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86497 2024-09-13T14:54:07.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.766253+0000", 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.766253+0000", 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.766252+0000", 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.766258+0000", 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9819999999999996e-06 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:07.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.766269+0000", 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1562e-05 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.766275+0000", 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7209999999999999e-06 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.777511+0000", 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011235976999999999 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.783931+0000", 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006420027 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.880861+0000", 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.096930263000000003 2024-09-13T14:54:07.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.887418+0000", 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065565959999999996 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.143442+0000", 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2560246309999998 2024-09-13T14:54:07.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.148589+0000", 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051468169999999997 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.157958+0000", 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009369311 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.184406+0000", 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026447539999999999 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.186325+0000", 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019187690000000001 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.212582+0000", 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026257427 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.215930+0000", 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003348303 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.228749+0000", 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012818868000000001 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86503 6.4 6:22a5415a:::benchmark_data_smithi154_175640_object727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:53.869973+0000", 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 253.43467169600001, 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3124468540000001, 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86503 2024-09-13T14:54:07.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.869973+0000", 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.869973+0000", 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.869971+0000", 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.869978+0000", 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7410000000000003e-06 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.869991+0000", 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3053e-05 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.869999+0000", 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8960000000000003e-06 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.958984+0000", 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.088985388999999998 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.968050+0000", 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009065498 2024-09-13T14:54:07.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.999530+0000", 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031479877000000003 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.005771+0000", 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006241623 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.115548+0000", 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1097768800000001 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.120681+0000", 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051324439999999999 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.129688+0000", 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090073229999999994 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.141235+0000", 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011546983 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.142871+0000", 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001635779 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.167341+0000", 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024470024999999999 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.170454+0000", 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003113537 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.182419+0000", 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011965227 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86510 6.4 6:28492659:::benchmark_data_smithi154_175640_object734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:53.967947+0000", 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 253.33669767500001, 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6834381430000001, 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86510 2024-09-13T14:54:07.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.967947+0000", 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.967947+0000", 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.967945+0000", 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.967952+0000", 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.905e-06 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.967963+0000", 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0764000000000001e-05 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.967967+0000", 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6399999999999996e-06 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.968666+0000", 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00069841399999999996 2024-09-13T14:54:07.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:53.976213+0000", 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007547252 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.037899+0000", 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061685770000000001 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.044539+0000", 2024-09-13T14:54:07.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066403850000000004 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.534467+0000", 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4899280880000001 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.539403+0000", 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049354530000000002 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.549677+0000", 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010274422 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.633232+0000", 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083554501000000003 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.634926+0000", 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016946260000000001 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.637234+0000", 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002308091 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.639968+0000", 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027339119999999998 2024-09-13T14:54:07.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.651385+0000", 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011416321 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86519 6.4 6:28a73d6d:::benchmark_data_smithi154_175640_object743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:54.246056+0000", 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 253.05858845899999, 2024-09-13T14:54:07.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4236396239999998, 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86519 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.246056+0000", 2024-09-13T14:54:07.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.246056+0000", 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.246054+0000", 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.246061+0000", 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4749999999999996e-06 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.246073+0000", 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1545e-05 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.246079+0000", 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6289999999999998e-06 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.287062+0000", 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040983153000000001 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.295485+0000", 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008423224 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.492238+0000", 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.196753079 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.504174+0000", 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011935916 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.594508+0000", 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0903339440000002 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.599514+0000", 2024-09-13T14:54:07.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005006361 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.608343+0000", 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088287209999999994 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.635465+0000", 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027122165 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.653135+0000", 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017669567000000001 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.654788+0000", 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016528580000000001 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.657548+0000", 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00276035 2024-09-13T14:54:07.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.669695+0000", 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012147415999999999 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86539 6.4 6:238ae2ef:::benchmark_data_smithi154_175640_object763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:54.368380+0000", 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 252.93626414799999, 2024-09-13T14:54:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4236672669999999, 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86539 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.368380+0000", 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.368380+0000", 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.368378+0000", 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.368385+0000", 2024-09-13T14:54:07.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0260000000000004e-06 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.368396+0000", 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1054e-05 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.368401+0000", 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7819999999999999e-06 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.505371+0000", 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.136970121 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.519322+0000", 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013950459 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.555296+0000", 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035974378000000001 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.561585+0000", 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062882679999999996 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.742706+0000", 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1811218879999998 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.748098+0000", 2024-09-13T14:54:07.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053918619999999999 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.757624+0000", 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095252770000000004 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.775014+0000", 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017390777 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.775546+0000", 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053115499999999995 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.777234+0000", 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016885579999999999 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.780060+0000", 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028258950000000001 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.792047+0000", 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011987378 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86547 6.4 6:2fa3c193:::benchmark_data_smithi154_175640_object771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.396 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:54.526207+0000", 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 252.77843689599999, 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4716504750000001, 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86547 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.526207+0000", 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.526207+0000", 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.526206+0000", 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.526215+0000", 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0780000000000002e-06 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.526227+0000", 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2393e-05 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.526234+0000", 2024-09-13T14:54:07.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1960000000000002e-06 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.610942+0000", 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084707603000000006 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.617093+0000", 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061511630000000003 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.662768+0000", 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045674996000000002 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.670321+0000", 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0075532400000000001 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.864103+0000", 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1937817339999999 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.868995+0000", 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048915879999999997 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.877785+0000", 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087906519999999995 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.930166+0000", 2024-09-13T14:54:07.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.052380453 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.980871+0000", 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.050705158 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.982565+0000", 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001693752 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.985275+0000", 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027099609999999999 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.997858+0000", 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012583267 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86553 6.4 6:22f1a583:::benchmark_data_smithi154_175640_object777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:54.554304+0000", 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 252.750340236, 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.463005092, 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86553 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.405 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.554304+0000", 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.554304+0000", 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.554302+0000", 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.554310+0000", 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7350000000000002e-06 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.554326+0000", 2024-09-13T14:54:07.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6121999999999999e-05 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.554336+0000", 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7680000000000003e-06 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.671008+0000", 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.116671262 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.677948+0000", 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0069403779999999997 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.711791+0000", 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033843363000000001 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.717816+0000", 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060247 2024-09-13T14:54:07.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.909062+0000", 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1912463519999998 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.914084+0000", 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050221809999999997 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.922865+0000", 2024-09-13T14:54:07.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087807449999999995 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.954980+0000", 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032115155999999999 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.999549+0000", 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044568155999999998 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.002053+0000", 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002504844 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.004819+0000", 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027657789999999999 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.017309+0000", 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012489945000000001 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86565 6.4 6:27e9c882:::benchmark_data_smithi154_175640_object789:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:54.826442+0000", 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 252.478202358, 2024-09-13T14:54:07.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2139713030000001, 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86565 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.826442+0000", 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.826442+0000", 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.826440+0000", 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.826448+0000", 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7659999999999998e-06 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.826464+0000", 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6093e-05 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.826472+0000", 2024-09-13T14:54:07.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3429999999999996e-06 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.857276+0000", 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030804043 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.865833+0000", 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085560470000000007 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.974214+0000", 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10838152500000001 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.981642+0000", 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074279339999999998 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.955632+0000", 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9739903829999998 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.961204+0000", 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055715449999999998 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.970890+0000", 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096863729999999999 2024-09-13T14:54:07.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.000088+0000", 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029198142 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.018922+0000", 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018833684999999999 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.021284+0000", 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023616990000000001 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.024165+0000", 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002881411 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.040413+0000", 2024-09-13T14:54:07.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01624799 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86571 6.4 6:2b292160:::benchmark_data_smithi154_175640_object795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:54.877094+0000", 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 252.42754989900001, 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4189754880000001, 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:07.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86571 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.877094+0000", 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.877094+0000", 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.877093+0000", 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.877099+0000", 2024-09-13T14:54:07.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4710000000000004e-06 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.877110+0000", 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0881e-05 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.877119+0000", 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2309999999999996e-06 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.987599+0000", 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.110480853 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:54.994249+0000", 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066496350000000001 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.043308+0000", 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049059402000000002 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.055125+0000", 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011816606 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.232634+0000", 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.177508934 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.238529+0000", 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058945689999999997 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.249681+0000", 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011152133999999999 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.261749+0000", 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012068389000000001 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.263895+0000", 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021456800000000001 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.281174+0000", 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017279544000000001 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.284085+0000", 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029103240000000002 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.296070+0000", 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011985167 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86673 6.4 6:24d0d169:::benchmark_data_smithi154_175640_object897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:55.526899+0000", 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 251.777744837, 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9806079329999999, 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86673 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.431 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.526899+0000", 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.526899+0000", 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.526898+0000", 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.526905+0000", 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3289999999999996e-06 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.526920+0000", 2024-09-13T14:54:07.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4649e-05 2024-09-13T14:54:07.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.526927+0000", 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3730000000000004e-06 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.586883+0000", 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.059956187000000001 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.593473+0000", 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065892779999999996 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.623171+0000", 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029698387999999999 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.629319+0000", 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061479050000000004 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.493824+0000", 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8645050670000001 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.499021+0000", 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051969859999999998 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.509727+0000", 2024-09-13T14:54:07.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010706059 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.553999+0000", 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044271410999999997 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.596422+0000", 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042423473000000003 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:07.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.596438+0000", 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.647e-05 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507384+0000", 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91094535600000004 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507507+0000", 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123516 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86674 6.4 6:26d83999:::benchmark_data_smithi154_175640_object898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:55.659436+0000", 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 251.64520794699999, 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8483548509999999, 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.440 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86674 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.659436+0000", 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.659436+0000", 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.659434+0000", 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.659443+0000", 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.49e-06 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.659456+0000", 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3468e-05 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.659466+0000", 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.961e-06 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.689681+0000", 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030214926 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.696855+0000", 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071743329999999998 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.783137+0000", 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.086281618000000004 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:55.789232+0000", 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060952059999999997 2024-09-13T14:54:07.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597140+0000", 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8079073559999999 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597144+0000", 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.087e-06 2024-09-13T14:54:07.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507782+0000", 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91063832499999997 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507791+0000", 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1519999999999998e-06 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86702 6.4 6:2bd62bb7:::benchmark_data_smithi154_175640_object926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:56.085531+0000", 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 251.21911333700001, 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4220444329999999, 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86702 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.085531+0000", 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.085531+0000", 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.085529+0000", 2024-09-13T14:54:07.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.085537+0000", 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4159999999999998e-06 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.085550+0000", 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3636e-05 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.085558+0000", 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1670000000000002e-06 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.203758+0000", 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.118200108 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.211961+0000", 2024-09-13T14:54:07.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082034900000000008 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.257450+0000", 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045488639999999997 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.264002+0000", 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006552268 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597028+0000", 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.333025999 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597033+0000", 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.639e-06 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507566+0000", 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91053342100000001 2024-09-13T14:54:07.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507575+0000", 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2320000000000007e-06 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86716 6.4 6:26c9dec4:::benchmark_data_smithi154_175640_object940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:56.216134+0000", 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 251.08850976599999, 2024-09-13T14:54:07.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2915192179999999, 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86716 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.216134+0000", 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.216134+0000", 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.216133+0000", 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.216141+0000", 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6139999999999992e-06 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.216155+0000", 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3671e-05 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.216165+0000", 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9219999999999992e-06 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.333517+0000", 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.117352437 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.340595+0000", 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0070776160000000001 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.403265+0000", 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.06267056 2024-09-13T14:54:07.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.409350+0000", 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060842309999999998 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597064+0000", 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1877147830000001 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597069+0000", 2024-09-13T14:54:07.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2409999999999997e-06 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507645+0000", 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91057602199999998 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507654+0000", 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9360000000000005e-06 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.462 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86721 6.4 6:2594c329:::benchmark_data_smithi154_175640_object945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:56.253489+0000", 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 251.05115517999999, 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2542001549999999, 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.463 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86721 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.253489+0000", 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.253489+0000", 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.253487+0000", 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.253496+0000", 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7790000000000001e-06 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.253510+0000", 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.402e-05 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.253518+0000", 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9910000000000002e-06 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.341209+0000", 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.087691190000000002 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for missing object", 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.348262+0000", 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007052956 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.394525+0000", 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.046262404999999999 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.400839+0000", 2024-09-13T14:54:07.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063142249999999997 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597082+0000", 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.196243607 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.597086+0000", 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.895e-06 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507680+0000", 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91059355399999997 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.507689+0000", 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3680000000000008e-06 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.469 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86887 6.c 6:302ee6f9:::benchmark_data_smithi154_175640_object1111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:59.508016+0000", 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 247.796628075, 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 23.286470577999999, 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86887 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:07.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.508016+0000", 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.508016+0000", 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.508015+0000", 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.508020+0000", 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4480000000000002e-06 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.508033+0000", 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2481e-05 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.508040+0000", 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8920000000000003e-06 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.509137+0000", 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010975290000000001 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:59.527135+0000", 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017997918000000002 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:50:19.632704+0000", 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 20.105568568999999 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:50:19.632717+0000", 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3671e-05 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:50:22.794473+0000", 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1617552629999999 2024-09-13T14:54:07.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:50:22.794487+0000", 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4022999999999999e-05 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:07.475 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:54:07.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:07.500 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:07.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:07.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:07.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:07.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:07.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:07.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:07.996 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:08.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:08.036 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:54:08.036 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:08.074 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:54:08.074 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:08.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:08.110 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:54:08.110 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:08.141 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:54:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:08.164 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:54:08.164 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:08.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:08.193 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:54:08.193 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.217 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:08.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:08.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:54:08.223 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:54:08.223 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:08.256 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:54:08.256 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:08.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.343 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:08.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:08.404 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:08 smithi154 ceph-mon[107038]: pgmap v764: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 899 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 592 op/s; 1296/43029 objects degraded (3.012%); 841/43029 objects misplaced (1.954%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:08.405 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2388122908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:08.405 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:08 smithi154 ceph-mon[102314]: pgmap v764: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 899 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 592 op/s; 1296/43029 objects degraded (3.012%); 841/43029 objects misplaced (1.954%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:08.405 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2388122908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:08.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:08.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:08 smithi172 ceph-mon[100696]: pgmap v764: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 899 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 592 op/s; 1296/43029 objects degraded (3.012%); 841/43029 objects misplaced (1.954%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2388122908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:08.512 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:08.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:08.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:08.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:08.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:08.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:08.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:08.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:08.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:09.022 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:09.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:09.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:09.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:09.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:09.380 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:09.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:09.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:09.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:09.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:09.654 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.678 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:09.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:09.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:09.805 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:09.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:09.931 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:09.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:10.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:10.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:10.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:10.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:10.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:10.333 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:10.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:10.453 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[107038]: pgmap v765: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 850 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 1296/40644 objects degraded (3.189%); 841/40644 objects misplaced (2.069%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:10.454 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:10.454 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:10.454 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1296/40644 objects degraded (3.189%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.485 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:10.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:10.505 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:10.505 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:10.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:10.610 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:10.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[102314]: pgmap v765: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 850 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 1296/40644 objects degraded (3.189%); 841/40644 objects misplaced (2.069%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1296/40644 objects degraded (3.189%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:10 smithi172 ceph-mon[100696]: pgmap v765: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 850 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 601 op/s; 1296/40644 objects degraded (3.189%); 841/40644 objects misplaced (2.069%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:10 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:10 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1296/40644 objects degraded (3.189%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:10.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:10.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:10.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:10.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:10.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:11.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:11.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:11.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:11.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:11.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:54:11.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:11.303 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:11.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:11.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:11.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:11.473 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:11.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:11.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:11.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:11 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:11.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:11 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:11.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:11 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:11.647 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:11 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:11.648 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:11 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:11.648 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:11 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:11.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:11 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:54:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:11 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:54:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:11 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:11.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:11.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:11.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:11.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:11.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:54:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:54:11] "GET /metrics HTTP/1.1" 200 36515 "" "Prometheus/2.51.0" 2024-09-13T14:54:11.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:12.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:12.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.170 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:12.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:12.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:12.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:12.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:12.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:12.531 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:12.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:12.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:12.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:12.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:12.800 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:12 smithi154 ceph-mon[102314]: pgmap v766: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 699 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 625 op/s; 1502/33396 objects degraded (4.498%); 635/33396 objects misplaced (1.901%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:12.800 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:12 smithi154 ceph-mon[107038]: pgmap v766: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 699 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 625 op/s; 1502/33396 objects degraded (4.498%); 635/33396 objects misplaced (1.901%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:12.801 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:12 smithi172 ceph-mon[100696]: pgmap v766: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 699 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 625 op/s; 1502/33396 objects degraded (4.498%); 635/33396 objects misplaced (1.901%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:12.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:12.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:12.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:12.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:13.004 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:13.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:13.125 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:13.125 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:13.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:13.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:13.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:13.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:13.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:13.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:13.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:13.549 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:13.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:13.679 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:13.680 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:13.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:13.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:13.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:13.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:54:13.900 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:13.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:13.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:13.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:14.047 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:54:14.048 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1502/33396 objects degraded (4.498%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1069,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":253,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238500,"num_in_osds":7,"osd_in_since":1726238900,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":88},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":11132,"data_bytes":732436477,"bytes_used":4718497792,"bytes_avail":667246751744,"bytes_total":671965249536,"degraded_objects":1502,"degraded_total":33396,"degraded_ratio":0.04497544616121691,"misplaced_objects":635,"misplaced_total":33396,"misplaced_ratio":0.019014253203976525,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":535},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi154.daekzd","status":"up:active","gid":24539}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"}},"servicemap":{"epoch":25,"modified":"2024-09-13T14:50:26.951855+0000","services":{}},"progress_events":{}} 2024-09-13T14:54:14.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:14.110 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:14.129 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:14.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:14.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:14.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:14.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:14.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:14.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:14.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:14.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:14.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:14.665 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:54:14.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:14.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:14.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:14.891 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:14 smithi172 ceph-mon[100696]: pgmap v767: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 699 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 1502/33396 objects degraded (4.498%); 635/33396 objects misplaced (1.901%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:14.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1905739929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:14.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/317486484' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:54:14.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:14.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:14.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:14 smithi154 ceph-mon[102314]: pgmap v767: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 699 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 1502/33396 objects degraded (4.498%); 635/33396 objects misplaced (1.901%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:14.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1905739929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:14.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/317486484' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:54:14.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:14 smithi154 ceph-mon[107038]: pgmap v767: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 699 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 1502/33396 objects degraded (4.498%); 635/33396 objects misplaced (1.901%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:14.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1905739929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:14.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/317486484' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:54:15.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:15.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:15.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:15.177 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-13T14:54:15.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:54:15.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:15.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:15.366 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:15.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:15.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:15.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:15.637 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:15.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:15.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:15.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:15.880 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:15 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:15.880 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1905739929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:15.880 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:15 smithi172 ceph-mon[100696]: osdmap e254: 8 total, 8 up, 7 in 2024-09-13T14:54:15.880 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1502/33396 objects degraded (4.498%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:15.905 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:15.905 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:15.907 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:15.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1905739929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:15.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[102314]: osdmap e254: 8 total, 8 up, 7 in 2024-09-13T14:54:15.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1502/33396 objects degraded (4.498%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:54:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1905739929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[107038]: osdmap e254: 8 total, 8 up, 7 in 2024-09-13T14:54:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1502/33396 objects degraded (4.498%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:16.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:16.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:16.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:16.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:16.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:16.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:16.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:16.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:16.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:54:16.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:16.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:16.859 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:16.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:16.898 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:16 smithi154 ceph-mon[102314]: pgmap v769: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 530 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 1502/25317 objects degraded (5.933%); 635/25317 objects misplaced (2.508%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:16.898 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:16 smithi154 ceph-mon[107038]: pgmap v769: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 530 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 1502/25317 objects degraded (5.933%); 635/25317 objects misplaced (2.508%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:16.905 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:16.988 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:16 smithi172 ceph-mon[100696]: pgmap v769: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 530 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 1502/25317 objects degraded (5.933%); 635/25317 objects misplaced (2.508%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:17.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:17.033 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:17.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:17.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:17.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:17.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:17.327 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:17.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:17.403 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:54:17.403 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:17.440 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:54:17.441 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:17.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:17.482 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:54:17.482 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:17.514 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:54:17.514 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:17.545 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:54:17.545 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:17.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:17.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:17.576 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:54:17.576 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:17.608 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:54:17.608 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:17.641 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:54:17.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:17.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:17.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:17 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2456465814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:17.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:17 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2456465814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:17.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:17.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:17.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:17.979 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:17.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:17 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2456465814' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:18.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:18.107 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:18.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:54:18.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:18.233 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:18.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:18.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:18.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:18.519 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:18.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:18.635 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:54:18.635 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:54:18.635 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T14:54:18.635 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:54:18.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:18.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:18.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:18.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:18.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:18.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:18.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:18 smithi154 ceph-mon[102314]: pgmap v770: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 489 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 1732/23337 objects degraded (7.422%); 405/23337 objects misplaced (1.735%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:18 smithi154 ceph-mon[107038]: pgmap v770: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 489 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 1732/23337 objects degraded (7.422%); 405/23337 objects misplaced (1.735%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:18 smithi172 ceph-mon[100696]: pgmap v770: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 489 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 1732/23337 objects degraded (7.422%); 405/23337 objects misplaced (1.735%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:19.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.083 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:19.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:19.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:19.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:19.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:19.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:19.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.504 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:19.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:19.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:54:19.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:19.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:19.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:19.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:19.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:19.836 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:19.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:19.923 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:19.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:20.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:20.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:20.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:20.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:20.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:20.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:20.434 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:20.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:20.468 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:20.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.597 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:20.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:20.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:20.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:20.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:20.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:20.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:20.969 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:20 smithi154 ceph-mon[102314]: pgmap v771: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 436 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 661 op/s; 1732/20793 objects degraded (8.330%); 405/20793 objects misplaced (1.948%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1732/20793 objects degraded (8.330%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:20 smithi154 ceph-mon[107038]: pgmap v771: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 436 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 661 op/s; 1732/20793 objects degraded (8.330%); 405/20793 objects misplaced (1.948%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:20.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1732/20793 objects degraded (8.330%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:20.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:20 smithi172 ceph-mon[100696]: pgmap v771: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 436 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 661 op/s; 1732/20793 objects degraded (8.330%); 405/20793 objects misplaced (1.948%); 0 B/s, 1 objects/s recovering 2024-09-13T14:54:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1732/20793 objects degraded (8.330%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:21.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:21.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:21.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:21.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:21.280 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:21.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:21.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:21.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:21.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:21.616 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:54:21.617 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:54:21.621 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:54:21.621 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:54:21.621 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:54:21.641 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:54:21.643 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:54:21.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:21.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:21.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:21.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:21.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:21.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:21.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:22.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:22.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:22.110 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:54:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:54:21] "GET /metrics HTTP/1.1" 200 36515 "" "Prometheus/2.51.0" 2024-09-13T14:54:22.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:22.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:22.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:54:22.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:22.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:22.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:22.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:22.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:22.533 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":772,"stamp":"2024-09-13T14:54:21.030448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":275781629,"num_objects":4164,"num_object_clones":0,"num_object_copies":12492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":196,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":168535,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44919,"num_bytes_recovered":2812234885,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129466,"ondisk_log_size":129466,"up":267,"acting":266,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":269,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3252472,"kb_used_data":1123456,"kb_used_omap":32,"kb_used_meta":2128927,"kb_avail":652963592,"statfs":{"total":671965249536,"available":668634718208,"internally_reserved":0,"allocated":1150418944,"data_stored":1140846529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2180021808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,6,6,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-456654848,"num_objects":-6968,"num_object_clones":0,"num_object_copies":-20904,"num_objects_missing_on_primary":-38,"num_objects_missing":-38,"num_objects_degraded":400,"num_objects_misplaced":-439,"num_objects_unfound":0,"num_objects_dirty":-6968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":2555904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005627"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":390,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.691949+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:54:14.691949+0000","last_peered":"2024-09-13T14:54:14.691949+0000","last_clean":"2024-09-13T14:54:14.691949+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:54:14.691949+0000","last_undegraded":"2024-09-13T14:54:14.691949+0000","last_fullsized":"2024-09-13T14:54:14.691949+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":120,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336397+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:54:01.336397+0000","last_peered":"2024-09-13T14:54:01.336397+0000","last_clean":"2024-09-13T14:54:01.336397+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:54:01.336397+0000","last_undegraded":"2024-09-13T14:54:01.336397+0000","last_fullsized":"2024-09-13T14:54:01.336397+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":90,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333423+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:54:01.333423+0000","last_peered":"2024-09-13T14:54:01.333423+0000","last_clean":"2024-09-13T14:54:01.333423+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:54:01.333423+0000","last_undegraded":"2024-09-13T14:54:01.333423+0000","last_fullsized":"2024-09-13T14:54:01.333423+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":380,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.689379+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:54:14.689379+0000","last_peered":"2024-09-13T14:54:14.689379+0000","last_clean":"2024-09-13T14:54:14.689379+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:54:14.689379+0000","last_undegraded":"2024-09-13T14:54:14.689379+0000","last_fullsized":"2024-09-13T14:54:14.689379+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":379,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.684026+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:54:14.684026+0000","last_peered":"2024-09-13T14:54:14.684026+0000","last_clean":"2024-09-13T14:54:14.684026+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:54:14.684026+0000","last_undegraded":"2024-09-13T14:54:14.684026+0000","last_fullsized":"2024-09-13T14:54:14.684026+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":190,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336303+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:54:01.336303+0000","last_peered":"2024-09-13T14:54:01.336303+0000","last_clean":"2024-09-13T14:54:01.336303+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:54:01.336303+0000","last_undegraded":"2024-09-13T14:54:01.336303+0000","last_fullsized":"2024-09-13T14:54:01.336303+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":105,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.685122+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:54:14.685122+0000","last_peered":"2024-09-13T14:54:14.685122+0000","last_clean":"2024-09-13T14:54:14.685122+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:54:14.685122+0000","last_undegraded":"2024-09-13T14:54:14.685122+0000","last_fullsized":"2024-09-13T14:54:14.685122+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":167,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.682141+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:54:14.682141+0000","last_peered":"2024-09-13T14:54:14.682141+0000","last_clean":"2024-09-13T14:54:14.682141+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:54:14.682141+0000","last_undegraded":"2024-09-13T14:54:14.682141+0000","last_fullsized":"2024-09-13T14:54:14.682141+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"254'10629","reported_seq":11478,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.507516+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:54:19.507516+0000","last_peered":"2024-09-13T14:54:19.507516+0000","last_clean":"2024-09-13T14:54:19.507516+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:54:19.507516+0000","last_undegraded":"2024-09-13T14:54:19.507516+0000","last_fullsized":"2024-09-13T14:54:19.507516+0000","mapping_epoch":195,"log_start":"164'600","ondisk_log_start":"164'600","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":267,"num_object_clones":0,"num_object_copies":801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10629,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":314,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.687033+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:54:14.687033+0000","last_peered":"2024-09-13T14:54:14.687033+0000","last_clean":"2024-09-13T14:54:14.687033+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:54:14.687033+0000","last_undegraded":"2024-09-13T14:54:14.687033+0000","last_fullsized":"2024-09-13T14:54:14.687033+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":117,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.332999+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:54:01.332999+0000","last_peered":"2024-09-13T14:54:01.332999+0000","last_clean":"2024-09-13T14:54:01.332999+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:54:01.332999+0000","last_undegraded":"2024-09-13T14:54:01.332999+0000","last_fullsized":"2024-09-13T14:54:01.332999+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"254'10373","reported_seq":12011,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:20.407999+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:54:20.407999+0000","last_peered":"2024-09-13T14:54:20.407999+0000","last_clean":"2024-09-13T14:54:20.407999+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:54:20.407999+0000","last_undegraded":"2024-09-13T14:54:20.407999+0000","last_fullsized":"2024-09-13T14:54:20.407999+0000","mapping_epoch":170,"log_start":"189'3900","ondisk_log_start":"189'3900","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10373,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":118,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336782+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:54:01.336782+0000","last_peered":"2024-09-13T14:54:01.336782+0000","last_clean":"2024-09-13T14:54:01.336782+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:54:01.336782+0000","last_undegraded":"2024-09-13T14:54:01.336782+0000","last_fullsized":"2024-09-13T14:54:01.336782+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":271,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.676565+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:54:14.676565+0000","last_peered":"2024-09-13T14:54:14.676565+0000","last_clean":"2024-09-13T14:54:14.676565+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:54:14.676565+0000","last_undegraded":"2024-09-13T14:54:14.676565+0000","last_fullsized":"2024-09-13T14:54:14.676565+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"254'10575","reported_seq":12305,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.311709+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:54:19.311709+0000","last_peered":"2024-09-13T14:54:19.311709+0000","last_clean":"2024-09-13T14:54:19.311709+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:54:19.311709+0000","last_undegraded":"2024-09-13T14:54:19.311709+0000","last_fullsized":"2024-09-13T14:54:19.311709+0000","mapping_epoch":175,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8875,"log_dups_size":0,"ondisk_log_size":8875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629184,"num_objects":269,"num_object_clones":0,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10575,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":94,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333511+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:54:01.333511+0000","last_peered":"2024-09-13T14:54:01.333511+0000","last_clean":"2024-09-13T14:54:01.333511+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:54:01.333511+0000","last_undegraded":"2024-09-13T14:54:01.333511+0000","last_fullsized":"2024-09-13T14:54:01.333511+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":50,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.339314+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:54:01.339314+0000","last_peered":"2024-09-13T14:54:01.339314+0000","last_clean":"2024-09-13T14:54:01.339314+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:54:01.339314+0000","last_undegraded":"2024-09-13T14:54:01.339314+0000","last_fullsized":"2024-09-13T14:54:01.339314+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"254'10163","reported_seq":16287,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:18.024252+0000","last_change":"2024-09-13T14:52:50.434519+0000","last_active":"2024-09-13T14:54:18.024252+0000","last_peered":"2024-09-13T14:54:18.024252+0000","last_clean":"2024-09-13T14:54:18.024252+0000","last_became_active":"2024-09-13T14:52:50.417699+0000","last_became_peered":"2024-09-13T14:52:50.417699+0000","last_unstale":"2024-09-13T14:54:18.024252+0000","last_undegraded":"2024-09-13T14:54:18.024252+0000","last_fullsized":"2024-09-13T14:54:18.024252+0000","mapping_epoch":238,"log_start":"171'1800","ondisk_log_start":"171'1800","created":160,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8363,"log_dups_size":0,"ondisk_log_size":8363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:10.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10163,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2948,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":106,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333393+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:54:01.333393+0000","last_peered":"2024-09-13T14:54:01.333393+0000","last_clean":"2024-09-13T14:54:01.333393+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:54:01.333393+0000","last_undegraded":"2024-09-13T14:54:01.333393+0000","last_fullsized":"2024-09-13T14:54:01.333393+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":359,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.684655+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:54:14.684655+0000","last_peered":"2024-09-13T14:54:14.684655+0000","last_clean":"2024-09-13T14:54:14.684655+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:54:14.684655+0000","last_undegraded":"2024-09-13T14:54:14.684655+0000","last_fullsized":"2024-09-13T14:54:14.684655+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"254'10503","reported_seq":27769,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.149921+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:54:19.149921+0000","last_peered":"2024-09-13T14:54:19.149921+0000","last_clean":"2024-09-13T14:54:19.149921+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:54:19.149921+0000","last_undegraded":"2024-09-13T14:54:19.149921+0000","last_fullsized":"2024-09-13T14:54:19.149921+0000","mapping_epoch":232,"log_start":"181'3000","ondisk_log_start":"181'3000","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7503,"log_dups_size":0,"ondisk_log_size":7503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16449536,"num_objects":251,"num_object_clones":0,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":367,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.681422+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:54:14.681422+0000","last_peered":"2024-09-13T14:54:14.681422+0000","last_clean":"2024-09-13T14:54:14.681422+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:54:14.681422+0000","last_undegraded":"2024-09-13T14:54:14.681422+0000","last_fullsized":"2024-09-13T14:54:14.681422+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":237,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336284+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:54:01.336284+0000","last_peered":"2024-09-13T14:54:01.336284+0000","last_clean":"2024-09-13T14:54:01.336284+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:54:01.336284+0000","last_undegraded":"2024-09-13T14:54:01.336284+0000","last_fullsized":"2024-09-13T14:54:01.336284+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"254'10368","reported_seq":15384,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.212286+0000","last_change":"2024-09-13T14:53:24.611648+0000","last_active":"2024-09-13T14:54:19.212286+0000","last_peered":"2024-09-13T14:54:19.212286+0000","last_clean":"2024-09-13T14:54:19.212286+0000","last_became_active":"2024-09-13T14:53:24.585428+0000","last_became_peered":"2024-09-13T14:53:24.585428+0000","last_unstale":"2024-09-13T14:54:19.212286+0000","last_undegraded":"2024-09-13T14:54:19.212286+0000","last_fullsized":"2024-09-13T14:54:19.212286+0000","mapping_epoch":248,"log_start":"171'1700","ondisk_log_start":"171'1700","created":160,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:41.636547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10368,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":147914752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":138,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.339388+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:54:01.339388+0000","last_peered":"2024-09-13T14:54:01.339388+0000","last_clean":"2024-09-13T14:54:01.339388+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:54:01.339388+0000","last_undegraded":"2024-09-13T14:54:01.339388+0000","last_fullsized":"2024-09-13T14:54:01.339388+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":234,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336586+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:54:01.336586+0000","last_peered":"2024-09-13T14:54:01.336586+0000","last_clean":"2024-09-13T14:54:01.336586+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:54:01.336586+0000","last_undegraded":"2024-09-13T14:54:01.336586+0000","last_fullsized":"2024-09-13T14:54:01.336586+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"254'10611","reported_seq":15625,"reported_epoch":254,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:54:20.459566+0000","last_change":"2024-09-13T14:53:29.139613+0000","last_active":"2024-09-13T14:54:20.459566+0000","last_peered":"2024-09-13T14:54:20.459566+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:54:20.459566+0000","last_undegraded":"2024-09-13T14:53:29.127369+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"189'4200","ondisk_log_start":"189'4200","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6411,"log_dups_size":0,"ondisk_log_size":6411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":211,"num_object_clones":0,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":196,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10611,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2492,"num_bytes_recovered":158138368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"1,3,4,5","objects":1},{"shards":"1,3,5","objects":195},{"shards":"3,5","objects":-1887}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":230,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.676659+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:54:14.676659+0000","last_peered":"2024-09-13T14:54:14.676659+0000","last_clean":"2024-09-13T14:54:14.676659+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:54:14.676659+0000","last_undegraded":"2024-09-13T14:54:14.676659+0000","last_fullsized":"2024-09-13T14:54:14.676659+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":264,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336784+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:54:01.336784+0000","last_peered":"2024-09-13T14:54:01.336784+0000","last_clean":"2024-09-13T14:54:01.336784+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:54:01.336784+0000","last_undegraded":"2024-09-13T14:54:01.336784+0000","last_fullsized":"2024-09-13T14:54:01.336784+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"254'10241","reported_seq":14683,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.498124+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:54:19.498124+0000","last_peered":"2024-09-13T14:54:19.498124+0000","last_clean":"2024-09-13T14:54:19.498124+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:54:19.498124+0000","last_undegraded":"2024-09-13T14:54:19.498124+0000","last_fullsized":"2024-09-13T14:54:19.498124+0000","mapping_epoch":186,"log_start":"164'200","ondisk_log_start":"164'200","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17301528,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10241,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":105,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.687428+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:54:14.687428+0000","last_peered":"2024-09-13T14:54:14.687428+0000","last_clean":"2024-09-13T14:54:14.687428+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:54:14.687428+0000","last_undegraded":"2024-09-13T14:54:14.687428+0000","last_fullsized":"2024-09-13T14:54:14.687428+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":215,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.687127+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:54:14.687127+0000","last_peered":"2024-09-13T14:54:14.687127+0000","last_clean":"2024-09-13T14:54:14.687127+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:54:14.687127+0000","last_undegraded":"2024-09-13T14:54:14.687127+0000","last_fullsized":"2024-09-13T14:54:14.687127+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"254'10510","reported_seq":11773,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.596656+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:54:19.596656+0000","last_peered":"2024-09-13T14:54:19.596656+0000","last_clean":"2024-09-13T14:54:19.596656+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:54:19.596656+0000","last_undegraded":"2024-09-13T14:54:19.596656+0000","last_fullsized":"2024-09-13T14:54:19.596656+0000","mapping_epoch":204,"log_start":"164'465","ondisk_log_start":"164'465","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15335424,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10510,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":562,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.683913+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:54:14.683913+0000","last_peered":"2024-09-13T14:54:14.683913+0000","last_clean":"2024-09-13T14:54:14.683913+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:54:14.683913+0000","last_undegraded":"2024-09-13T14:54:14.683913+0000","last_fullsized":"2024-09-13T14:54:14.683913+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":821,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333220+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:54:01.333220+0000","last_peered":"2024-09-13T14:54:01.333220+0000","last_clean":"2024-09-13T14:54:01.333220+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:54:01.333220+0000","last_undegraded":"2024-09-13T14:54:01.333220+0000","last_fullsized":"2024-09-13T14:54:01.333220+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":325,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.680283+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:54:14.680283+0000","last_peered":"2024-09-13T14:54:14.680283+0000","last_clean":"2024-09-13T14:54:14.680283+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:54:14.680283+0000","last_undegraded":"2024-09-13T14:54:14.680283+0000","last_fullsized":"2024-09-13T14:54:14.680283+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":50,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.335968+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:54:01.335968+0000","last_peered":"2024-09-13T14:54:01.335968+0000","last_clean":"2024-09-13T14:54:01.335968+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:54:01.335968+0000","last_undegraded":"2024-09-13T14:54:01.335968+0000","last_fullsized":"2024-09-13T14:54:01.335968+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"254'10604","reported_seq":15357,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.771286+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:54:19.771286+0000","last_peered":"2024-09-13T14:54:19.771286+0000","last_clean":"2024-09-13T14:54:19.771286+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:54:19.771286+0000","last_undegraded":"2024-09-13T14:54:19.771286+0000","last_fullsized":"2024-09-13T14:54:19.771286+0000","mapping_epoch":223,"log_start":"189'4018","ondisk_log_start":"189'4018","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15990784,"num_objects":244,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10604,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":407,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.670294+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:54:14.670294+0000","last_peered":"2024-09-13T14:54:14.670294+0000","last_clean":"2024-09-13T14:54:14.670294+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:54:14.670294+0000","last_undegraded":"2024-09-13T14:54:14.670294+0000","last_fullsized":"2024-09-13T14:54:14.670294+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":373,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.684238+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:54:14.684238+0000","last_peered":"2024-09-13T14:54:14.684238+0000","last_clean":"2024-09-13T14:54:14.684238+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:54:14.684238+0000","last_undegraded":"2024-09-13T14:54:14.684238+0000","last_fullsized":"2024-09-13T14:54:14.684238+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":384,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.684251+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:54:14.684251+0000","last_peered":"2024-09-13T14:54:14.684251+0000","last_clean":"2024-09-13T14:54:14.684251+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:54:14.684251+0000","last_undegraded":"2024-09-13T14:54:14.684251+0000","last_fullsized":"2024-09-13T14:54:14.684251+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"254'10313","reported_seq":20643,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:18.858391+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:54:18.858391+0000","last_peered":"2024-09-13T14:54:18.858391+0000","last_clean":"2024-09-13T14:54:18.858391+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:54:18.858391+0000","last_undegraded":"2024-09-13T14:54:18.858391+0000","last_fullsized":"2024-09-13T14:54:18.858391+0000","mapping_epoch":235,"log_start":"174'2200","ondisk_log_start":"174'2200","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8113,"log_dups_size":0,"ondisk_log_size":8113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546688,"num_objects":283,"num_object_clones":0,"num_object_copies":849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10313,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":910,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.682396+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:54:14.682396+0000","last_peered":"2024-09-13T14:54:14.682396+0000","last_clean":"2024-09-13T14:54:14.682396+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:54:14.682396+0000","last_undegraded":"2024-09-13T14:54:14.682396+0000","last_fullsized":"2024-09-13T14:54:14.682396+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":364,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.670506+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:54:14.670506+0000","last_peered":"2024-09-13T14:54:14.670506+0000","last_clean":"2024-09-13T14:54:14.670506+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:54:14.670506+0000","last_undegraded":"2024-09-13T14:54:14.670506+0000","last_fullsized":"2024-09-13T14:54:14.670506+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":117,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.675459+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:54:14.675459+0000","last_peered":"2024-09-13T14:54:14.675459+0000","last_clean":"2024-09-13T14:54:14.675459+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:54:14.675459+0000","last_undegraded":"2024-09-13T14:54:14.675459+0000","last_fullsized":"2024-09-13T14:54:14.675459+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"254'10523","reported_seq":19868,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:20.460461+0000","last_change":"2024-09-13T14:52:52.443442+0000","last_active":"2024-09-13T14:54:20.460461+0000","last_peered":"2024-09-13T14:54:20.460461+0000","last_clean":"2024-09-13T14:54:20.460461+0000","last_became_active":"2024-09-13T14:52:52.421083+0000","last_became_peered":"2024-09-13T14:52:52.421083+0000","last_unstale":"2024-09-13T14:54:20.460461+0000","last_undegraded":"2024-09-13T14:54:20.460461+0000","last_fullsized":"2024-09-13T14:54:20.460461+0000","mapping_epoch":240,"log_start":"189'4100","ondisk_log_start":"189'4100","created":160,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6423,"log_dups_size":0,"ondisk_log_size":6423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:14.887813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12910592,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4569,"num_bytes_recovered":287244288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":108,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333396+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:54:01.333396+0000","last_peered":"2024-09-13T14:54:01.333396+0000","last_clean":"2024-09-13T14:54:01.333396+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:54:01.333396+0000","last_undegraded":"2024-09-13T14:54:01.333396+0000","last_fullsized":"2024-09-13T14:54:01.333396+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":239,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336598+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:54:01.336598+0000","last_peered":"2024-09-13T14:54:01.336598+0000","last_clean":"2024-09-13T14:54:01.336598+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:54:01.336598+0000","last_undegraded":"2024-09-13T14:54:01.336598+0000","last_fullsized":"2024-09-13T14:54:01.336598+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333236+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:54:01.333236+0000","last_peered":"2024-09-13T14:54:01.333236+0000","last_clean":"2024-09-13T14:54:01.333236+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:54:01.333236+0000","last_undegraded":"2024-09-13T14:54:01.333236+0000","last_fullsized":"2024-09-13T14:54:01.333236+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"254'10491","reported_seq":15107,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:18.005201+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:54:18.005201+0000","last_peered":"2024-09-13T14:54:18.005201+0000","last_clean":"2024-09-13T14:54:18.005201+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:54:18.005201+0000","last_undegraded":"2024-09-13T14:54:18.005201+0000","last_fullsized":"2024-09-13T14:54:18.005201+0000","mapping_epoch":226,"log_start":"173'2100","ondisk_log_start":"173'2100","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":323,"num_object_clones":0,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10491,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":108,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333396+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:54:01.333396+0000","last_peered":"2024-09-13T14:54:01.333396+0000","last_clean":"2024-09-13T14:54:01.333396+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:54:01.333396+0000","last_undegraded":"2024-09-13T14:54:01.333396+0000","last_fullsized":"2024-09-13T14:54:01.333396+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":919,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.672580+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:54:14.672580+0000","last_peered":"2024-09-13T14:54:14.672580+0000","last_clean":"2024-09-13T14:54:14.672580+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:54:14.672580+0000","last_undegraded":"2024-09-13T14:54:14.672580+0000","last_fullsized":"2024-09-13T14:54:14.672580+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.685682+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:54:14.685682+0000","last_peered":"2024-09-13T14:54:14.685682+0000","last_clean":"2024-09-13T14:54:14.685682+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:54:14.685682+0000","last_undegraded":"2024-09-13T14:54:14.685682+0000","last_fullsized":"2024-09-13T14:54:14.685682+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"254'10665","reported_seq":15407,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.786483+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:54:19.786483+0000","last_peered":"2024-09-13T14:54:19.786483+0000","last_clean":"2024-09-13T14:54:19.786483+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:54:19.786483+0000","last_undegraded":"2024-09-13T14:54:19.786483+0000","last_fullsized":"2024-09-13T14:54:19.786483+0000","mapping_epoch":218,"log_start":"189'4045","ondisk_log_start":"189'4045","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6620,"log_dups_size":0,"ondisk_log_size":6620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15925248,"num_objects":243,"num_object_clones":0,"num_object_copies":729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":371,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.672499+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:54:14.672499+0000","last_peered":"2024-09-13T14:54:14.672499+0000","last_clean":"2024-09-13T14:54:14.672499+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:54:14.672499+0000","last_undegraded":"2024-09-13T14:54:14.672499+0000","last_fullsized":"2024-09-13T14:54:14.672499+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":593,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.685041+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:54:14.685041+0000","last_peered":"2024-09-13T14:54:14.685041+0000","last_clean":"2024-09-13T14:54:14.685041+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:54:14.685041+0000","last_undegraded":"2024-09-13T14:54:14.685041+0000","last_fullsized":"2024-09-13T14:54:14.685041+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":369,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.684586+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:54:14.684586+0000","last_peered":"2024-09-13T14:54:14.684586+0000","last_clean":"2024-09-13T14:54:14.684586+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:54:14.684586+0000","last_undegraded":"2024-09-13T14:54:14.684586+0000","last_fullsized":"2024-09-13T14:54:14.684586+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"254'10260","reported_seq":16827,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:19.191043+0000","last_change":"2024-09-13T14:53:05.481759+0000","last_active":"2024-09-13T14:54:19.191043+0000","last_peered":"2024-09-13T14:54:19.191043+0000","last_clean":"2024-09-13T14:54:19.191043+0000","last_became_active":"2024-09-13T14:53:05.465653+0000","last_became_peered":"2024-09-13T14:53:05.465653+0000","last_unstale":"2024-09-13T14:54:19.191043+0000","last_undegraded":"2024-09-13T14:54:19.191043+0000","last_fullsized":"2024-09-13T14:54:19.191043+0000","mapping_epoch":243,"log_start":"181'2700","ondisk_log_start":"181'2700","created":160,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:28.220387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384000,"num_objects":250,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10260,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3214,"num_bytes_recovered":205586432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":389,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.685470+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:54:14.685470+0000","last_peered":"2024-09-13T14:54:14.685470+0000","last_clean":"2024-09-13T14:54:14.685470+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:54:14.685470+0000","last_undegraded":"2024-09-13T14:54:14.685470+0000","last_fullsized":"2024-09-13T14:54:14.685470+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":821,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.673150+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:54:14.673150+0000","last_peered":"2024-09-13T14:54:14.673150+0000","last_clean":"2024-09-13T14:54:14.673150+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:54:14.673150+0000","last_undegraded":"2024-09-13T14:54:14.673150+0000","last_fullsized":"2024-09-13T14:54:14.673150+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":105,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.687333+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:54:14.687333+0000","last_peered":"2024-09-13T14:54:14.687333+0000","last_clean":"2024-09-13T14:54:14.687333+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:54:14.687333+0000","last_undegraded":"2024-09-13T14:54:14.687333+0000","last_fullsized":"2024-09-13T14:54:14.687333+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"254'10583","reported_seq":12172,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:18.916159+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:54:18.916159+0000","last_peered":"2024-09-13T14:54:18.916159+0000","last_clean":"2024-09-13T14:54:18.916159+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:54:18.916159+0000","last_undegraded":"2024-09-13T14:54:18.916159+0000","last_fullsized":"2024-09-13T14:54:18.916159+0000","mapping_epoch":188,"log_start":"178'2400","ondisk_log_start":"178'2400","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8183,"log_dups_size":0,"ondisk_log_size":8183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18153472,"num_objects":277,"num_object_clones":0,"num_object_copies":831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10583,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":364,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.683912+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:54:14.683912+0000","last_peered":"2024-09-13T14:54:14.683912+0000","last_clean":"2024-09-13T14:54:14.683912+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:54:14.683912+0000","last_undegraded":"2024-09-13T14:54:14.683912+0000","last_fullsized":"2024-09-13T14:54:14.683912+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":806,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.335962+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:54:01.335962+0000","last_peered":"2024-09-13T14:54:01.335962+0000","last_clean":"2024-09-13T14:54:01.335962+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:54:01.335962+0000","last_undegraded":"2024-09-13T14:54:01.335962+0000","last_fullsized":"2024-09-13T14:54:01.335962+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":241,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336220+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:54:01.336220+0000","last_peered":"2024-09-13T14:54:01.336220+0000","last_clean":"2024-09-13T14:54:01.336220+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:54:01.336220+0000","last_undegraded":"2024-09-13T14:54:01.336220+0000","last_fullsized":"2024-09-13T14:54:01.336220+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":237,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336544+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:54:01.336544+0000","last_peered":"2024-09-13T14:54:01.336544+0000","last_clean":"2024-09-13T14:54:01.336544+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:54:01.336544+0000","last_undegraded":"2024-09-13T14:54:01.336544+0000","last_fullsized":"2024-09-13T14:54:01.336544+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":285,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:04.220003+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:54:04.220003+0000","last_peered":"2024-09-13T14:54:04.220003+0000","last_clean":"2024-09-13T14:54:04.220003+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:54:04.220003+0000","last_undegraded":"2024-09-13T14:54:04.220003+0000","last_fullsized":"2024-09-13T14:54:04.220003+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":368,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.670797+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:54:14.670797+0000","last_peered":"2024-09-13T14:54:14.670797+0000","last_clean":"2024-09-13T14:54:14.670797+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:54:14.670797+0000","last_undegraded":"2024-09-13T14:54:14.670797+0000","last_fullsized":"2024-09-13T14:54:14.670797+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":209,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.680547+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:54:14.680547+0000","last_peered":"2024-09-13T14:54:14.680547+0000","last_clean":"2024-09-13T14:54:14.680547+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:54:14.680547+0000","last_undegraded":"2024-09-13T14:54:14.680547+0000","last_fullsized":"2024-09-13T14:54:14.680547+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":117,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333322+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:54:01.333322+0000","last_peered":"2024-09-13T14:54:01.333322+0000","last_clean":"2024-09-13T14:54:01.333322+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:54:01.333322+0000","last_undegraded":"2024-09-13T14:54:01.333322+0000","last_fullsized":"2024-09-13T14:54:01.333322+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":346,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.679941+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:54:14.679941+0000","last_peered":"2024-09-13T14:54:14.679941+0000","last_clean":"2024-09-13T14:54:14.679941+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:54:14.679941+0000","last_undegraded":"2024-09-13T14:54:14.679941+0000","last_fullsized":"2024-09-13T14:54:14.679941+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":400,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.683955+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:54:14.683955+0000","last_peered":"2024-09-13T14:54:14.683955+0000","last_clean":"2024-09-13T14:54:14.683955+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:54:14.683955+0000","last_undegraded":"2024-09-13T14:54:14.683955+0000","last_fullsized":"2024-09-13T14:54:14.683955+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":397,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.687616+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:54:14.687616+0000","last_peered":"2024-09-13T14:54:14.687616+0000","last_clean":"2024-09-13T14:54:14.687616+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:54:14.687616+0000","last_undegraded":"2024-09-13T14:54:14.687616+0000","last_fullsized":"2024-09-13T14:54:14.687616+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":147,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.335882+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:54:01.335882+0000","last_peered":"2024-09-13T14:54:01.335882+0000","last_clean":"2024-09-13T14:54:01.335882+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:54:01.335882+0000","last_undegraded":"2024-09-13T14:54:01.335882+0000","last_fullsized":"2024-09-13T14:54:01.335882+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":364,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.690626+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:54:14.690626+0000","last_peered":"2024-09-13T14:54:14.690626+0000","last_clean":"2024-09-13T14:54:14.690626+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:54:14.690626+0000","last_undegraded":"2024-09-13T14:54:14.690626+0000","last_fullsized":"2024-09-13T14:54:14.690626+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":381,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.684175+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:54:14.684175+0000","last_peered":"2024-09-13T14:54:14.684175+0000","last_clean":"2024-09-13T14:54:14.684175+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:54:14.684175+0000","last_undegraded":"2024-09-13T14:54:14.684175+0000","last_fullsized":"2024-09-13T14:54:14.684175+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":349,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.670746+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:54:14.670746+0000","last_peered":"2024-09-13T14:54:14.670746+0000","last_clean":"2024-09-13T14:54:14.670746+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:54:14.670746+0000","last_undegraded":"2024-09-13T14:54:14.670746+0000","last_fullsized":"2024-09-13T14:54:14.670746+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":213,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.680453+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:54:14.680453+0000","last_peered":"2024-09-13T14:54:14.680453+0000","last_clean":"2024-09-13T14:54:14.680453+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:54:14.680453+0000","last_undegraded":"2024-09-13T14:54:14.680453+0000","last_fullsized":"2024-09-13T14:54:14.680453+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":235,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336421+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:54:01.336421+0000","last_peered":"2024-09-13T14:54:01.336421+0000","last_clean":"2024-09-13T14:54:01.336421+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:54:01.336421+0000","last_undegraded":"2024-09-13T14:54:01.336421+0000","last_fullsized":"2024-09-13T14:54:01.336421+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":108,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.682235+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:54:14.682235+0000","last_peered":"2024-09-13T14:54:14.682235+0000","last_clean":"2024-09-13T14:54:14.682235+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:54:14.682235+0000","last_undegraded":"2024-09-13T14:54:14.682235+0000","last_fullsized":"2024-09-13T14:54:14.682235+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":373,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.684503+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:54:14.684503+0000","last_peered":"2024-09-13T14:54:14.684503+0000","last_clean":"2024-09-13T14:54:14.684503+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:54:14.684503+0000","last_undegraded":"2024-09-13T14:54:14.684503+0000","last_fullsized":"2024-09-13T14:54:14.684503+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":245,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336470+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:54:01.336470+0000","last_peered":"2024-09-13T14:54:01.336470+0000","last_clean":"2024-09-13T14:54:01.336470+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:54:01.336470+0000","last_undegraded":"2024-09-13T14:54:01.336470+0000","last_fullsized":"2024-09-13T14:54:01.336470+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":92,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333312+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:54:01.333312+0000","last_peered":"2024-09-13T14:54:01.333312+0000","last_clean":"2024-09-13T14:54:01.333312+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:54:01.333312+0000","last_undegraded":"2024-09-13T14:54:01.333312+0000","last_fullsized":"2024-09-13T14:54:01.333312+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":233,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336379+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:54:01.336379+0000","last_peered":"2024-09-13T14:54:01.336379+0000","last_clean":"2024-09-13T14:54:01.336379+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:54:01.336379+0000","last_undegraded":"2024-09-13T14:54:01.336379+0000","last_fullsized":"2024-09-13T14:54:01.336379+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":94,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.333476+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:54:01.333476+0000","last_peered":"2024-09-13T14:54:01.333476+0000","last_clean":"2024-09-13T14:54:01.333476+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:54:01.333476+0000","last_undegraded":"2024-09-13T14:54:01.333476+0000","last_fullsized":"2024-09-13T14:54:01.333476+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":367,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.680837+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:54:14.680837+0000","last_peered":"2024-09-13T14:54:14.680837+0000","last_clean":"2024-09-13T14:54:14.680837+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:54:14.680837+0000","last_undegraded":"2024-09-13T14:54:14.680837+0000","last_fullsized":"2024-09-13T14:54:14.680837+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":383,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.680678+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:54:14.680678+0000","last_peered":"2024-09-13T14:54:14.680678+0000","last_clean":"2024-09-13T14:54:14.680678+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:54:14.680678+0000","last_undegraded":"2024-09-13T14:54:14.680678+0000","last_fullsized":"2024-09-13T14:54:14.680678+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":263,"reported_epoch":253,"state":"active+clean","last_fresh":"2024-09-13T14:54:01.336220+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:54:01.336220+0000","last_peered":"2024-09-13T14:54:01.336220+0000","last_clean":"2024-09-13T14:54:01.336220+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:54:01.336220+0000","last_undegraded":"2024-09-13T14:54:01.336220+0000","last_fullsized":"2024-09-13T14:54:01.336220+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":436,"reported_epoch":254,"state":"active+clean","last_fresh":"2024-09-13T14:54:14.683841+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:54:14.683841+0000","last_peered":"2024-09-13T14:54:14.683841+0000","last_clean":"2024-09-13T14:54:14.683841+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:54:14.683841+0000","last_undegraded":"2024-09-13T14:54:14.683841+0000","last_fullsized":"2024-09-13T14:54:14.683841+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":270991384,"num_objects":4136,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1902,"num_objects_misplaced":196,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167412,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44763,"num_bytes_recovered":2809856000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1084829696,"data_stored":1084817480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128284,"ondisk_log_size":128284,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495959,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487188,"kb_used_data":153356,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93257964,"statfs":{"total":95995035648,"available":95496155136,"internally_reserved":0,"allocated":157036544,"data_stored":155668436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757594,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306892,"kb_used_data":95300,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":93438260,"statfs":{"total":95995035648,"available":95680778240,"internally_reserved":0,"allocated":97587200,"data_stored":96222887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":216660423},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019234,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":394900,"kb_used_data":93068,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":93350252,"statfs":{"total":95995035648,"available":95590658048,"internally_reserved":0,"allocated":95301632,"data_stored":93933585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309066182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468119,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":468936,"kb_used_data":170752,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93276216,"statfs":{"total":95995035648,"available":95514845184,"internally_reserved":0,"allocated":174850048,"data_stored":173489581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":305327268},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631648,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555160,"kb_used_data":162512,"kb_used_omap":8,"kb_used_meta":392631,"kb_avail":93189992,"statfs":{"total":95995035648,"available":95426551808,"internally_reserved":0,"allocated":166412288,"data_stored":165040671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":402055135},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795171,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645712,"kb_used_data":347464,"kb_used_omap":7,"kb_used_meta":298232,"kb_avail":93099440,"statfs":{"total":95995035648,"available":95333826560,"internally_reserved":0,"allocated":355803136,"data_stored":354434926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":305390194},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958701,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":393684,"kb_used_data":101004,"kb_used_omap":7,"kb_used_meta":292664,"kb_avail":93351468,"statfs":{"total":95995035648,"available":95591903232,"internally_reserved":0,"allocated":103428096,"data_stored":102056443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":299688423},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87035904,"data_stored":87031832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":333713408,"data_stored":333709336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":156041216,"data_stored":156041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":150667264,"data_stored":150667264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":85852160,"data_stored":85852160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":82972672,"data_stored":82968600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":136577024,"data_stored":136577024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":51970048,"data_stored":51970048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:54:22.538 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:54:22.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:22.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:22.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:22.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:22.805 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:22 smithi154 ceph-mon[107038]: pgmap v772: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 263 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 1902/12492 objects degraded (15.226%); 196/12492 objects misplaced (1.569%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:22.805 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:22 smithi154 ceph-mon[107038]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:22.805 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:22 smithi154 ceph-mon[102314]: pgmap v772: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 263 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 1902/12492 objects degraded (15.226%); 196/12492 objects misplaced (1.569%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:22.805 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:22 smithi154 ceph-mon[102314]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:22.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:22.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:22.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:22.916 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:22 smithi172 ceph-mon[100696]: pgmap v772: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 263 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 1902/12492 objects degraded (15.226%); 196/12492 objects misplaced (1.569%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:22.916 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:22 smithi172 ceph-mon[100696]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:23.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:23.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:23.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:23.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:23.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:23.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:23.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:23.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:23.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:23.643 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:23.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:23.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:23.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:23.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:23.923 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:23.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:24.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:24.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:24.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:24.195 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:24.297 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:97375 5.9 5:9dd925ab:::benchmark_data_smithi154_144140_object97374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 595.36379866599998, 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.11292851499999999, 2024-09-13T14:54:24.298 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 97375 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930167+0000", 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.930165+0000", 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975030+0000", 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044864996999999997 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975066+0000", 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5460999999999998e-05 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.975076+0000", 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0397999999999999e-05 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.979893+0000", 2024-09-13T14:54:24.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048163379999999999 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.980059+0000", 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016674300000000001 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.980241+0000", 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018151899999999999 2024-09-13T14:54:24.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.988970+0000", 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087295719999999997 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:28.989030+0000", 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9859999999999999e-05 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.040919+0000", 2024-09-13T14:54:24.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.051888722999999998 2024-09-13T14:54:24.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.040974+0000", 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4549999999999998e-05 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:29.043096+0000", 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002122199 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.306 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:113747 5.9 5:91f8fbcd:::benchmark_data_smithi154_144140_object3394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 572.05334317799998, 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.091457920999999998, 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.307 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113747 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240623+0000", 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240622+0000", 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240625+0000", 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7530000000000002e-06 2024-09-13T14:54:24.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240630+0000", 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2979999999999999e-06 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.240635+0000", 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3989999999999997e-06 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247577+0000", 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0069425770000000001 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247719+0000", 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014185900000000001 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.247864+0000", 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014463900000000001 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.250756+0000", 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028924279999999998 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.251141+0000", 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038421899999999999 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.330437+0000", 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079296273 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.330474+0000", 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7418e-05 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:52.332081+0000", 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001606447 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:114992 5.9 5:9bf9d9df:::benchmark_data_smithi154_144140_object4639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 570.30931113099996, 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.85779442800000005, 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114992 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984655+0000", 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984654+0000", 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984658+0000", 2024-09-13T14:54:24.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7450000000000001e-06 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984665+0000", 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5020000000000003e-06 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984675+0000", 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.132e-06 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984715+0000", 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0701000000000002e-05 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:53.984723+0000", 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6140000000000003e-06 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824713+0000", 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.83998967899999999 2024-09-13T14:54:24.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824847+0000", 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000134356 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.824939+0000", 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1880999999999997e-05 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830175+0000", 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052358680000000003 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.833865+0000", 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003690189 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.840699+0000", 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00683411 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.840754+0000", 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5126e-05 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842449+0000", 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016951170000000001 2024-09-13T14:54:24.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:115007 5.9 5:93772757:::benchmark_data_smithi154_144140_object4654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 570.28129875000002, 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.83119907900000001, 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.323 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115007 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012667+0000", 2024-09-13T14:54:24.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012666+0000", 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012675+0000", 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6880000000000005e-06 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012685+0000", 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.696e-06 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012692+0000", 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8549999999999996e-06 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012728+0000", 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6328000000000002e-05 2024-09-13T14:54:24.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.012736+0000", 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.011e-06 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830212+0000", 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81747611099999995 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830363+0000", 2024-09-13T14:54:24.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015098299999999999 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.830536+0000", 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173227 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834179+0000", 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003643178 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838231+0000", 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040517900000000004 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842523+0000", 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042920570000000002 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.842581+0000", 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7868999999999998e-05 2024-09-13T14:54:24.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843866+0000", 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001285071 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:115010 5.9 5:9c0819f0:::benchmark_data_smithi154_144140_object4657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 570.27678816599996, 2024-09-13T14:54:24.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.82826413700000001, 2024-09-13T14:54:24.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.331 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.331 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.331 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.332 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115010 2024-09-13T14:54:24.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017178+0000", 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017177+0000", 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017181+0000", 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6619999999999998e-06 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017188+0000", 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0960000000000003e-06 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017192+0000", 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0330000000000002e-06 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017219+0000", 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7807999999999999e-05 2024-09-13T14:54:24.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.017226+0000", 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4629999999999998e-06 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834282+0000", 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.81705591399999999 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834452+0000", 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170314 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4,6", 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.834602+0000", 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149732 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838287+0000", 2024-09-13T14:54:24.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00368491 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.838568+0000", 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028085099999999998 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843931+0000", 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053631590000000002 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.843991+0000", 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0174999999999999e-05 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:54.845442+0000", 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001450914 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117184 5.4 5:285a1434:::benchmark_data_smithi154_144140_object6831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:54:24.339 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:54:24.340 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 567.123610214, 2024-09-13T14:54:24.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15315772, 2024-09-13T14:54:24.340 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117184 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170356+0000", 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170355+0000", 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:24.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170358+0000", 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.19e-06 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170365+0000", 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9419999999999998e-06 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.170371+0000", 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7899999999999996e-06 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174205+0000", 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038343880000000002 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174315+0000", 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109225 2024-09-13T14:54:24.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.174416+0000", 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010148 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.177943+0000", 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035270409999999999 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.183599+0000", 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056554109999999999 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323412+0000", 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.13981297000000001 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323466+0000", 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4549000000000002e-05 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.323513+0000", 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7312999999999999e-05 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117538 5.4 5:2cd2fa81:::benchmark_data_smithi154_144140_object7185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 566.65499050799997, 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.140160653, 2024-09-13T14:54:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117538 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638975+0000", 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638974+0000", 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638981+0000", 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3900000000000004e-06 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.638999+0000", 2024-09-13T14:54:24.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7948000000000002e-05 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639007+0000", 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9670000000000001e-06 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639049+0000", 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1593e-05 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639234+0000", 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185253 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639344+0000", 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109447 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648634+0000", 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092907010000000002 2024-09-13T14:54:24.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648647+0000", 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3182e-05 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779063+0000", 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.130415064 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779100+0000", 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7110000000000002e-05 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779136+0000", 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6436999999999997e-05 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117539 5.4 5:27e8a4d2:::benchmark_data_smithi154_144140_object7186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:54:24.354 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 566.65478330999997, 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.140068675, 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117539 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639183+0000", 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639182+0000", 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639187+0000", 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.758e-06 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639198+0000", 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1674999999999999e-05 2024-09-13T14:54:24.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.639205+0000", 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5409999999999997e-06 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644291+0000", 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050859499999999997 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644444+0000", 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015357700000000001 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.644545+0000", 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010090999999999999 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655177+0000", 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010632157 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.776212+0000", 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.121034903 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779189+0000", 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029772129999999998 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779215+0000", 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5933999999999999e-05 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779251+0000", 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5936000000000003e-05 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117544 5.4 5:239bf146:::benchmark_data_smithi154_144140_object7191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 566.65308099499998, 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.138439275, 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.362 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117544 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640885+0000", 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640884+0000", 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640889+0000", 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7640000000000002e-06 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640902+0000", 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3046e-05 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.640909+0000", 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1840000000000004e-06 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648692+0000", 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077836470000000003 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648798+0000", 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010568700000000001 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648898+0000", 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100207 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655188+0000", 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062894869999999999 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.776255+0000", 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12106747599999999 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779281+0000", 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030257999999999999 2024-09-13T14:54:24.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779305+0000", 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4136000000000001e-05 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.779324+0000", 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9199999999999999e-05 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.369 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117548 5.4 5:28e11507:::benchmark_data_smithi154_144140_object7195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:54:24.369 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:54:24.369 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 566.64677135199997, 2024-09-13T14:54:24.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15636461800000001, 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117548 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.370 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647195+0000", 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647193+0000", 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647199+0000", 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4439999999999999e-06 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647209+0000", 2024-09-13T14:54:24.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0275000000000001e-05 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.647215+0000", 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1040000000000003e-06 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652094+0000", 2024-09-13T14:54:24.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048785010000000004 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652244+0000", 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015033599999999999 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.652372+0000", 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012824700000000001 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.658282+0000", 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0059094480000000003 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.783415+0000", 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12513291300000001 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803465+0000", 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020050179000000001 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803514+0000", 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8661e-05 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803559+0000", 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5626e-05 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.376 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117551 5.4 5:28c6b1cf:::benchmark_data_smithi154_144140_object7198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 566.64526491900006, 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.15501442800000001, 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117551 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648701+0000", 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648700+0000", 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648704+0000", 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4279999999999998e-06 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648712+0000", 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0320000000000003e-06 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.648718+0000", 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5620000000000003e-06 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655217+0000", 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064988140000000003 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655329+0000", 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112169 2024-09-13T14:54:24.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.655446+0000", 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116567 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.658387+0000", 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029411540000000001 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.783571+0000", 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12518472899999999 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803616+0000", 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020044480999999999 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803682+0000", 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6157999999999999e-05 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:57.803715+0000", 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3432000000000001e-05 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117857 5.4 5:20ac363d:::benchmark_data_smithi154_144140_object7504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:54:24.383 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 566.19327718299996, 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12765149100000001, 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117857 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100689+0000", 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100687+0000", 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100696+0000", 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5680000000000005e-06 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100716+0000", 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9788000000000001e-05 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100724+0000", 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4519999999999997e-06 2024-09-13T14:54:24.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100794+0000", 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9370999999999997e-05 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.100907+0000", 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113435 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.101008+0000", 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100407 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107808+0000", 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068003430000000004 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107856+0000", 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7775000000000003e-05 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228218+0000", 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12036281 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228302+0000", 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3688000000000002e-05 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228340+0000", 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8090999999999998e-05 2024-09-13T14:54:24.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:117860 5.4 5:29f284c5:::benchmark_data_smithi154_144140_object7507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 566.19013334199997, 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.124665336, 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117860 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103833+0000", 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103831+0000", 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103839+0000", 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8129999999999996e-06 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103855+0000", 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6900000000000001e-05 2024-09-13T14:54:24.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.103864+0000", 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000003e-06 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104514+0000", 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000650856 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104600+0000", 2024-09-13T14:54:24.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5352999999999994e-05 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.104680+0000", 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0687999999999997e-05 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.107887+0000", 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032067060000000001 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228173+0000", 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12028613 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228378+0000", 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020511699999999999 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228456+0000", 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7156000000000005e-05 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:44:58.228498+0000", 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2373000000000002e-05 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:172124 5.4 5:24c92774:::benchmark_data_smithi154_144140_object61771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:06.268190+0000", 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 498.02577635599999, 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.115191713, 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.396 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 172124 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268190+0000", 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268190+0000", 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268188+0000", 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268195+0000", 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8820000000000003e-06 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268210+0000", 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4799e-05 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268215+0000", 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.604e-06 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268251+0000", 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6044000000000003e-05 2024-09-13T14:54:24.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268339+0000", 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7818000000000005e-05 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.268416+0000", 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7059999999999997e-05 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.271372+0000", 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029560469999999998 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.283085+0000", 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011712573 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383290+0000", 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.100204814 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383354+0000", 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4091999999999996e-05 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383381+0000", 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7559000000000002e-05 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.402 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:172133 5.4 5:24332ad7:::benchmark_data_smithi154_144140_object61780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:06.276130+0000", 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 498.01783571599998, 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10733804399999999, 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 172133 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276130+0000", 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276130+0000", 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276129+0000", 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276134+0000", 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6999999999999999e-06 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276146+0000", 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276151+0000", 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9450000000000001e-06 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276196+0000", 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5734e-05 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276358+0000", 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016167800000000001 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.276486+0000", 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127806 2024-09-13T14:54:24.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.280384+0000", 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038979990000000001 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.283921+0000", 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035375150000000002 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383414+0000", 2024-09-13T14:54:24.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.099493087999999993 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383446+0000", 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1621000000000001e-05 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:06.383468+0000", 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2294999999999999e-05 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.408 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.409 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:173360 5.4 5:2b9b10d0:::benchmark_data_smithi154_144140_object63007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-09-13T14:54:24.409 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:54:24.409 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 496.35844292600001, 2024-09-13T14:54:24.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10636651599999999, 2024-09-13T14:54:24.409 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 173360 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935523+0000", 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935525+0000", 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935531+0000", 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7969999999999997e-06 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.935536+0000", 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0270000000000003e-06 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.941411+0000", 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058755359999999998 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.941518+0000", 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106265 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.941619+0000", 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101735 2024-09-13T14:54:24.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.947007+0000", 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053876490000000004 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:07.948811+0000", 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001803526 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:08.041799+0000", 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.092988238000000001 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:08.041835+0000", 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5636000000000002e-05 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:08.041890+0000", 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5003000000000002e-05 2024-09-13T14:54:24.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:193138 5.4 5:259154cc:::benchmark_data_smithi154_144140_object82785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:46:33.885070+0000", 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 470.40889628399998, 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.087953613, 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.417 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193138 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885070+0000", 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885070+0000", 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885068+0000", 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885076+0000", 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9859999999999994e-06 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885092+0000", 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6833e-05 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885101+0000", 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3680000000000002e-06 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885151+0000", 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0373e-05 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885278+0000", 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012707699999999999 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.885400+0000", 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012204700000000001 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.888403+0000", 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030024539999999999 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.963692+0000", 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.075288788999999995 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.972936+0000", 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0092441669999999993 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.972985+0000", 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9475000000000003e-05 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:46:33.973023+0000", 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8074999999999997e-05 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:215616 5.4 5:24d075d9:::benchmark_data_smithi154_144140_object105263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:47:03.377793+0000", 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 440.91617254699997, 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12910099899999999, 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 215616 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377793+0000", 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377793+0000", 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377792+0000", 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:24.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377799+0000", 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1740000000000003e-06 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377813+0000", 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3534e-05 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377820+0000", 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7279999999999995e-06 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.377877+0000", 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6202000000000002e-05 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.378006+0000", 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129111 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.378116+0000", 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110344 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382011+0000", 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038949259999999999 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.397756+0000", 2024-09-13T14:54:24.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015745037 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506830+0000", 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.109073429 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506867+0000", 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7662e-05 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506894+0000", 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7200999999999999e-05 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:215621 5.4 5:28250e67:::benchmark_data_smithi154_144140_object105268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:47:03.382845+0000", 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 440.91112049100002, 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12415194, 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 215621 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382845+0000", 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382845+0000", 2024-09-13T14:54:24.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382844+0000", 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382851+0000", 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1600000000000001e-06 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382864+0000", 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2646999999999999e-05 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382872+0000", 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9519999999999994e-06 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382911+0000", 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9765999999999998e-05 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.382986+0000", 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5123000000000006e-05 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.383055+0000", 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8615999999999995e-05 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.386242+0000", 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031869229999999999 2024-09-13T14:54:24.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.402788+0000", 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016545814999999998 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506944+0000", 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10415627099999999 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506975+0000", 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.074e-05 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.506997+0000", 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2572999999999999e-05 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.34620.0:215632 5.4 5:23b35507:::benchmark_data_smithi154_144140_object105279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:54:24.436 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 440.90053598600002, 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.113641928, 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.34620", 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 215632 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393430+0000", 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393432+0000", 2024-09-13T14:54:24.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3949999999999999e-06 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393438+0000", 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6350000000000003e-06 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393442+0000", 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8149999999999996e-06 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393493+0000", 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0192999999999999e-05 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393607+0000", 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114048 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,2", 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.393727+0000", 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120284 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.397785+0000", 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040582680000000003 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.402931+0000", 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051453310000000004 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.507022+0000", 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10409193999999999 2024-09-13T14:54:24.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.507047+0000", 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4746999999999999e-05 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:47:03.507072+0000", 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4695e-05 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:54:24.443 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:54:24.443 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:54:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:24.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:24.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:24.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:24.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:24.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:24.732 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:54:24.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:24.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:24.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:24.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:24.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:24.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:24.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:24 smithi154 ceph-mon[102314]: pgmap v773: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 263 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 1902/12492 objects degraded (15.226%); 196/12492 objects misplaced (1.569%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:24.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2003542573' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:24.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:24 smithi154 ceph-mon[107038]: pgmap v773: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 263 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 1902/12492 objects degraded (15.226%); 196/12492 objects misplaced (1.569%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:24.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2003542573' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:24 smithi172 ceph-mon[100696]: pgmap v773: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 263 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 1902/12492 objects degraded (15.226%); 196/12492 objects misplaced (1.569%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2003542573' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:54:25.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:25.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:25.138 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Removed 85773 objects 2024-09-13T14:54:25.138 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Clean up completed and total clean up time :279.898 2024-09-13T14:54:25.209 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:25.231 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:25.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:25.354 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:25.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:25.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:25.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:54:25.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:25.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:25.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:25.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:25.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:25.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:25.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:54:25.947 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:25.947 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2003542573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:25.947 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[102314]: osdmap e255: 8 total, 8 up, 7 in 2024-09-13T14:54:25.947 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1902/12492 objects degraded (15.226%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:25.948 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:25.948 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2003542573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:25.948 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[107038]: osdmap e255: 8 total, 8 up, 7 in 2024-09-13T14:54:25.948 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1902/12492 objects degraded (15.226%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:25.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:25.972 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:25.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:25 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:25 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2003542573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:54:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:25 smithi172 ceph-mon[100696]: osdmap e255: 8 total, 8 up, 7 in 2024-09-13T14:54:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1902/12492 objects degraded (15.226%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:26.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:26.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:26.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:26.218 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:26.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:26.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:26.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:26.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:26.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:26.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:26.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:26.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:26.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:26.834 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:26.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:26.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:26.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:26 smithi172 ceph-mon[100696]: pgmap v775: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 77 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 725 op/s; 1902/3549 objects degraded (53.593%); 196/3549 objects misplaced (5.523%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:27.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:27.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:27.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:27.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:26 smithi154 ceph-mon[102314]: pgmap v775: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 77 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 725 op/s; 1902/3549 objects degraded (53.593%); 196/3549 objects misplaced (5.523%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:27.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:26 smithi154 ceph-mon[107038]: pgmap v775: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 77 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 725 op/s; 1902/3549 objects degraded (53.593%); 196/3549 objects misplaced (5.523%); 249 KiB/s, 7 objects/s recovering 2024-09-13T14:54:27.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:27.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:27.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:27.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:27.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:27.575 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:27.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:27.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:27.723 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-13T14:54:27.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-13T14:54:27.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:27.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:27.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:27.946 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:27.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:28.048 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:28.069 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:28.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:28.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:28.192 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:28.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:28.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:28.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:28.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:28.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:28.562 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:28.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:28.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:28.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:28.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:28.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:28.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:28.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:28 smithi172 ceph-mon[100696]: pgmap v776: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 38 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 1922/1710 objects degraded (112.398%); 1.3 MiB/s, 33 objects/s recovering 2024-09-13T14:54:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:29.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:29.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:29.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:29.179 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:28 smithi154 ceph-mon[102314]: pgmap v776: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 38 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 1922/1710 objects degraded (112.398%); 1.3 MiB/s, 33 objects/s recovering 2024-09-13T14:54:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:29.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:28 smithi154 ceph-mon[107038]: pgmap v776: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 38 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 1922/1710 objects degraded (112.398%); 1.3 MiB/s, 33 objects/s recovering 2024-09-13T14:54:29.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:29.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:29.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:29.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:29.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:29.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:29.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:29.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:29.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.799 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:29.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:29.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:29.921 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:29.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:30.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.045 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:30.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:30.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:30.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:30.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:30.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:30.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:30.412 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:30.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:30.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:30.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:30.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:30.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:30.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:30.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:30.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:30.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:30.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:30 smithi172 ceph-mon[100696]: pgmap v777: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 20 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 1922/819 objects degraded (234.676%); 1.3 MiB/s, 33 objects/s recovering 2024-09-13T14:54:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1922/819 objects degraded (234.676%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:30.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:30 smithi154 ceph-mon[107038]: pgmap v777: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 20 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 1922/819 objects degraded (234.676%); 1.3 MiB/s, 33 objects/s recovering 2024-09-13T14:54:30.996 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1922/819 objects degraded (234.676%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:30.996 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:30 smithi154 ceph-mon[102314]: pgmap v777: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 20 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 1922/819 objects degraded (234.676%); 1.3 MiB/s, 33 objects/s recovering 2024-09-13T14:54:30.996 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1922/819 objects degraded (234.676%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:31.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:31.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:31.079 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:31.080 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:31.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:31.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:31.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:31.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:31.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:31.404 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:31.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:31.526 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:31.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:31.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:31.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:31.758 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:54:31.758 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:31.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:31.789 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:54:31.789 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:31.819 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:54:31.819 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:31.835 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:54:31.835 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:31.851 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:54:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:31.867 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:54:31.868 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:31.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:31.883 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:54:31.884 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:31.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:31.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:31.903 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:54:31.903 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:32.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:32.015 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:31 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/34497020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:32.016 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:54:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:54:31] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T14:54:32.016 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:31 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/34497020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:32.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:32.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:32.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:31 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/34497020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:32.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:32.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:32.373 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:32.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:32.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:54:32.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:32.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:32.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:32.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:32.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:32.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:32.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:32.890 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:32.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:32.947 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:32 smithi154 ceph-mon[102314]: pgmap v778: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 579/84 objects degraded (689.286%); 1.1 MiB/s, 160 objects/s recovering 2024-09-13T14:54:32.947 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:32 smithi154 ceph-mon[107038]: pgmap v778: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 579/84 objects degraded (689.286%); 1.1 MiB/s, 160 objects/s recovering 2024-09-13T14:54:32.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:33.012 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:33.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:33.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:33.184 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:33.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:32 smithi172 ceph-mon[100696]: pgmap v778: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 579/84 objects degraded (689.286%); 1.1 MiB/s, 160 objects/s recovering 2024-09-13T14:54:33.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:33.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:33.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:33.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:33.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:33.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:33.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:33.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:33.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:33.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:33.883 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:33.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:33.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:33 smithi154 ceph-mon[102314]: osdmap e256: 8 total, 8 up, 7 in 2024-09-13T14:54:33.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:33 smithi154 ceph-mon[107038]: osdmap e256: 8 total, 8 up, 7 in 2024-09-13T14:54:33.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:34.007 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:34.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:34.131 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:34.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:33 smithi172 ceph-mon[100696]: osdmap e256: 8 total, 8 up, 7 in 2024-09-13T14:54:34.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:34.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:34.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:34.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:34.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:34.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:34.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:34.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:34.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:34.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:54:34.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:34.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:34.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:34.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:34.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:35.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:35.109 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:34 smithi154 ceph-mon[102314]: pgmap v780: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 193 op/s; 579/84 objects degraded (689.286%); 1.3 MiB/s, 193 objects/s recovering 2024-09-13T14:54:35.109 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:34 smithi154 ceph-mon[102314]: osdmap e257: 8 total, 8 up, 7 in 2024-09-13T14:54:35.110 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:34 smithi154 ceph-mon[107038]: pgmap v780: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 193 op/s; 579/84 objects degraded (689.286%); 1.3 MiB/s, 193 objects/s recovering 2024-09-13T14:54:35.110 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:34 smithi154 ceph-mon[107038]: osdmap e257: 8 total, 8 up, 7 in 2024-09-13T14:54:35.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:35.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:34 smithi172 ceph-mon[100696]: pgmap v780: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 193 op/s; 579/84 objects degraded (689.286%); 1.3 MiB/s, 193 objects/s recovering 2024-09-13T14:54:35.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:34 smithi172 ceph-mon[100696]: osdmap e257: 8 total, 8 up, 7 in 2024-09-13T14:54:35.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:35.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:35.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:35.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:35.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:35.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:35.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:35.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:35.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:35.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:35.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:54:35.992 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:54:36.071 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 579/84 objects degraded (689.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:36.072 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 579/84 objects degraded (689.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:36.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.265 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:36.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:36.265 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 579/84 objects degraded (689.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-13T14:54:36.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.399 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:36.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:36.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:36.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:36.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.654 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:36.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:36.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:36.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:36.849 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_1' removed 2024-09-13T14:54:36.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:36.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:36.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:37.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:37.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:37.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:37.136 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:36 smithi154 ceph-mon[102314]: pgmap v782: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 67 op/s; 579/84 objects degraded (689.286%); 0 B/s, 168 objects/s recovering 2024-09-13T14:54:37.136 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1655293045' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:37.136 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:36 smithi154 ceph-mon[107038]: pgmap v782: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 67 op/s; 579/84 objects degraded (689.286%); 0 B/s, 168 objects/s recovering 2024-09-13T14:54:37.136 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1655293045' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:37.154 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2024-09-13T14:54:35.033944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":172671,"num_write_kb":5504514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46528,"num_bytes_recovered":2823769221,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129702,"ondisk_log_size":129702,"up":267,"acting":266,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":269,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2457728,"kb_used_data":269416,"kb_used_omap":32,"kb_used_meta":2188191,"kb_avail":653758336,"statfs":{"total":671965249536,"available":669448536064,"internally_reserved":0,"allocated":275881984,"data_stored":266262562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2240708144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35455000,"num_objects":-542,"num_object_clones":0,"num_object_copies":-1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1343,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001923"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":392,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.739188+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:54:24.739188+0000","last_peered":"2024-09-13T14:54:24.739188+0000","last_clean":"2024-09-13T14:54:24.739188+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:54:24.739188+0000","last_undegraded":"2024-09-13T14:54:24.739188+0000","last_fullsized":"2024-09-13T14:54:24.739188+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":123,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749806+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:54:32.749806+0000","last_peered":"2024-09-13T14:54:32.749806+0000","last_clean":"2024-09-13T14:54:32.749806+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:54:32.749806+0000","last_undegraded":"2024-09-13T14:54:32.749806+0000","last_fullsized":"2024-09-13T14:54:32.749806+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":93,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759385+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:54:32.759385+0000","last_peered":"2024-09-13T14:54:32.759385+0000","last_clean":"2024-09-13T14:54:32.759385+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:54:32.759385+0000","last_undegraded":"2024-09-13T14:54:32.759385+0000","last_fullsized":"2024-09-13T14:54:32.759385+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":382,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733380+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:54:24.733380+0000","last_peered":"2024-09-13T14:54:24.733380+0000","last_clean":"2024-09-13T14:54:24.733380+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:54:24.733380+0000","last_undegraded":"2024-09-13T14:54:24.733380+0000","last_fullsized":"2024-09-13T14:54:24.733380+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":382,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733339+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:54:24.733339+0000","last_peered":"2024-09-13T14:54:24.733339+0000","last_clean":"2024-09-13T14:54:24.733339+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:54:24.733339+0000","last_undegraded":"2024-09-13T14:54:24.733339+0000","last_fullsized":"2024-09-13T14:54:24.733339+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":193,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749536+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:54:32.749536+0000","last_peered":"2024-09-13T14:54:32.749536+0000","last_clean":"2024-09-13T14:54:32.749536+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:54:32.749536+0000","last_undegraded":"2024-09-13T14:54:32.749536+0000","last_fullsized":"2024-09-13T14:54:32.749536+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":107,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733974+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:54:24.733974+0000","last_peered":"2024-09-13T14:54:24.733974+0000","last_clean":"2024-09-13T14:54:24.733974+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:54:24.733974+0000","last_undegraded":"2024-09-13T14:54:24.733974+0000","last_fullsized":"2024-09-13T14:54:24.733974+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":170,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.760032+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:54:32.760032+0000","last_peered":"2024-09-13T14:54:32.760032+0000","last_clean":"2024-09-13T14:54:32.760032+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:54:32.760032+0000","last_undegraded":"2024-09-13T14:54:32.760032+0000","last_fullsized":"2024-09-13T14:54:32.760032+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"255'10896","reported_seq":11748,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759134+0000","last_change":"2024-09-13T14:49:35.373298+0000","last_active":"2024-09-13T14:54:32.759134+0000","last_peered":"2024-09-13T14:54:32.759134+0000","last_clean":"2024-09-13T14:54:32.759134+0000","last_became_active":"2024-09-13T14:49:35.367838+0000","last_became_peered":"2024-09-13T14:49:35.367838+0000","last_unstale":"2024-09-13T14:54:32.759134+0000","last_undegraded":"2024-09-13T14:54:32.759134+0000","last_fullsized":"2024-09-13T14:54:32.759134+0000","mapping_epoch":195,"log_start":"167'800","ondisk_log_start":"167'800","created":160,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:02.739741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":348672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2258,"num_bytes_recovered":147980288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":320,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.936257+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:54:33.936257+0000","last_peered":"2024-09-13T14:54:33.936257+0000","last_clean":"2024-09-13T14:54:33.936257+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:54:33.936257+0000","last_undegraded":"2024-09-13T14:54:33.936257+0000","last_fullsized":"2024-09-13T14:54:33.936257+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":120,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.758685+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:54:32.758685+0000","last_peered":"2024-09-13T14:54:32.758685+0000","last_clean":"2024-09-13T14:54:32.758685+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:54:32.758685+0000","last_undegraded":"2024-09-13T14:54:32.758685+0000","last_fullsized":"2024-09-13T14:54:32.758685+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"255'10578","reported_seq":12218,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:25.128506+0000","last_change":"2024-09-13T14:48:10.765173+0000","last_active":"2024-09-13T14:54:25.128506+0000","last_peered":"2024-09-13T14:54:25.128506+0000","last_clean":"2024-09-13T14:54:25.128506+0000","last_became_active":"2024-09-13T14:48:10.762836+0000","last_became_peered":"2024-09-13T14:48:10.762836+0000","last_unstale":"2024-09-13T14:54:25.128506+0000","last_undegraded":"2024-09-13T14:54:25.128506+0000","last_fullsized":"2024-09-13T14:54:25.128506+0000","mapping_epoch":170,"log_start":"189'4100","ondisk_log_start":"189'4100","created":160,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6478,"log_dups_size":0,"ondisk_log_size":6478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:41.444070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10578,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":121,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.750194+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:54:32.750194+0000","last_peered":"2024-09-13T14:54:32.750194+0000","last_clean":"2024-09-13T14:54:32.750194+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:54:32.750194+0000","last_undegraded":"2024-09-13T14:54:32.750194+0000","last_fullsized":"2024-09-13T14:54:32.750194+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":274,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749678+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:54:32.749678+0000","last_peered":"2024-09-13T14:54:32.749678+0000","last_clean":"2024-09-13T14:54:32.749678+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:54:32.749678+0000","last_undegraded":"2024-09-13T14:54:32.749678+0000","last_fullsized":"2024-09-13T14:54:32.749678+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"255'10844","reported_seq":12581,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.934650+0000","last_change":"2024-09-13T14:48:20.091425+0000","last_active":"2024-09-13T14:54:33.934650+0000","last_peered":"2024-09-13T14:54:33.934650+0000","last_clean":"2024-09-13T14:54:33.934650+0000","last_became_active":"2024-09-13T14:48:20.088960+0000","last_became_peered":"2024-09-13T14:48:20.088960+0000","last_unstale":"2024-09-13T14:54:33.934650+0000","last_undegraded":"2024-09-13T14:54:33.934650+0000","last_fullsized":"2024-09-13T14:54:33.934650+0000","mapping_epoch":175,"log_start":"173'2000","ondisk_log_start":"173'2000","created":160,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8844,"log_dups_size":0,"ondisk_log_size":8844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:14.115026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":347008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":762,"num_bytes_recovered":49938432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":97,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759522+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:54:32.759522+0000","last_peered":"2024-09-13T14:54:32.759522+0000","last_clean":"2024-09-13T14:54:32.759522+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:54:32.759522+0000","last_undegraded":"2024-09-13T14:54:32.759522+0000","last_fullsized":"2024-09-13T14:54:32.759522+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":53,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759719+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:54:32.759719+0000","last_peered":"2024-09-13T14:54:32.759719+0000","last_clean":"2024-09-13T14:54:32.759719+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:54:32.759719+0000","last_undegraded":"2024-09-13T14:54:32.759719+0000","last_fullsized":"2024-09-13T14:54:32.759719+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"255'10508","reported_seq":16636,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:25.121224+0000","last_change":"2024-09-13T14:52:50.434519+0000","last_active":"2024-09-13T14:54:25.121224+0000","last_peered":"2024-09-13T14:54:25.121224+0000","last_clean":"2024-09-13T14:54:25.121224+0000","last_became_active":"2024-09-13T14:52:50.417699+0000","last_became_peered":"2024-09-13T14:52:50.417699+0000","last_unstale":"2024-09-13T14:54:25.121224+0000","last_undegraded":"2024-09-13T14:54:25.121224+0000","last_fullsized":"2024-09-13T14:54:25.121224+0000","mapping_epoch":238,"log_start":"173'2100","ondisk_log_start":"173'2100","created":160,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8408,"log_dups_size":0,"ondisk_log_size":8408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:10.140061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10508,"num_write_kb":336256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2948,"num_bytes_recovered":189136896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":109,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759554+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:54:32.759554+0000","last_peered":"2024-09-13T14:54:32.759554+0000","last_clean":"2024-09-13T14:54:32.759554+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:54:32.759554+0000","last_undegraded":"2024-09-13T14:54:32.759554+0000","last_fullsized":"2024-09-13T14:54:32.759554+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":361,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734044+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:54:24.734044+0000","last_peered":"2024-09-13T14:54:24.734044+0000","last_clean":"2024-09-13T14:54:24.734044+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:54:24.734044+0000","last_undegraded":"2024-09-13T14:54:24.734044+0000","last_fullsized":"2024-09-13T14:54:24.734044+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"255'10754","reported_seq":28024,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749980+0000","last_change":"2024-09-13T14:52:18.175858+0000","last_active":"2024-09-13T14:54:32.749980+0000","last_peered":"2024-09-13T14:54:32.749980+0000","last_clean":"2024-09-13T14:54:32.749980+0000","last_became_active":"2024-09-13T14:52:18.083568+0000","last_became_peered":"2024-09-13T14:52:18.083568+0000","last_unstale":"2024-09-13T14:54:32.749980+0000","last_undegraded":"2024-09-13T14:54:32.749980+0000","last_fullsized":"2024-09-13T14:54:32.749980+0000","mapping_epoch":232,"log_start":"184'3200","ondisk_log_start":"184'3200","created":160,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:08:55.321442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":344128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8562,"num_bytes_recovered":505872384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":369,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733900+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:54:24.733900+0000","last_peered":"2024-09-13T14:54:24.733900+0000","last_clean":"2024-09-13T14:54:24.733900+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:54:24.733900+0000","last_undegraded":"2024-09-13T14:54:24.733900+0000","last_fullsized":"2024-09-13T14:54:24.733900+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":240,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749679+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:54:32.749679+0000","last_peered":"2024-09-13T14:54:32.749679+0000","last_clean":"2024-09-13T14:54:32.749679+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:54:32.749679+0000","last_undegraded":"2024-09-13T14:54:32.749679+0000","last_fullsized":"2024-09-13T14:54:32.749679+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"255'10640","reported_seq":15662,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.936156+0000","last_change":"2024-09-13T14:53:24.611648+0000","last_active":"2024-09-13T14:54:33.936156+0000","last_peered":"2024-09-13T14:54:33.936156+0000","last_clean":"2024-09-13T14:54:33.936156+0000","last_became_active":"2024-09-13T14:53:24.585428+0000","last_became_peered":"2024-09-13T14:53:24.585428+0000","last_unstale":"2024-09-13T14:54:33.936156+0000","last_undegraded":"2024-09-13T14:54:33.936156+0000","last_fullsized":"2024-09-13T14:54:33.936156+0000","mapping_epoch":248,"log_start":"172'1800","ondisk_log_start":"172'1800","created":160,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:41.636547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10640,"num_write_kb":340480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2333,"num_bytes_recovered":147914752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":141,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759829+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:54:32.759829+0000","last_peered":"2024-09-13T14:54:32.759829+0000","last_clean":"2024-09-13T14:54:32.759829+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:54:32.759829+0000","last_undegraded":"2024-09-13T14:54:32.759829+0000","last_fullsized":"2024-09-13T14:54:32.759829+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":237,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749976+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:54:32.749976+0000","last_peered":"2024-09-13T14:54:32.749976+0000","last_clean":"2024-09-13T14:54:32.749976+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:54:32.749976+0000","last_undegraded":"2024-09-13T14:54:32.749976+0000","last_fullsized":"2024-09-13T14:54:32.749976+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"255'10822","reported_seq":19056,"reported_epoch":255,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-13T14:54:30.466059+0000","last_change":"2024-09-13T14:53:29.139613+0000","last_active":"2024-09-13T14:54:30.466059+0000","last_peered":"2024-09-13T14:54:30.466059+0000","last_clean":"2024-09-13T14:48:01.060341+0000","last_became_active":"2024-09-13T14:50:48.100444+0000","last_became_peered":"2024-09-13T14:50:48.100444+0000","last_unstale":"2024-09-13T14:54:30.466059+0000","last_undegraded":"2024-09-13T14:53:29.127369+0000","last_fullsized":"2024-09-13T14:50:37.946651+0000","mapping_epoch":215,"log_start":"190'4400","ondisk_log_start":"190'4400","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10822,"num_write_kb":346304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4101,"num_bytes_recovered":169672704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","5","7"],"object_location_counts":[{"shards":"3,5","objects":-579}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":233,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749705+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:54:32.749705+0000","last_peered":"2024-09-13T14:54:32.749705+0000","last_clean":"2024-09-13T14:54:32.749705+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:54:32.749705+0000","last_undegraded":"2024-09-13T14:54:32.749705+0000","last_fullsized":"2024-09-13T14:54:32.749705+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":267,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.750191+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:54:32.750191+0000","last_peered":"2024-09-13T14:54:32.750191+0000","last_clean":"2024-09-13T14:54:32.750191+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:54:32.750191+0000","last_undegraded":"2024-09-13T14:54:32.750191+0000","last_fullsized":"2024-09-13T14:54:32.750191+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"255'10506","reported_seq":14951,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759436+0000","last_change":"2024-09-13T14:48:52.028056+0000","last_active":"2024-09-13T14:54:32.759436+0000","last_peered":"2024-09-13T14:54:32.759436+0000","last_clean":"2024-09-13T14:54:32.759436+0000","last_became_active":"2024-09-13T14:48:52.025358+0000","last_became_peered":"2024-09-13T14:48:52.025358+0000","last_unstale":"2024-09-13T14:54:32.759436+0000","last_undegraded":"2024-09-13T14:54:32.759436+0000","last_fullsized":"2024-09-13T14:54:32.759436+0000","mapping_epoch":186,"log_start":"164'500","ondisk_log_start":"164'500","created":160,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:28.157976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10506,"num_write_kb":336129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2148,"num_bytes_recovered":140771328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":107,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734054+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:54:24.734054+0000","last_peered":"2024-09-13T14:54:24.734054+0000","last_clean":"2024-09-13T14:54:24.734054+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:54:24.734054+0000","last_undegraded":"2024-09-13T14:54:24.734054+0000","last_fullsized":"2024-09-13T14:54:24.734054+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":221,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.936346+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:54:33.936346+0000","last_peered":"2024-09-13T14:54:33.936346+0000","last_clean":"2024-09-13T14:54:33.936346+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:54:33.936346+0000","last_undegraded":"2024-09-13T14:54:33.936346+0000","last_fullsized":"2024-09-13T14:54:33.936346+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"255'10744","reported_seq":12011,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759392+0000","last_change":"2024-09-13T14:50:07.720888+0000","last_active":"2024-09-13T14:54:32.759392+0000","last_peered":"2024-09-13T14:54:32.759392+0000","last_clean":"2024-09-13T14:54:32.759392+0000","last_became_active":"2024-09-13T14:50:07.716242+0000","last_became_peered":"2024-09-13T14:50:07.716242+0000","last_unstale":"2024-09-13T14:54:32.759392+0000","last_undegraded":"2024-09-13T14:54:32.759392+0000","last_fullsized":"2024-09-13T14:54:32.759392+0000","mapping_epoch":204,"log_start":"164'665","ondisk_log_start":"164'665","created":160,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:04.347933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":343808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":138870784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":564,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.735076+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:54:24.735076+0000","last_peered":"2024-09-13T14:54:24.735076+0000","last_clean":"2024-09-13T14:54:24.735076+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:54:24.735076+0000","last_undegraded":"2024-09-13T14:54:24.735076+0000","last_fullsized":"2024-09-13T14:54:24.735076+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":824,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.758891+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:54:32.758891+0000","last_peered":"2024-09-13T14:54:32.758891+0000","last_clean":"2024-09-13T14:54:32.758891+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:54:32.758891+0000","last_undegraded":"2024-09-13T14:54:32.758891+0000","last_fullsized":"2024-09-13T14:54:32.758891+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":327,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733770+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:54:24.733770+0000","last_peered":"2024-09-13T14:54:24.733770+0000","last_clean":"2024-09-13T14:54:24.733770+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:54:24.733770+0000","last_undegraded":"2024-09-13T14:54:24.733770+0000","last_fullsized":"2024-09-13T14:54:24.733770+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":53,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.760075+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:54:32.760075+0000","last_peered":"2024-09-13T14:54:32.760075+0000","last_clean":"2024-09-13T14:54:32.760075+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:54:32.760075+0000","last_undegraded":"2024-09-13T14:54:32.760075+0000","last_fullsized":"2024-09-13T14:54:32.760075+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"255'10848","reported_seq":15607,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935299+0000","last_change":"2024-09-13T14:51:10.864099+0000","last_active":"2024-09-13T14:54:33.935299+0000","last_peered":"2024-09-13T14:54:33.935299+0000","last_clean":"2024-09-13T14:54:33.935299+0000","last_became_active":"2024-09-13T14:51:10.856250+0000","last_became_peered":"2024-09-13T14:51:10.856250+0000","last_unstale":"2024-09-13T14:54:33.935299+0000","last_undegraded":"2024-09-13T14:54:33.935299+0000","last_fullsized":"2024-09-13T14:54:33.935299+0000","mapping_epoch":223,"log_start":"189'4318","ondisk_log_start":"189'4318","created":160,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:37.336077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":347136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":139329536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":413,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.934927+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:54:33.934927+0000","last_peered":"2024-09-13T14:54:33.934927+0000","last_clean":"2024-09-13T14:54:33.934927+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:54:33.934927+0000","last_undegraded":"2024-09-13T14:54:33.934927+0000","last_fullsized":"2024-09-13T14:54:33.934927+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":375,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.739437+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:54:24.739437+0000","last_peered":"2024-09-13T14:54:24.739437+0000","last_clean":"2024-09-13T14:54:24.739437+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:54:24.739437+0000","last_undegraded":"2024-09-13T14:54:24.739437+0000","last_fullsized":"2024-09-13T14:54:24.739437+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":386,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733839+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:54:24.733839+0000","last_peered":"2024-09-13T14:54:24.733839+0000","last_clean":"2024-09-13T14:54:24.733839+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:54:24.733839+0000","last_undegraded":"2024-09-13T14:54:24.733839+0000","last_fullsized":"2024-09-13T14:54:24.733839+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"255'10596","reported_seq":20929,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759923+0000","last_change":"2024-09-13T14:52:32.381299+0000","last_active":"2024-09-13T14:54:32.759923+0000","last_peered":"2024-09-13T14:54:32.759923+0000","last_clean":"2024-09-13T14:54:32.759923+0000","last_became_active":"2024-09-13T14:52:32.362756+0000","last_became_peered":"2024-09-13T14:52:32.362756+0000","last_unstale":"2024-09-13T14:54:32.759923+0000","last_undegraded":"2024-09-13T14:54:32.759923+0000","last_fullsized":"2024-09-13T14:54:32.759923+0000","mapping_epoch":235,"log_start":"178'2400","ondisk_log_start":"178'2400","created":160,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8196,"log_dups_size":0,"ondisk_log_size":8196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:28.584907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10596,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5065,"num_bytes_recovered":321519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":916,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.936162+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:54:33.936162+0000","last_peered":"2024-09-13T14:54:33.936162+0000","last_clean":"2024-09-13T14:54:33.936162+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:54:33.936162+0000","last_undegraded":"2024-09-13T14:54:33.936162+0000","last_fullsized":"2024-09-13T14:54:33.936162+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":370,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935133+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:54:33.935133+0000","last_peered":"2024-09-13T14:54:33.935133+0000","last_clean":"2024-09-13T14:54:33.935133+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:54:33.935133+0000","last_undegraded":"2024-09-13T14:54:33.935133+0000","last_fullsized":"2024-09-13T14:54:33.935133+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":124,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935168+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:54:33.935168+0000","last_peered":"2024-09-13T14:54:33.935168+0000","last_clean":"2024-09-13T14:54:33.935168+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:54:33.935168+0000","last_undegraded":"2024-09-13T14:54:33.935168+0000","last_fullsized":"2024-09-13T14:54:33.935168+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"255'10720","reported_seq":20067,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:25.130630+0000","last_change":"2024-09-13T14:52:52.443442+0000","last_active":"2024-09-13T14:54:25.130630+0000","last_peered":"2024-09-13T14:54:25.130630+0000","last_clean":"2024-09-13T14:54:25.130630+0000","last_became_active":"2024-09-13T14:52:52.421083+0000","last_became_peered":"2024-09-13T14:52:52.421083+0000","last_unstale":"2024-09-13T14:54:25.130630+0000","last_undegraded":"2024-09-13T14:54:25.130630+0000","last_fullsized":"2024-09-13T14:54:25.130630+0000","mapping_epoch":240,"log_start":"189'4300","ondisk_log_start":"189'4300","created":160,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:14.887813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10720,"num_write_kb":343040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4569,"num_bytes_recovered":287244288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":111,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759045+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:54:32.759045+0000","last_peered":"2024-09-13T14:54:32.759045+0000","last_clean":"2024-09-13T14:54:32.759045+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:54:32.759045+0000","last_undegraded":"2024-09-13T14:54:32.759045+0000","last_fullsized":"2024-09-13T14:54:32.759045+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":242,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.750047+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:54:32.750047+0000","last_peered":"2024-09-13T14:54:32.750047+0000","last_clean":"2024-09-13T14:54:32.750047+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:54:32.750047+0000","last_undegraded":"2024-09-13T14:54:32.750047+0000","last_fullsized":"2024-09-13T14:54:32.750047+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759507+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:54:32.759507+0000","last_peered":"2024-09-13T14:54:32.759507+0000","last_clean":"2024-09-13T14:54:32.759507+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:54:32.759507+0000","last_undegraded":"2024-09-13T14:54:32.759507+0000","last_fullsized":"2024-09-13T14:54:32.759507+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"255'10814","reported_seq":15432,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:25.115297+0000","last_change":"2024-09-13T14:51:28.815689+0000","last_active":"2024-09-13T14:54:25.115297+0000","last_peered":"2024-09-13T14:54:25.115297+0000","last_clean":"2024-09-13T14:54:25.115297+0000","last_became_active":"2024-09-13T14:51:28.804016+0000","last_became_peered":"2024-09-13T14:51:28.804016+0000","last_unstale":"2024-09-13T14:54:25.115297+0000","last_undegraded":"2024-09-13T14:54:25.115297+0000","last_fullsized":"2024-09-13T14:54:25.115297+0000","mapping_epoch":226,"log_start":"178'2400","ondisk_log_start":"178'2400","created":160,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8414,"log_dups_size":0,"ondisk_log_size":8414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:52.417787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10814,"num_write_kb":346048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2199,"num_bytes_recovered":138674176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":111,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759297+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:54:32.759297+0000","last_peered":"2024-09-13T14:54:32.759297+0000","last_clean":"2024-09-13T14:54:32.759297+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:54:32.759297+0000","last_undegraded":"2024-09-13T14:54:32.759297+0000","last_fullsized":"2024-09-13T14:54:32.759297+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":925,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935512+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:54:33.935512+0000","last_peered":"2024-09-13T14:54:33.935512+0000","last_clean":"2024-09-13T14:54:33.935512+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:54:33.935512+0000","last_undegraded":"2024-09-13T14:54:33.935512+0000","last_fullsized":"2024-09-13T14:54:33.935512+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733234+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:54:24.733234+0000","last_peered":"2024-09-13T14:54:24.733234+0000","last_clean":"2024-09-13T14:54:24.733234+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:54:24.733234+0000","last_undegraded":"2024-09-13T14:54:24.733234+0000","last_fullsized":"2024-09-13T14:54:24.733234+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"255'10908","reported_seq":15658,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.934832+0000","last_change":"2024-09-13T14:50:48.998377+0000","last_active":"2024-09-13T14:54:33.934832+0000","last_peered":"2024-09-13T14:54:33.934832+0000","last_clean":"2024-09-13T14:54:33.934832+0000","last_became_active":"2024-09-13T14:50:48.989251+0000","last_became_peered":"2024-09-13T14:50:48.989251+0000","last_unstale":"2024-09-13T14:54:33.934832+0000","last_undegraded":"2024-09-13T14:54:33.934832+0000","last_fullsized":"2024-09-13T14:54:33.934832+0000","mapping_epoch":218,"log_start":"189'4345","ondisk_log_start":"189'4345","created":160,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:17.368649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10908,"num_write_kb":349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2263,"num_bytes_recovered":144310272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":377,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935405+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:54:33.935405+0000","last_peered":"2024-09-13T14:54:33.935405+0000","last_clean":"2024-09-13T14:54:33.935405+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:54:33.935405+0000","last_undegraded":"2024-09-13T14:54:33.935405+0000","last_fullsized":"2024-09-13T14:54:33.935405+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":595,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734483+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:54:24.734483+0000","last_peered":"2024-09-13T14:54:24.734483+0000","last_clean":"2024-09-13T14:54:24.734483+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:54:24.734483+0000","last_undegraded":"2024-09-13T14:54:24.734483+0000","last_fullsized":"2024-09-13T14:54:24.734483+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":371,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733957+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:54:24.733957+0000","last_peered":"2024-09-13T14:54:24.733957+0000","last_clean":"2024-09-13T14:54:24.733957+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:54:24.733957+0000","last_undegraded":"2024-09-13T14:54:24.733957+0000","last_fullsized":"2024-09-13T14:54:24.733957+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"255'10510","reported_seq":17080,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749680+0000","last_change":"2024-09-13T14:53:05.481759+0000","last_active":"2024-09-13T14:54:32.749680+0000","last_peered":"2024-09-13T14:54:32.749680+0000","last_clean":"2024-09-13T14:54:32.749680+0000","last_became_active":"2024-09-13T14:53:05.465653+0000","last_became_peered":"2024-09-13T14:53:05.465653+0000","last_unstale":"2024-09-13T14:54:32.749680+0000","last_undegraded":"2024-09-13T14:54:32.749680+0000","last_fullsized":"2024-09-13T14:54:32.749680+0000","mapping_epoch":243,"log_start":"181'3000","ondisk_log_start":"181'3000","created":160,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":7510,"log_dups_size":0,"ondisk_log_size":7510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:16:28.220387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10510,"num_write_kb":336320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3214,"num_bytes_recovered":205586432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":391,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734210+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:54:24.734210+0000","last_peered":"2024-09-13T14:54:24.734210+0000","last_clean":"2024-09-13T14:54:24.734210+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:54:24.734210+0000","last_undegraded":"2024-09-13T14:54:24.734210+0000","last_fullsized":"2024-09-13T14:54:24.734210+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":827,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935554+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:54:33.935554+0000","last_peered":"2024-09-13T14:54:33.935554+0000","last_clean":"2024-09-13T14:54:33.935554+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:54:33.935554+0000","last_undegraded":"2024-09-13T14:54:33.935554+0000","last_fullsized":"2024-09-13T14:54:33.935554+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":108,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734000+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:54:24.734000+0000","last_peered":"2024-09-13T14:54:24.734000+0000","last_clean":"2024-09-13T14:54:24.734000+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:54:24.734000+0000","last_undegraded":"2024-09-13T14:54:24.734000+0000","last_fullsized":"2024-09-13T14:54:24.734000+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"255'10860","reported_seq":12452,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759348+0000","last_change":"2024-09-13T14:48:56.056856+0000","last_active":"2024-09-13T14:54:32.759348+0000","last_peered":"2024-09-13T14:54:32.759348+0000","last_clean":"2024-09-13T14:54:32.759348+0000","last_became_active":"2024-09-13T14:48:56.052207+0000","last_became_peered":"2024-09-13T14:48:56.052207+0000","last_unstale":"2024-09-13T14:54:32.759348+0000","last_undegraded":"2024-09-13T14:54:32.759348+0000","last_fullsized":"2024-09-13T14:54:32.759348+0000","mapping_epoch":188,"log_start":"180'2700","ondisk_log_start":"180'2700","created":160,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:47:38.199052+0000","last_clean_scrub_stamp":"2024-09-13T14:47:38.199052+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:22.539829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10860,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":47906816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":366,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733193+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:54:24.733193+0000","last_peered":"2024-09-13T14:54:24.733193+0000","last_clean":"2024-09-13T14:54:24.733193+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:54:24.733193+0000","last_undegraded":"2024-09-13T14:54:24.733193+0000","last_fullsized":"2024-09-13T14:54:24.733193+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":809,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759921+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:54:32.759921+0000","last_peered":"2024-09-13T14:54:32.759921+0000","last_clean":"2024-09-13T14:54:32.759921+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:54:32.759921+0000","last_undegraded":"2024-09-13T14:54:32.759921+0000","last_fullsized":"2024-09-13T14:54:32.759921+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":244,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749593+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:54:32.749593+0000","last_peered":"2024-09-13T14:54:32.749593+0000","last_clean":"2024-09-13T14:54:32.749593+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:54:32.749593+0000","last_undegraded":"2024-09-13T14:54:32.749593+0000","last_fullsized":"2024-09-13T14:54:32.749593+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":240,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749948+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:54:32.749948+0000","last_peered":"2024-09-13T14:54:32.749948+0000","last_clean":"2024-09-13T14:54:32.749948+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:54:32.749948+0000","last_undegraded":"2024-09-13T14:54:32.749948+0000","last_fullsized":"2024-09-13T14:54:32.749948+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":288,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.750106+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:54:32.750106+0000","last_peered":"2024-09-13T14:54:32.750106+0000","last_clean":"2024-09-13T14:54:32.750106+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:54:32.750106+0000","last_undegraded":"2024-09-13T14:54:32.750106+0000","last_fullsized":"2024-09-13T14:54:32.750106+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":374,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935195+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:54:33.935195+0000","last_peered":"2024-09-13T14:54:33.935195+0000","last_clean":"2024-09-13T14:54:33.935195+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:54:33.935195+0000","last_undegraded":"2024-09-13T14:54:33.935195+0000","last_fullsized":"2024-09-13T14:54:33.935195+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":211,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734029+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:54:24.734029+0000","last_peered":"2024-09-13T14:54:24.734029+0000","last_clean":"2024-09-13T14:54:24.734029+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:54:24.734029+0000","last_undegraded":"2024-09-13T14:54:24.734029+0000","last_fullsized":"2024-09-13T14:54:24.734029+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":120,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.758992+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:54:32.758992+0000","last_peered":"2024-09-13T14:54:32.758992+0000","last_clean":"2024-09-13T14:54:32.758992+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:54:32.758992+0000","last_undegraded":"2024-09-13T14:54:32.758992+0000","last_fullsized":"2024-09-13T14:54:32.758992+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":348,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733449+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:54:24.733449+0000","last_peered":"2024-09-13T14:54:24.733449+0000","last_clean":"2024-09-13T14:54:24.733449+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:54:24.733449+0000","last_undegraded":"2024-09-13T14:54:24.733449+0000","last_fullsized":"2024-09-13T14:54:24.733449+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":403,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734105+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:54:24.734105+0000","last_peered":"2024-09-13T14:54:24.734105+0000","last_clean":"2024-09-13T14:54:24.734105+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:54:24.734105+0000","last_undegraded":"2024-09-13T14:54:24.734105+0000","last_fullsized":"2024-09-13T14:54:24.734105+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":399,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734269+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:54:24.734269+0000","last_peered":"2024-09-13T14:54:24.734269+0000","last_clean":"2024-09-13T14:54:24.734269+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:54:24.734269+0000","last_undegraded":"2024-09-13T14:54:24.734269+0000","last_fullsized":"2024-09-13T14:54:24.734269+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":150,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759924+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:54:32.759924+0000","last_peered":"2024-09-13T14:54:32.759924+0000","last_clean":"2024-09-13T14:54:32.759924+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:54:32.759924+0000","last_undegraded":"2024-09-13T14:54:32.759924+0000","last_fullsized":"2024-09-13T14:54:32.759924+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":366,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.734418+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:54:24.734418+0000","last_peered":"2024-09-13T14:54:24.734418+0000","last_clean":"2024-09-13T14:54:24.734418+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:54:24.734418+0000","last_undegraded":"2024-09-13T14:54:24.734418+0000","last_fullsized":"2024-09-13T14:54:24.734418+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":383,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733747+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:54:24.733747+0000","last_peered":"2024-09-13T14:54:24.733747+0000","last_clean":"2024-09-13T14:54:24.733747+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:54:24.733747+0000","last_undegraded":"2024-09-13T14:54:24.733747+0000","last_fullsized":"2024-09-13T14:54:24.733747+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":355,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.934922+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:54:33.934922+0000","last_peered":"2024-09-13T14:54:33.934922+0000","last_clean":"2024-09-13T14:54:33.934922+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:54:33.934922+0000","last_undegraded":"2024-09-13T14:54:33.934922+0000","last_fullsized":"2024-09-13T14:54:33.934922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":215,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733894+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:54:24.733894+0000","last_peered":"2024-09-13T14:54:24.733894+0000","last_clean":"2024-09-13T14:54:24.733894+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:54:24.733894+0000","last_undegraded":"2024-09-13T14:54:24.733894+0000","last_fullsized":"2024-09-13T14:54:24.733894+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":238,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749809+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:54:32.749809+0000","last_peered":"2024-09-13T14:54:32.749809+0000","last_clean":"2024-09-13T14:54:32.749809+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:54:32.749809+0000","last_undegraded":"2024-09-13T14:54:32.749809+0000","last_fullsized":"2024-09-13T14:54:32.749809+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":114,"reported_epoch":257,"state":"active+clean","last_fresh":"2024-09-13T14:54:33.935881+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:54:33.935881+0000","last_peered":"2024-09-13T14:54:33.935881+0000","last_clean":"2024-09-13T14:54:33.935881+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:54:33.935881+0000","last_undegraded":"2024-09-13T14:54:33.935881+0000","last_fullsized":"2024-09-13T14:54:33.935881+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":375,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733258+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:54:24.733258+0000","last_peered":"2024-09-13T14:54:24.733258+0000","last_clean":"2024-09-13T14:54:24.733258+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:54:24.733258+0000","last_undegraded":"2024-09-13T14:54:24.733258+0000","last_fullsized":"2024-09-13T14:54:24.733258+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":248,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749838+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:54:32.749838+0000","last_peered":"2024-09-13T14:54:32.749838+0000","last_clean":"2024-09-13T14:54:32.749838+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:54:32.749838+0000","last_undegraded":"2024-09-13T14:54:32.749838+0000","last_fullsized":"2024-09-13T14:54:32.749838+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":95,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759388+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:54:32.759388+0000","last_peered":"2024-09-13T14:54:32.759388+0000","last_clean":"2024-09-13T14:54:32.759388+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:54:32.759388+0000","last_undegraded":"2024-09-13T14:54:32.759388+0000","last_fullsized":"2024-09-13T14:54:32.759388+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":236,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749788+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:54:32.749788+0000","last_peered":"2024-09-13T14:54:32.749788+0000","last_clean":"2024-09-13T14:54:32.749788+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:54:32.749788+0000","last_undegraded":"2024-09-13T14:54:32.749788+0000","last_fullsized":"2024-09-13T14:54:32.749788+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":97,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.759469+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:54:32.759469+0000","last_peered":"2024-09-13T14:54:32.759469+0000","last_clean":"2024-09-13T14:54:32.759469+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:54:32.759469+0000","last_undegraded":"2024-09-13T14:54:32.759469+0000","last_fullsized":"2024-09-13T14:54:32.759469+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":369,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733868+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:54:24.733868+0000","last_peered":"2024-09-13T14:54:24.733868+0000","last_clean":"2024-09-13T14:54:24.733868+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:54:24.733868+0000","last_undegraded":"2024-09-13T14:54:24.733868+0000","last_fullsized":"2024-09-13T14:54:24.733868+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":386,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.738885+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:54:24.738885+0000","last_peered":"2024-09-13T14:54:24.738885+0000","last_clean":"2024-09-13T14:54:24.738885+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:54:24.738885+0000","last_undegraded":"2024-09-13T14:54:24.738885+0000","last_fullsized":"2024-09-13T14:54:24.738885+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":266,"reported_epoch":256,"state":"active+clean","last_fresh":"2024-09-13T14:54:32.749572+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:54:32.749572+0000","last_peered":"2024-09-13T14:54:32.749572+0000","last_clean":"2024-09-13T14:54:32.749572+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:54:32.749572+0000","last_undegraded":"2024-09-13T14:54:32.749572+0000","last_fullsized":"2024-09-13T14:54:32.749572+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":438,"reported_epoch":255,"state":"active+clean","last_fresh":"2024-09-13T14:54:24.733966+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:54:24.733966+0000","last_peered":"2024-09-13T14:54:24.733966+0000","last_clean":"2024-09-13T14:54:24.733966+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:54:24.733966+0000","last_undegraded":"2024-09-13T14:54:24.733966+0000","last_fullsized":"2024-09-13T14:54:24.733966+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171548,"num_write_kb":5489473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46372,"num_bytes_recovered":2821390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":287375360,"data_stored":287375360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128520,"ondisk_log_size":128520,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234326,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495962,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367920,"kb_used_data":3748,"kb_used_omap":1,"kb_used_meta":364158,"kb_avail":93377232,"statfs":{"total":95995035648,"available":95618285568,"internally_reserved":0,"allocated":3837952,"data_stored":2461956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":372898247},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757597,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218868,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":93526284,"statfs":{"total":95995035648,"available":95770914816,"internally_reserved":0,"allocated":3248128,"data_stored":1876034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":220854727},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019237,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314672,"kb_used_data":4644,"kb_used_omap":1,"kb_used_meta":310014,"kb_avail":93430480,"statfs":{"total":95995035648,"available":95672811520,"internally_reserved":0,"allocated":4755456,"data_stored":3379521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":317454790},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468121,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313752,"kb_used_data":7244,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93431400,"statfs":{"total":95995035648,"available":95673753600,"internally_reserved":0,"allocated":7417856,"data_stored":6053445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":313846948},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631651,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408256,"kb_used_data":7284,"kb_used_omap":8,"kb_used_meta":400951,"kb_avail":93336896,"statfs":{"total":95995035648,"available":95576981504,"internally_reserved":0,"allocated":7458816,"data_stored":6075346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":410574815},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795174,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542072,"kb_used_data":235496,"kb_used_omap":7,"kb_used_meta":306552,"kb_avail":93203080,"statfs":{"total":95995035648,"available":95439953920,"internally_reserved":0,"allocated":241147904,"data_stored":239771913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":313909874},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958704,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292188,"kb_used_data":7828,"kb_used_omap":7,"kb_used_meta":284344,"kb_avail":93452964,"statfs":{"total":95995035648,"available":95695835136,"internally_reserved":0,"allocated":8015872,"data_stored":6644347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":291168743},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237305856,"data_stored":237305856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":50069504,"data_stored":50069504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:54:37.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph status --format=json 2024-09-13T14:54:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:37.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:36 smithi172 ceph-mon[100696]: pgmap v782: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 67 op/s; 579/84 objects degraded (689.286%); 0 B/s, 168 objects/s recovering 2024-09-13T14:54:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1655293045' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T14:54:37.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.287 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:37.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:37.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:37.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:37.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:37.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:37.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:37.659 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:37.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:37.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:37.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:37.911 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:37.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:37.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:37 smithi154 ceph-mon[102314]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:37.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:37 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1655293045' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:54:37.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:37 smithi154 ceph-mon[102314]: osdmap e258: 8 total, 8 up, 7 in 2024-09-13T14:54:37.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:37 smithi154 ceph-mon[107038]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:37.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:37 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1655293045' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:54:37.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:37 smithi154 ceph-mon[107038]: osdmap e258: 8 total, 8 up, 7 in 2024-09-13T14:54:38.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:38.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:38.116 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:38.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:38.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:37 smithi172 ceph-mon[100696]: from='client.45320 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:37 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1655293045' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T14:54:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:37 smithi172 ceph-mon[100696]: osdmap e258: 8 total, 8 up, 7 in 2024-09-13T14:54:38.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:38.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:38.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:38.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:38.482 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T14:54:38.483 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T14:54:38.483 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-13T14:54:38.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-13T14:54:38.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:38.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:38.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:38.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:38.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:38.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:38.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:38.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:38.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:38.931 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:39.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:39.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:39.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:39.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:38 smithi154 ceph-mon[107038]: pgmap v784: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:39.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:38 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 579/84 objects degraded (689.286%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:54:39.131 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:38 smithi154 ceph-mon[102314]: pgmap v784: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:39.131 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:38 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 579/84 objects degraded (689.286%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:54:39.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:38 smithi172 ceph-mon[100696]: pgmap v784: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:38 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 579/84 objects degraded (689.286%), 1 pg degraded, 1 pg undersized) 2024-09-13T14:54:39.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:39.269 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:39.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:39.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:39.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:39.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:39.588 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:54:39.588 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:54:39.596 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:39.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:39.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:39.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:39.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:39.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:39.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:39.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:39.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:40.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:40.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:40.144 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:40.246 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:40.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:40.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:40.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:40.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:40.519 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:40.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:40.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:40.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:40.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:40.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:40.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:40.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:40.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:41.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:41.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:41.130 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:40 smithi154 ceph-mon[102314]: pgmap v785: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:41.130 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:40 smithi154 ceph-mon[107038]: pgmap v785: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:41.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:40 smithi172 ceph-mon[100696]: pgmap v785: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:41.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:41.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:41.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:41.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:41.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:41.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:41.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:41.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:41.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:41.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:41.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:41.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:41.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:41.962 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:54:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:54:41] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-13T14:54:41.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:42.004 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:42.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:42.127 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:42.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:42.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:42.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:42.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:42.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:42.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:42.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:42.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:42.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:42.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:42.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:54:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:42.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:42.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:42.873 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:42.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:42.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:42 smithi154 ceph-mon[102314]: pgmap v786: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:42.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:42 smithi154 ceph-mon[107038]: pgmap v786: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:42.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:42.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:43.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:43.124 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:43.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:43.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:42 smithi172 ceph-mon[100696]: pgmap v786: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:43.248 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:43.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:43.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:43.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.497 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:43.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:43.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:43.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:43.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:43.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:43.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:43.870 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:43.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:43.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:43.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:43.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:44.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:44.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:44.154 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:44.154 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:44.154 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:43 smithi154 ceph-mon[102314]: pgmap v787: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:44.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:44.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:44.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:43 smithi154 ceph-mon[107038]: pgmap v787: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:44.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:43 smithi172 ceph-mon[100696]: pgmap v787: 73 pgs: 73 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:44.246 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:44.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:44.359 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:44.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:44.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:44.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:44.500 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:54:44.500 INFO:teuthology.orchestra.run.smithi154.stdout:{"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1100,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":258,"num_osds":8,"num_up_osds":8,"osd_up_since":1726238500,"num_in_osds":7,"osd_in_since":1726238900,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":73}],"num_pgs":73,"num_pools":4,"num_objects":28,"data_bytes":4790245,"bytes_used":2526060544,"bytes_avail":669439188992,"bytes_total":671965249536},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi154.daekzd","status":"up:active","gid":24539}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.154:8443/","prometheus":"http://172.21.15.154:9283/"}},"servicemap":{"epoch":25,"modified":"2024-09-13T14:50:26.951855+0000","services":{}},"progress_events":{}} 2024-09-13T14:54:44.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:44.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:44.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:44.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:44.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:44.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:44.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:44.986 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:54:44.997 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:44.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:45.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1465416362' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/110427330' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:54:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1465416362' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/110427330' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:54:45.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1465416362' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:54:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/110427330' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-13T14:54:45.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:45.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.392 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-13T14:54:45.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:54:45.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.504 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:45.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:45.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.755 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:45.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.756 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:45.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:45.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:45.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:45.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:46.008 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:46.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:46.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1465416362' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[102314]: osdmap e259: 8 total, 8 up, 7 in 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[102314]: pgmap v789: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1465416362' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[107038]: osdmap e259: 8 total, 8 up, 7 in 2024-09-13T14:54:46.205 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:45 smithi154 ceph-mon[107038]: pgmap v789: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:46.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:45 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:45 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1465416362' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:54:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:45 smithi172 ceph-mon[100696]: osdmap e259: 8 total, 8 up, 7 in 2024-09-13T14:54:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:45 smithi172 ceph-mon[100696]: pgmap v789: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:46.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:46.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:46.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:46.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:46.508 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:46.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:46.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:46.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:46.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:46.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:46.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:46.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:46.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:46.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:47.011 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:47.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:47.139 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:47 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/346971379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:47.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:47 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/346971379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:47.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:47.265 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:47.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:47.391 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:47 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/346971379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:54:47.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:47.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:47.619 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:47.642 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:47.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:47.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:47.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:47.892 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:47.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:47.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:48.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:48.075 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:54:48.075 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:54:48.075 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:54:48.075 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:54:48.075 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:54:48.076 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:54:48.077 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:54:48.077 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:54:48.077 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:54:48.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:48.144 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:48.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:48.246 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:48.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:48.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:48.378 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:48 smithi154 ceph-mon[102314]: pgmap v790: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:48.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:48 smithi154 ceph-mon[107038]: pgmap v790: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:48.400 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:48 smithi172 ceph-mon[100696]: pgmap v790: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:48.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:48.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:48.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:48.657 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:48.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:48.780 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:48.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:48.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:48.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:49.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:49.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:49.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:49.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:49.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:49.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:49.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:49.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:49.408 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:49.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:54:49.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:49.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:49.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:49.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:49.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:49.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:49.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:49.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:49.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:50.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:50.033 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:50.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:50.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:50.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:50.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:50.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:50 smithi154 ceph-mon[102314]: pgmap v791: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:50.393 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:50 smithi154 ceph-mon[107038]: pgmap v791: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:50.408 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:50 smithi172 ceph-mon[100696]: pgmap v791: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:50.493 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:54:50.493 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:54:50.493 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:54:50.493 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:54:50.493 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:54:50.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:50.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:50.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:50.663 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:50.686 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:50.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:54:50.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:50.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:50.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:50.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:50.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:51.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:51.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:51.099 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_2' created 2024-09-13T14:54:51.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:51.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:51.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:54:51.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:51.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3193547995' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:54:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3193547995' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:54:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3193547995' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-13T14:54:51.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.539 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:51.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:51.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:51.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:51.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:51.916 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:51.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:51.981 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:51.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-13T14:54:52.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:52.041 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:52.138 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:54:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:54:51] "GET /metrics HTTP/1.1" 200 34156 "" "Prometheus/2.51.0" 2024-09-13T14:54:52.138 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:52 smithi154 ceph-mon[102314]: pgmap v792: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:52.138 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:52 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3193547995' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:54:52.138 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:52 smithi154 ceph-mon[102314]: osdmap e260: 8 total, 8 up, 7 in 2024-09-13T14:54:52.138 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:52 smithi154 ceph-mon[107038]: pgmap v792: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:52.139 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:52 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3193547995' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:54:52.139 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:52 smithi154 ceph-mon[107038]: osdmap e260: 8 total, 8 up, 7 in 2024-09-13T14:54:52.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:52.164 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:52.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:52.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:52.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:54:52.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:54:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:52 smithi172 ceph-mon[100696]: pgmap v792: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:52 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3193547995' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-13T14:54:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:52 smithi172 ceph-mon[100696]: osdmap e260: 8 total, 8 up, 7 in 2024-09-13T14:54:52.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:52.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:52.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:52.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:52.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:52.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:52.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:52.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:52.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:53.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:53.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:53.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:53.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:53.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:53.280 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:53.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:53.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:53.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:53 smithi154 ceph-mon[102314]: osdmap e261: 8 total, 8 up, 7 in 2024-09-13T14:54:53.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:53 smithi154 ceph-mon[107038]: osdmap e261: 8 total, 8 up, 7 in 2024-09-13T14:54:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:53 smithi172 ceph-mon[100696]: osdmap e261: 8 total, 8 up, 7 in 2024-09-13T14:54:53.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:53.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:53.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:53.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:53.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:53.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:53.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:53.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:53.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:53.949 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:54.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:54.027 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:54.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:54.140 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:54 smithi154 ceph-mon[102314]: pgmap v795: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:54.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:54 smithi154 ceph-mon[102314]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:54:54.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:54 smithi154 ceph-mon[102314]: osdmap e262: 8 total, 8 up, 7 in 2024-09-13T14:54:54.141 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:54 smithi154 ceph-mon[107038]: pgmap v795: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:54.141 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:54 smithi154 ceph-mon[107038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:54:54.141 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:54 smithi154 ceph-mon[107038]: osdmap e262: 8 total, 8 up, 7 in 2024-09-13T14:54:54.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:54.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:54.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:54.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:54.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:54 smithi172 ceph-mon[100696]: pgmap v795: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:54 smithi172 ceph-mon[100696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T14:54:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:54 smithi172 ceph-mon[100696]: osdmap e262: 8 total, 8 up, 7 in 2024-09-13T14:54:54.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:54.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:54.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:54.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:54.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:54.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:54.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:54.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:54.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:54.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:54.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:55.024 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:55.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:55.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:55.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:55.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:55.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:55.288 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:55.288 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:54:55.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:55.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:55.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:55.526 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:55.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:55.653 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:55.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:55.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:55.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:55.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:55.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:56.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:56.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:56.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:56.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:56.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:56.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:56.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:56.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:56.413 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:56 smithi154 ceph-mon[102314]: pgmap v797: 89 pgs: 3 unknown, 86 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:56.414 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:56 smithi154 ceph-mon[107038]: pgmap v797: 89 pgs: 3 unknown, 86 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:56 smithi172 ceph-mon[100696]: pgmap v797: 89 pgs: 3 unknown, 86 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:56.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:56.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:56.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:56.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:56.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:56.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:56.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:56.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:56.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:57.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:57.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:57.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:57.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:57.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:57.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:57.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:57.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:57.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:57.481 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:54:57.482 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:180635 5.3 5:ce8d9a3e:::benchmark_data_smithi154_144140_object70282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:17.034898+0000", 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 520.16834300599999, 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0093376780000001, 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.522 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 180635 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.034898+0000", 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.034898+0000", 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.034899+0000", 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4649999999999999e-06 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.034904+0000", 2024-09-13T14:54:57.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9799999999999998e-06 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.034920+0000", 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6235e-05 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.034928+0000", 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6879999999999999e-06 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.039714+0000", 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047854239999999999 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.039722+0000", 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4489999999999999e-06 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.039260+0000", 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99953778500000001 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.039394+0000", 2024-09-13T14:54:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013424699999999999 2024-09-13T14:54:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.039519+0000", 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012514500000000001 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.043960+0000", 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044406840000000003 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.043971+0000", 2024-09-13T14:54:57.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1558e-05 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.044086+0000", 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114947 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.044193+0000", 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010673100000000001 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.044235+0000", 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2339999999999998e-05 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:180669 5.3 5:c96c74a0:::benchmark_data_smithi154_144140_object70316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:17.071003+0000", 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 520.13223761899997, 2024-09-13T14:54:57.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98747756900000005, 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 180669 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071003+0000", 2024-09-13T14:54:57.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071003+0000", 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071004+0000", 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.381e-06 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071010+0000", 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4539999999999998e-06 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071020+0000", 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0122999999999999e-05 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071027+0000", 2024-09-13T14:54:57.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8460000000000002e-06 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071110+0000", 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2868999999999996e-05 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.071119+0000", 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3740000000000004e-06 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.044283+0000", 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97316416800000005 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.044419+0000", 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013531 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.044542+0000", 2024-09-13T14:54:57.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012370600000000001 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.052100+0000", 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075572429999999999 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.052141+0000", 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1100999999999998e-05 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.058374+0000", 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062338189999999998 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.058427+0000", 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2648999999999997e-05 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.058481+0000", 2024-09-13T14:54:57.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3526000000000002e-05 2024-09-13T14:54:57.538 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.538 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.538 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:180675 5.3 5:c90075ca:::benchmark_data_smithi154_144140_object70322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:17.076650+0000", 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 520.12659057300004, 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98192327899999998, 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 180675 2024-09-13T14:54:57.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076650+0000", 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076650+0000", 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076652+0000", 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.643e-06 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076656+0000", 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5310000000000003e-06 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076670+0000", 2024-09-13T14:54:57.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3788000000000001e-05 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076677+0000", 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9840000000000004e-06 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076742+0000", 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4620000000000001e-05 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.076753+0000", 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1721e-05 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.048275+0000", 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97152145000000001 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.048372+0000", 2024-09-13T14:54:57.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7362999999999996e-05 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.048483+0000", 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110543 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.052173+0000", 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036898040000000001 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.052209+0000", 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6128999999999999e-05 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.058494+0000", 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062849869999999997 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.058537+0000", 2024-09-13T14:54:57.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3084000000000003e-05 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.058573+0000", 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6631999999999998e-05 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:180713 5.3 5:c58e535a:::benchmark_data_smithi154_144140_object70360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:17.127871+0000", 2024-09-13T14:54:57.546 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 520.07536966700002, 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.93389260500000004, 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 180713 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.127871+0000", 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.127871+0000", 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.127872+0000", 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1850000000000001e-06 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.127877+0000", 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1850000000000003e-06 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.127892+0000", 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4876e-05 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.127901+0000", 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0399999999999998e-06 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.127994+0000", 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3015000000000003e-05 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.128005+0000", 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0334e-05 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.052232+0000", 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.92422735899999997 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.052313+0000", 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0656999999999997e-05 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.052392+0000", 2024-09-13T14:54:57.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8936000000000005e-05 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.055324+0000", 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029325649999999998 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061636+0000", 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063114190000000004 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061650+0000", 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4722999999999999e-05 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061720+0000", 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9298e-05 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061764+0000", 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4013e-05 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.34620.0:180725 5.3 5:c92f52d6:::benchmark_data_smithi154_144140_object70372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:46:17.140772+0000", 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 520.06246826100005, 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.92116983299999999, 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.554 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.34620", 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/511591426", 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 180725 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140772+0000", 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140772+0000", 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140774+0000", 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.671e-06 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140780+0000", 2024-09-13T14:54:57.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0379999999999996e-06 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140797+0000", 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7022000000000001e-05 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140805+0000", 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1829999999999994e-06 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140905+0000", 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.98e-05 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:17.140915+0000", 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4979999999999995e-06 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.055373+0000", 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.91445820899999997 2024-09-13T14:54:57.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.055490+0000", 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011744299999999999 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.055617+0000", 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012661500000000001 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061771+0000", 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061541190000000004 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061811+0000", 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0105999999999999e-05 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061882+0000", 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1184000000000002e-05 2024-09-13T14:54:57.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061919+0000", 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6779000000000001e-05 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:46:18.061942+0000", 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3166000000000001e-05 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:99412 6.0 6:0e5ca8c0:::benchmark_data_smithi154_175640_object13636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:27.920512+0000", 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 209.28272862899999, 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89906116800000002, 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 99412 2024-09-13T14:54:57.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920512+0000", 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920512+0000", 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920514+0000", 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.542e-06 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920521+0000", 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4250000000000001e-06 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920538+0000", 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6626999999999999e-05 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920545+0000", 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0849999999999999e-06 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920601+0000", 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5792999999999999e-05 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920608+0000", 2024-09-13T14:54:57.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7130000000000005e-06 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.811980+0000", 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89137181600000004 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.812104+0000", 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000123647 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.812208+0000", 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010425 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.815905+0000", 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036968489999999999 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.815918+0000", 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2677000000000001e-05 2024-09-13T14:54:57.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819459+0000", 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035411840000000002 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819524+0000", 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5659000000000003e-05 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819573+0000", 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8900999999999998e-05 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:99418 6.0 6:0fb8d46d:::benchmark_data_smithi154_175640_object13642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:27.948072+0000", 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 209.255168677, 2024-09-13T14:54:57.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.88284572800000005, 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 99418 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948072+0000", 2024-09-13T14:54:57.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948072+0000", 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948075+0000", 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6749999999999998e-06 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948085+0000", 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0324e-05 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948108+0000", 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3227e-05 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948119+0000", 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1058e-05 2024-09-13T14:54:57.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948206+0000", 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6368999999999999e-05 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948220+0000", 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3973e-05 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.815945+0000", 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86772523000000001 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.816019+0000", 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4066000000000005e-05 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.816098+0000", 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9357999999999995e-05 2024-09-13T14:54:57.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819620+0000", 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035217120000000002 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819657+0000", 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6918999999999997e-05 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830811+0000", 2024-09-13T14:54:57.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011153573999999999 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830870+0000", 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9154999999999999e-05 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830918+0000", 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8087999999999998e-05 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.576 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:99429 6.0 6:046c2ea6:::benchmark_data_smithi154_175640_object13653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:27.979663+0000", 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 209.223577326, 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.85791414200000005, 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.577 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 99429 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979663+0000", 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979663+0000", 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979665+0000", 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6300000000000001e-06 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979672+0000", 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.029e-06 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979685+0000", 2024-09-13T14:54:57.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2554e-05 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979692+0000", 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.34e-06 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979752+0000", 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9768000000000001e-05 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979760+0000", 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5150000000000006e-06 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819683+0000", 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83992291600000002 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819812+0000", 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012889100000000001 2024-09-13T14:54:57.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819945+0000", 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013302400000000001 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830928+0000", 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0109833 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837394+0000", 2024-09-13T14:54:57.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064660960000000002 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837453+0000", 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8161000000000001e-05 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837528+0000", 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5260999999999997e-05 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837578+0000", 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9657000000000002e-05 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:119325 6.9 6:90d7e752:::benchmark_data_smithi154_175640_object33549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:49.420653+0000", 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 127.78258778999999, 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0220450670000001, 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:54:57.584 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 119325 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420653+0000", 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420653+0000", 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420655+0000", 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2369999999999999e-06 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420664+0000", 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5890000000000003e-06 2024-09-13T14:54:57.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420689+0000", 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4791e-05 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420700+0000", 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1528e-05 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420778+0000", 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7929999999999994e-05 2024-09-13T14:54:57.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420789+0000", 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1442e-05 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.430918+0000", 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.01012883 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:54:57.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.431019+0000", 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101117 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.431099+0000", 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9592000000000006e-05 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.434734+0000", 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036347409999999999 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.589 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.434830+0000", 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5964000000000005e-05 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.442599+0000", 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077691039999999998 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.442652+0000", 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3643000000000001e-05 2024-09-13T14:54:57.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.442698+0000", 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5559000000000002e-05 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:54:57.591 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:54:57.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:57.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:57.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:54:57.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:54:57.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:57.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:57.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:57.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:57.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:57.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:58.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:58.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:58.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:58.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:54:58.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:58.367 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:58.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[102314]: pgmap v798: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1236788378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[102314]: from='client.45362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[107038]: pgmap v798: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1236788378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[107038]: from='client.45362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:58 smithi172 ceph-mon[100696]: pgmap v798: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:54:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1236788378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:54:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:58 smithi172 ceph-mon[100696]: from='client.45362 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:54:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:54:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:54:58.494 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:58.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:58.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:58.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:58.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:58.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:58.869 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:58.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:54:58.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:58.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:54:59.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:54:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:54:59.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:59.224 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:54:59.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:59.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:54:59.369 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:54:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:59 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:59 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:59.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:54:59 smithi154 ceph-mon[107038]: osdmap e263: 8 total, 8 up, 7 in 2024-09-13T14:54:59.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:59 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:59.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:59 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:59.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:54:59 smithi154 ceph-mon[102314]: osdmap e263: 8 total, 8 up, 7 in 2024-09-13T14:54:59.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:54:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:59 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:54:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:59 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:54:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:54:59 smithi172 ceph-mon[100696]: osdmap e263: 8 total, 8 up, 7 in 2024-09-13T14:54:59.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:54:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:54:59.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:59.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:59.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:59.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:54:59.747 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:54:59.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:54:59.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:54:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:54:59.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:54:59.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:00.000 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:00.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:00.128 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:00.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:00 smithi154 ceph-mon[102314]: pgmap v800: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:55:00.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:00 smithi154 ceph-mon[107038]: pgmap v800: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:55:00.230 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:00.256 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:00.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:00.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:00.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:00 smithi172 ceph-mon[100696]: pgmap v800: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail 2024-09-13T14:55:00.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:00.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:00.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:00.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:00.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:00.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:00.892 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:00.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:00.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:01.019 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:01.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:01.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:01.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:01.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:01.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:01.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:01.420 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2024-09-13T14:54:57.038508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6319,"num_read_kb":13776,"num_write":1123,"num_write_kb":15041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2378885,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1182,"ondisk_log_size":1182,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":264,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2436468,"kb_used_data":230104,"kb_used_omap":32,"kb_used_meta":2206239,"kb_avail":653779596,"statfs":{"total":671965249536,"available":669470306304,"internally_reserved":0,"allocated":235626496,"data_stored":225825430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2259189296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001304"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":406,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113814+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:54:53.113814+0000","last_peered":"2024-09-13T14:54:53.113814+0000","last_clean":"2024-09-13T14:54:53.113814+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:54:53.113814+0000","last_undegraded":"2024-09-13T14:54:53.113814+0000","last_fullsized":"2024-09-13T14:54:53.113814+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":128,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101471+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:54:52.101471+0000","last_peered":"2024-09-13T14:54:52.101471+0000","last_clean":"2024-09-13T14:54:52.101471+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:54:52.101471+0000","last_undegraded":"2024-09-13T14:54:52.101471+0000","last_fullsized":"2024-09-13T14:54:52.101471+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":98,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.108013+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:54:52.108013+0000","last_peered":"2024-09-13T14:54:52.108013+0000","last_clean":"2024-09-13T14:54:52.108013+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:54:52.108013+0000","last_undegraded":"2024-09-13T14:54:52.108013+0000","last_fullsized":"2024-09-13T14:54:52.108013+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":394,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105735+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:54:52.105735+0000","last_peered":"2024-09-13T14:54:52.105735+0000","last_clean":"2024-09-13T14:54:52.105735+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:54:52.105735+0000","last_undegraded":"2024-09-13T14:54:52.105735+0000","last_fullsized":"2024-09-13T14:54:52.105735+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":394,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.109422+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:54:52.109422+0000","last_peered":"2024-09-13T14:54:52.109422+0000","last_clean":"2024-09-13T14:54:52.109422+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:54:52.109422+0000","last_undegraded":"2024-09-13T14:54:52.109422+0000","last_fullsized":"2024-09-13T14:54:52.109422+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":198,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101233+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:54:52.101233+0000","last_peered":"2024-09-13T14:54:52.101233+0000","last_clean":"2024-09-13T14:54:52.101233+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:54:52.101233+0000","last_undegraded":"2024-09-13T14:54:52.101233+0000","last_fullsized":"2024-09-13T14:54:52.101233+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":121,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113729+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:54:53.113729+0000","last_peered":"2024-09-13T14:54:53.113729+0000","last_clean":"2024-09-13T14:54:53.113729+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:54:53.113729+0000","last_undegraded":"2024-09-13T14:54:53.113729+0000","last_fullsized":"2024-09-13T14:54:53.113729+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":175,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105890+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:54:52.105890+0000","last_peered":"2024-09-13T14:54:52.105890+0000","last_clean":"2024-09-13T14:54:52.105890+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:54:52.105890+0000","last_undegraded":"2024-09-13T14:54:52.105890+0000","last_fullsized":"2024-09-13T14:54:52.105890+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378396+0000","last_change":"2024-09-13T14:54:52.112411+0000","last_active":"2024-09-13T14:54:53.378396+0000","last_peered":"2024-09-13T14:54:53.378396+0000","last_clean":"2024-09-13T14:54:53.378396+0000","last_became_active":"2024-09-13T14:54:52.112143+0000","last_became_peered":"2024-09-13T14:54:52.112143+0000","last_unstale":"2024-09-13T14:54:53.378396+0000","last_undegraded":"2024-09-13T14:54:53.378396+0000","last_fullsized":"2024-09-13T14:54:53.378396+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:37.763310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":331,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113027+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:54:53.113027+0000","last_peered":"2024-09-13T14:54:53.113027+0000","last_clean":"2024-09-13T14:54:53.113027+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:54:53.113027+0000","last_undegraded":"2024-09-13T14:54:53.113027+0000","last_fullsized":"2024-09-13T14:54:53.113027+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":126,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378055+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:54:53.378055+0000","last_peered":"2024-09-13T14:54:53.378055+0000","last_clean":"2024-09-13T14:54:53.378055+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:54:53.378055+0000","last_undegraded":"2024-09-13T14:54:53.378055+0000","last_fullsized":"2024-09-13T14:54:53.378055+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113347+0000","last_change":"2024-09-13T14:54:52.111899+0000","last_active":"2024-09-13T14:54:53.113347+0000","last_peered":"2024-09-13T14:54:53.113347+0000","last_clean":"2024-09-13T14:54:53.113347+0000","last_became_active":"2024-09-13T14:54:52.110464+0000","last_became_peered":"2024-09-13T14:54:52.110464+0000","last_unstale":"2024-09-13T14:54:53.113347+0000","last_undegraded":"2024-09-13T14:54:53.113347+0000","last_fullsized":"2024-09-13T14:54:53.113347+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:58.130820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":127,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101893+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:54:52.101893+0000","last_peered":"2024-09-13T14:54:52.101893+0000","last_clean":"2024-09-13T14:54:52.101893+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:54:52.101893+0000","last_undegraded":"2024-09-13T14:54:52.101893+0000","last_fullsized":"2024-09-13T14:54:52.101893+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":279,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101563+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:54:52.101563+0000","last_peered":"2024-09-13T14:54:52.101563+0000","last_clean":"2024-09-13T14:54:52.101563+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:54:52.101563+0000","last_undegraded":"2024-09-13T14:54:52.101563+0000","last_fullsized":"2024-09-13T14:54:52.101563+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.114091+0000","last_change":"2024-09-13T14:54:52.113408+0000","last_active":"2024-09-13T14:54:52.114091+0000","last_peered":"2024-09-13T14:54:52.114091+0000","last_clean":"2024-09-13T14:54:52.114091+0000","last_became_active":"2024-09-13T14:54:52.113069+0000","last_became_peered":"2024-09-13T14:54:52.113069+0000","last_unstale":"2024-09-13T14:54:52.114091+0000","last_undegraded":"2024-09-13T14:54:52.114091+0000","last_fullsized":"2024-09-13T14:54:52.114091+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:21.527175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":102,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.108078+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:54:52.108078+0000","last_peered":"2024-09-13T14:54:52.108078+0000","last_clean":"2024-09-13T14:54:52.108078+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:54:52.108078+0000","last_undegraded":"2024-09-13T14:54:52.108078+0000","last_fullsized":"2024-09-13T14:54:52.108078+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":58,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.106489+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:54:52.106489+0000","last_peered":"2024-09-13T14:54:52.106489+0000","last_clean":"2024-09-13T14:54:52.106489+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:54:52.106489+0000","last_undegraded":"2024-09-13T14:54:52.106489+0000","last_fullsized":"2024-09-13T14:54:52.106489+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112563+0000","last_change":"2024-09-13T14:54:52.116627+0000","last_active":"2024-09-13T14:54:53.112563+0000","last_peered":"2024-09-13T14:54:53.112563+0000","last_clean":"2024-09-13T14:54:53.112563+0000","last_became_active":"2024-09-13T14:54:52.116334+0000","last_became_peered":"2024-09-13T14:54:52.116334+0000","last_unstale":"2024-09-13T14:54:53.112563+0000","last_undegraded":"2024-09-13T14:54:53.112563+0000","last_fullsized":"2024-09-13T14:54:53.112563+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:09:46.738656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":114,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107879+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:54:52.107879+0000","last_peered":"2024-09-13T14:54:52.107879+0000","last_clean":"2024-09-13T14:54:52.107879+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:54:52.107879+0000","last_undegraded":"2024-09-13T14:54:52.107879+0000","last_fullsized":"2024-09-13T14:54:52.107879+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":373,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105840+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:54:52.105840+0000","last_peered":"2024-09-13T14:54:52.105840+0000","last_clean":"2024-09-13T14:54:52.105840+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:54:52.105840+0000","last_undegraded":"2024-09-13T14:54:52.105840+0000","last_fullsized":"2024-09-13T14:54:52.105840+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.118064+0000","last_change":"2024-09-13T14:54:52.117749+0000","last_active":"2024-09-13T14:54:52.118064+0000","last_peered":"2024-09-13T14:54:52.118064+0000","last_clean":"2024-09-13T14:54:52.118064+0000","last_became_active":"2024-09-13T14:54:52.117446+0000","last_became_peered":"2024-09-13T14:54:52.117446+0000","last_unstale":"2024-09-13T14:54:52.118064+0000","last_undegraded":"2024-09-13T14:54:52.118064+0000","last_fullsized":"2024-09-13T14:54:52.118064+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:27.665097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":383,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113584+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:54:53.113584+0000","last_peered":"2024-09-13T14:54:53.113584+0000","last_clean":"2024-09-13T14:54:53.113584+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:54:53.113584+0000","last_undegraded":"2024-09-13T14:54:53.113584+0000","last_fullsized":"2024-09-13T14:54:53.113584+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":245,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101620+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:54:52.101620+0000","last_peered":"2024-09-13T14:54:52.101620+0000","last_clean":"2024-09-13T14:54:52.101620+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:54:52.101620+0000","last_undegraded":"2024-09-13T14:54:52.101620+0000","last_fullsized":"2024-09-13T14:54:52.101620+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112989+0000","last_change":"2024-09-13T14:54:52.113650+0000","last_active":"2024-09-13T14:54:53.112989+0000","last_peered":"2024-09-13T14:54:53.112989+0000","last_clean":"2024-09-13T14:54:53.112989+0000","last_became_active":"2024-09-13T14:54:52.113338+0000","last_became_peered":"2024-09-13T14:54:52.113338+0000","last_unstale":"2024-09-13T14:54:53.112989+0000","last_undegraded":"2024-09-13T14:54:53.112989+0000","last_fullsized":"2024-09-13T14:54:53.112989+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:27:43.933445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":146,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105957+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:54:52.105957+0000","last_peered":"2024-09-13T14:54:52.105957+0000","last_clean":"2024-09-13T14:54:52.105957+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:54:52.105957+0000","last_undegraded":"2024-09-13T14:54:52.105957+0000","last_fullsized":"2024-09-13T14:54:52.105957+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":242,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101827+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:54:52.101827+0000","last_peered":"2024-09-13T14:54:52.101827+0000","last_clean":"2024-09-13T14:54:52.101827+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:54:52.101827+0000","last_undegraded":"2024-09-13T14:54:52.101827+0000","last_fullsized":"2024-09-13T14:54:52.101827+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.255026+0000","last_change":"2024-09-13T14:54:52.113259+0000","last_active":"2024-09-13T14:54:53.255026+0000","last_peered":"2024-09-13T14:54:53.255026+0000","last_clean":"2024-09-13T14:54:53.255026+0000","last_became_active":"2024-09-13T14:54:52.112910+0000","last_became_peered":"2024-09-13T14:54:52.112910+0000","last_unstale":"2024-09-13T14:54:53.255026+0000","last_undegraded":"2024-09-13T14:54:53.255026+0000","last_fullsized":"2024-09-13T14:54:53.255026+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:37.906148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":238,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101651+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:54:52.101651+0000","last_peered":"2024-09-13T14:54:52.101651+0000","last_clean":"2024-09-13T14:54:52.101651+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:54:52.101651+0000","last_undegraded":"2024-09-13T14:54:52.101651+0000","last_fullsized":"2024-09-13T14:54:52.101651+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":272,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101915+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:54:52.101915+0000","last_peered":"2024-09-13T14:54:52.101915+0000","last_clean":"2024-09-13T14:54:52.101915+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:54:52.101915+0000","last_undegraded":"2024-09-13T14:54:52.101915+0000","last_fullsized":"2024-09-13T14:54:52.101915+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.254919+0000","last_change":"2024-09-13T14:54:52.113761+0000","last_active":"2024-09-13T14:54:53.254919+0000","last_peered":"2024-09-13T14:54:53.254919+0000","last_clean":"2024-09-13T14:54:53.254919+0000","last_became_active":"2024-09-13T14:54:52.113441+0000","last_became_peered":"2024-09-13T14:54:52.113441+0000","last_unstale":"2024-09-13T14:54:53.254919+0000","last_undegraded":"2024-09-13T14:54:53.254919+0000","last_fullsized":"2024-09-13T14:54:53.254919+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:01.375020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":122,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112992+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:54:53.112992+0000","last_peered":"2024-09-13T14:54:53.112992+0000","last_clean":"2024-09-13T14:54:53.112992+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:54:53.112992+0000","last_undegraded":"2024-09-13T14:54:53.112992+0000","last_fullsized":"2024-09-13T14:54:53.112992+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":231,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113125+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:54:53.113125+0000","last_peered":"2024-09-13T14:54:53.113125+0000","last_clean":"2024-09-13T14:54:53.113125+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:54:53.113125+0000","last_undegraded":"2024-09-13T14:54:53.113125+0000","last_fullsized":"2024-09-13T14:54:53.113125+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.117313+0000","last_change":"2024-09-13T14:54:52.116764+0000","last_active":"2024-09-13T14:54:52.117313+0000","last_peered":"2024-09-13T14:54:52.117313+0000","last_clean":"2024-09-13T14:54:52.117313+0000","last_became_active":"2024-09-13T14:54:52.116144+0000","last_became_peered":"2024-09-13T14:54:52.116144+0000","last_unstale":"2024-09-13T14:54:52.117313+0000","last_undegraded":"2024-09-13T14:54:52.117313+0000","last_fullsized":"2024-09-13T14:54:52.117313+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:21.882611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":576,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105909+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:54:52.105909+0000","last_peered":"2024-09-13T14:54:52.105909+0000","last_clean":"2024-09-13T14:54:52.105909+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:54:52.105909+0000","last_undegraded":"2024-09-13T14:54:52.105909+0000","last_fullsized":"2024-09-13T14:54:52.105909+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":829,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107605+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:54:52.107605+0000","last_peered":"2024-09-13T14:54:52.107605+0000","last_clean":"2024-09-13T14:54:52.107605+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:54:52.107605+0000","last_undegraded":"2024-09-13T14:54:52.107605+0000","last_fullsized":"2024-09-13T14:54:52.107605+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":342,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113405+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:54:53.113405+0000","last_peered":"2024-09-13T14:54:53.113405+0000","last_clean":"2024-09-13T14:54:53.113405+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:54:53.113405+0000","last_undegraded":"2024-09-13T14:54:53.113405+0000","last_fullsized":"2024-09-13T14:54:53.113405+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":59,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.106292+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:54:52.106292+0000","last_peered":"2024-09-13T14:54:52.106292+0000","last_clean":"2024-09-13T14:54:52.106292+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:54:52.106292+0000","last_undegraded":"2024-09-13T14:54:52.106292+0000","last_fullsized":"2024-09-13T14:54:52.106292+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113070+0000","last_change":"2024-09-13T14:54:52.115847+0000","last_active":"2024-09-13T14:54:53.113070+0000","last_peered":"2024-09-13T14:54:53.113070+0000","last_clean":"2024-09-13T14:54:53.113070+0000","last_became_active":"2024-09-13T14:54:52.115534+0000","last_became_peered":"2024-09-13T14:54:52.115534+0000","last_unstale":"2024-09-13T14:54:53.113070+0000","last_undegraded":"2024-09-13T14:54:53.113070+0000","last_fullsized":"2024-09-13T14:54:53.113070+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:05.798814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":423,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.377828+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:54:53.377828+0000","last_peered":"2024-09-13T14:54:53.377828+0000","last_clean":"2024-09-13T14:54:53.377828+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:54:53.377828+0000","last_undegraded":"2024-09-13T14:54:53.377828+0000","last_fullsized":"2024-09-13T14:54:53.377828+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":388,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.108120+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:54:52.108120+0000","last_peered":"2024-09-13T14:54:52.108120+0000","last_clean":"2024-09-13T14:54:52.108120+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:54:52.108120+0000","last_undegraded":"2024-09-13T14:54:52.108120+0000","last_fullsized":"2024-09-13T14:54:52.108120+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":398,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.109802+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:54:52.109802+0000","last_peered":"2024-09-13T14:54:52.109802+0000","last_clean":"2024-09-13T14:54:52.109802+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:54:52.109802+0000","last_undegraded":"2024-09-13T14:54:52.109802+0000","last_fullsized":"2024-09-13T14:54:52.109802+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.115319+0000","last_change":"2024-09-13T14:54:52.114072+0000","last_active":"2024-09-13T14:54:52.115319+0000","last_peered":"2024-09-13T14:54:52.115319+0000","last_clean":"2024-09-13T14:54:52.115319+0000","last_became_active":"2024-09-13T14:54:52.113758+0000","last_became_peered":"2024-09-13T14:54:52.113758+0000","last_unstale":"2024-09-13T14:54:52.115319+0000","last_undegraded":"2024-09-13T14:54:52.115319+0000","last_fullsized":"2024-09-13T14:54:52.115319+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:28.578531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":926,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112727+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:54:53.112727+0000","last_peered":"2024-09-13T14:54:53.112727+0000","last_clean":"2024-09-13T14:54:53.112727+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:54:53.112727+0000","last_undegraded":"2024-09-13T14:54:53.112727+0000","last_fullsized":"2024-09-13T14:54:53.112727+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":381,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378015+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:54:53.378015+0000","last_peered":"2024-09-13T14:54:53.378015+0000","last_clean":"2024-09-13T14:54:53.378015+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:54:53.378015+0000","last_undegraded":"2024-09-13T14:54:53.378015+0000","last_fullsized":"2024-09-13T14:54:53.378015+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":134,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378169+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:54:53.378169+0000","last_peered":"2024-09-13T14:54:53.378169+0000","last_clean":"2024-09-13T14:54:53.378169+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:54:53.378169+0000","last_undegraded":"2024-09-13T14:54:53.378169+0000","last_fullsized":"2024-09-13T14:54:53.378169+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.255137+0000","last_change":"2024-09-13T14:54:52.113753+0000","last_active":"2024-09-13T14:54:53.255137+0000","last_peered":"2024-09-13T14:54:53.255137+0000","last_clean":"2024-09-13T14:54:53.255137+0000","last_became_active":"2024-09-13T14:54:52.113413+0000","last_became_peered":"2024-09-13T14:54:52.113413+0000","last_unstale":"2024-09-13T14:54:53.255137+0000","last_undegraded":"2024-09-13T14:54:53.255137+0000","last_fullsized":"2024-09-13T14:54:53.255137+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:42.771873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":117,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107782+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:54:52.107782+0000","last_peered":"2024-09-13T14:54:52.107782+0000","last_clean":"2024-09-13T14:54:52.107782+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:54:52.107782+0000","last_undegraded":"2024-09-13T14:54:52.107782+0000","last_fullsized":"2024-09-13T14:54:52.107782+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":247,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101620+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:54:52.101620+0000","last_peered":"2024-09-13T14:54:52.101620+0000","last_clean":"2024-09-13T14:54:52.101620+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:54:52.101620+0000","last_undegraded":"2024-09-13T14:54:52.101620+0000","last_fullsized":"2024-09-13T14:54:52.101620+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107633+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:54:52.107633+0000","last_peered":"2024-09-13T14:54:52.107633+0000","last_clean":"2024-09-13T14:54:52.107633+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:54:52.107633+0000","last_undegraded":"2024-09-13T14:54:52.107633+0000","last_fullsized":"2024-09-13T14:54:52.107633+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378218+0000","last_change":"2024-09-13T14:54:52.107585+0000","last_active":"2024-09-13T14:54:53.378218+0000","last_peered":"2024-09-13T14:54:53.378218+0000","last_clean":"2024-09-13T14:54:53.378218+0000","last_became_active":"2024-09-13T14:54:52.106845+0000","last_became_peered":"2024-09-13T14:54:52.106845+0000","last_unstale":"2024-09-13T14:54:53.378218+0000","last_undegraded":"2024-09-13T14:54:53.378218+0000","last_fullsized":"2024-09-13T14:54:53.378218+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:32.999267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":117,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107883+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:54:52.107883+0000","last_peered":"2024-09-13T14:54:52.107883+0000","last_clean":"2024-09-13T14:54:52.107883+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:54:52.107883+0000","last_undegraded":"2024-09-13T14:54:52.107883+0000","last_fullsized":"2024-09-13T14:54:52.107883+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":935,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378135+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:54:53.378135+0000","last_peered":"2024-09-13T14:54:53.378135+0000","last_clean":"2024-09-13T14:54:53.378135+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:54:53.378135+0000","last_undegraded":"2024-09-13T14:54:53.378135+0000","last_fullsized":"2024-09-13T14:54:53.378135+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":40,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105640+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:54:52.105640+0000","last_peered":"2024-09-13T14:54:52.105640+0000","last_clean":"2024-09-13T14:54:52.105640+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:54:52.105640+0000","last_undegraded":"2024-09-13T14:54:52.105640+0000","last_fullsized":"2024-09-13T14:54:52.105640+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.377910+0000","last_change":"2024-09-13T14:54:52.114535+0000","last_active":"2024-09-13T14:54:53.377910+0000","last_peered":"2024-09-13T14:54:53.377910+0000","last_clean":"2024-09-13T14:54:53.377910+0000","last_became_active":"2024-09-13T14:54:52.114212+0000","last_became_peered":"2024-09-13T14:54:52.114212+0000","last_unstale":"2024-09-13T14:54:53.377910+0000","last_undegraded":"2024-09-13T14:54:53.377910+0000","last_fullsized":"2024-09-13T14:54:53.377910+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:31.266521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":387,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378139+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:54:53.378139+0000","last_peered":"2024-09-13T14:54:53.378139+0000","last_clean":"2024-09-13T14:54:53.378139+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:54:53.378139+0000","last_undegraded":"2024-09-13T14:54:53.378139+0000","last_fullsized":"2024-09-13T14:54:53.378139+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":607,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.110740+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:54:52.110740+0000","last_peered":"2024-09-13T14:54:52.110740+0000","last_clean":"2024-09-13T14:54:52.110740+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:54:52.110740+0000","last_undegraded":"2024-09-13T14:54:52.110740+0000","last_fullsized":"2024-09-13T14:54:52.110740+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":384,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105763+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:54:52.105763+0000","last_peered":"2024-09-13T14:54:52.105763+0000","last_clean":"2024-09-13T14:54:52.105763+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:54:52.105763+0000","last_undegraded":"2024-09-13T14:54:52.105763+0000","last_fullsized":"2024-09-13T14:54:52.105763+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112839+0000","last_change":"2024-09-13T14:54:52.112337+0000","last_active":"2024-09-13T14:54:53.112839+0000","last_peered":"2024-09-13T14:54:53.112839+0000","last_clean":"2024-09-13T14:54:53.112839+0000","last_became_active":"2024-09-13T14:54:52.112057+0000","last_became_peered":"2024-09-13T14:54:52.112057+0000","last_unstale":"2024-09-13T14:54:53.112839+0000","last_undegraded":"2024-09-13T14:54:53.112839+0000","last_fullsized":"2024-09-13T14:54:53.112839+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:58.256221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":406,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.114208+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:54:53.114208+0000","last_peered":"2024-09-13T14:54:53.114208+0000","last_clean":"2024-09-13T14:54:53.114208+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:54:53.114208+0000","last_undegraded":"2024-09-13T14:54:53.114208+0000","last_fullsized":"2024-09-13T14:54:53.114208+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":837,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378364+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:54:53.378364+0000","last_peered":"2024-09-13T14:54:53.378364+0000","last_clean":"2024-09-13T14:54:53.378364+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:54:53.378364+0000","last_undegraded":"2024-09-13T14:54:53.378364+0000","last_fullsized":"2024-09-13T14:54:53.378364+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":122,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112897+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:54:53.112897+0000","last_peered":"2024-09-13T14:54:53.112897+0000","last_clean":"2024-09-13T14:54:53.112897+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:54:53.112897+0000","last_undegraded":"2024-09-13T14:54:53.112897+0000","last_fullsized":"2024-09-13T14:54:53.112897+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113502+0000","last_change":"2024-09-13T14:54:52.112510+0000","last_active":"2024-09-13T14:54:53.113502+0000","last_peered":"2024-09-13T14:54:53.113502+0000","last_clean":"2024-09-13T14:54:53.113502+0000","last_became_active":"2024-09-13T14:54:52.112262+0000","last_became_peered":"2024-09-13T14:54:52.112262+0000","last_unstale":"2024-09-13T14:54:53.113502+0000","last_undegraded":"2024-09-13T14:54:53.113502+0000","last_fullsized":"2024-09-13T14:54:53.113502+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:51:02.681864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":378,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.109323+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:54:52.109323+0000","last_peered":"2024-09-13T14:54:52.109323+0000","last_clean":"2024-09-13T14:54:52.109323+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:54:52.109323+0000","last_undegraded":"2024-09-13T14:54:52.109323+0000","last_fullsized":"2024-09-13T14:54:52.109323+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":814,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.106813+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:54:52.106813+0000","last_peered":"2024-09-13T14:54:52.106813+0000","last_clean":"2024-09-13T14:54:52.106813+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:54:52.106813+0000","last_undegraded":"2024-09-13T14:54:52.106813+0000","last_fullsized":"2024-09-13T14:54:52.106813+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":249,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101321+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:54:52.101321+0000","last_peered":"2024-09-13T14:54:52.101321+0000","last_clean":"2024-09-13T14:54:52.101321+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:54:52.101321+0000","last_undegraded":"2024-09-13T14:54:52.101321+0000","last_fullsized":"2024-09-13T14:54:52.101321+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":245,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101573+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:54:52.101573+0000","last_peered":"2024-09-13T14:54:52.101573+0000","last_clean":"2024-09-13T14:54:52.101573+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:54:52.101573+0000","last_undegraded":"2024-09-13T14:54:52.101573+0000","last_fullsized":"2024-09-13T14:54:52.101573+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":293,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101769+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:54:52.101769+0000","last_peered":"2024-09-13T14:54:52.101769+0000","last_clean":"2024-09-13T14:54:52.101769+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:54:52.101769+0000","last_undegraded":"2024-09-13T14:54:52.101769+0000","last_fullsized":"2024-09-13T14:54:52.101769+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":385,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.377948+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:54:53.377948+0000","last_peered":"2024-09-13T14:54:53.377948+0000","last_clean":"2024-09-13T14:54:53.377948+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:54:53.377948+0000","last_undegraded":"2024-09-13T14:54:53.377948+0000","last_fullsized":"2024-09-13T14:54:53.377948+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":225,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113763+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:54:53.113763+0000","last_peered":"2024-09-13T14:54:53.113763+0000","last_clean":"2024-09-13T14:54:53.113763+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:54:53.113763+0000","last_undegraded":"2024-09-13T14:54:53.113763+0000","last_fullsized":"2024-09-13T14:54:53.113763+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":125,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107947+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:54:52.107947+0000","last_peered":"2024-09-13T14:54:52.107947+0000","last_clean":"2024-09-13T14:54:52.107947+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:54:52.107947+0000","last_undegraded":"2024-09-13T14:54:52.107947+0000","last_fullsized":"2024-09-13T14:54:52.107947+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":362,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112770+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:54:53.112770+0000","last_peered":"2024-09-13T14:54:53.112770+0000","last_clean":"2024-09-13T14:54:53.112770+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:54:53.112770+0000","last_undegraded":"2024-09-13T14:54:53.112770+0000","last_fullsized":"2024-09-13T14:54:53.112770+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":415,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.110252+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:54:52.110252+0000","last_peered":"2024-09-13T14:54:52.110252+0000","last_clean":"2024-09-13T14:54:52.110252+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:54:52.110252+0000","last_undegraded":"2024-09-13T14:54:52.110252+0000","last_fullsized":"2024-09-13T14:54:52.110252+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":413,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113251+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:54:53.113251+0000","last_peered":"2024-09-13T14:54:53.113251+0000","last_clean":"2024-09-13T14:54:53.113251+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:54:53.113251+0000","last_undegraded":"2024-09-13T14:54:53.113251+0000","last_fullsized":"2024-09-13T14:54:53.113251+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":155,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.106699+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:54:52.106699+0000","last_peered":"2024-09-13T14:54:52.106699+0000","last_clean":"2024-09-13T14:54:52.106699+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:54:52.106699+0000","last_undegraded":"2024-09-13T14:54:52.106699+0000","last_fullsized":"2024-09-13T14:54:52.106699+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":380,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113332+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:54:53.113332+0000","last_peered":"2024-09-13T14:54:53.113332+0000","last_clean":"2024-09-13T14:54:53.113332+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:54:53.113332+0000","last_undegraded":"2024-09-13T14:54:53.113332+0000","last_fullsized":"2024-09-13T14:54:53.113332+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":395,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.109679+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:54:52.109679+0000","last_peered":"2024-09-13T14:54:52.109679+0000","last_clean":"2024-09-13T14:54:52.109679+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:54:52.109679+0000","last_undegraded":"2024-09-13T14:54:52.109679+0000","last_fullsized":"2024-09-13T14:54:52.109679+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":365,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.377839+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:54:53.377839+0000","last_peered":"2024-09-13T14:54:53.377839+0000","last_clean":"2024-09-13T14:54:53.377839+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:54:53.377839+0000","last_undegraded":"2024-09-13T14:54:53.377839+0000","last_fullsized":"2024-09-13T14:54:53.377839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":229,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113672+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:54:53.113672+0000","last_peered":"2024-09-13T14:54:53.113672+0000","last_clean":"2024-09-13T14:54:53.113672+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:54:53.113672+0000","last_undegraded":"2024-09-13T14:54:53.113672+0000","last_fullsized":"2024-09-13T14:54:53.113672+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":243,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101409+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:54:52.101409+0000","last_peered":"2024-09-13T14:54:52.101409+0000","last_clean":"2024-09-13T14:54:52.101409+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:54:52.101409+0000","last_undegraded":"2024-09-13T14:54:52.101409+0000","last_fullsized":"2024-09-13T14:54:52.101409+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":124,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.112653+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:54:53.112653+0000","last_peered":"2024-09-13T14:54:53.112653+0000","last_clean":"2024-09-13T14:54:53.112653+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:54:53.112653+0000","last_undegraded":"2024-09-13T14:54:53.112653+0000","last_fullsized":"2024-09-13T14:54:53.112653+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":387,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.105663+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:54:52.105663+0000","last_peered":"2024-09-13T14:54:52.105663+0000","last_clean":"2024-09-13T14:54:52.105663+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:54:52.105663+0000","last_undegraded":"2024-09-13T14:54:52.105663+0000","last_fullsized":"2024-09-13T14:54:52.105663+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":253,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101492+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:54:52.101492+0000","last_peered":"2024-09-13T14:54:52.101492+0000","last_clean":"2024-09-13T14:54:52.101492+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:54:52.101492+0000","last_undegraded":"2024-09-13T14:54:52.101492+0000","last_fullsized":"2024-09-13T14:54:52.101492+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":100,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107823+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:54:52.107823+0000","last_peered":"2024-09-13T14:54:52.107823+0000","last_clean":"2024-09-13T14:54:52.107823+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:54:52.107823+0000","last_undegraded":"2024-09-13T14:54:52.107823+0000","last_fullsized":"2024-09-13T14:54:52.107823+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":241,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.101319+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:54:52.101319+0000","last_peered":"2024-09-13T14:54:52.101319+0000","last_clean":"2024-09-13T14:54:52.101319+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:54:52.101319+0000","last_undegraded":"2024-09-13T14:54:52.101319+0000","last_fullsized":"2024-09-13T14:54:52.101319+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":102,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.107986+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:54:52.107986+0000","last_peered":"2024-09-13T14:54:52.107986+0000","last_clean":"2024-09-13T14:54:52.107986+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:54:52.107986+0000","last_undegraded":"2024-09-13T14:54:52.107986+0000","last_fullsized":"2024-09-13T14:54:52.107986+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":383,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113098+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:54:53.113098+0000","last_peered":"2024-09-13T14:54:53.113098+0000","last_clean":"2024-09-13T14:54:53.113098+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:54:53.113098+0000","last_undegraded":"2024-09-13T14:54:53.113098+0000","last_fullsized":"2024-09-13T14:54:53.113098+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":400,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.113752+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:54:53.113752+0000","last_peered":"2024-09-13T14:54:53.113752+0000","last_clean":"2024-09-13T14:54:53.113752+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:54:53.113752+0000","last_undegraded":"2024-09-13T14:54:53.113752+0000","last_fullsized":"2024-09-13T14:54:53.113752+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":272,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-13T14:54:53.378032+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:54:53.378032+0000","last_peered":"2024-09-13T14:54:53.378032+0000","last_clean":"2024-09-13T14:54:53.378032+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:54:53.378032+0000","last_undegraded":"2024-09-13T14:54:53.378032+0000","last_fullsized":"2024-09-13T14:54:53.378032+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":451,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-13T14:54:52.110155+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:54:52.110155+0000","last_peered":"2024-09-13T14:54:52.110155+0000","last_clean":"2024-09-13T14:54:52.110155+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:54:52.110155+0000","last_undegraded":"2024-09-13T14:54:52.110155+0000","last_fullsized":"2024-09-13T14:54:52.110155+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234330,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495966,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367988,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":364158,"kb_avail":93377164,"statfs":{"total":95995035648,"available":95618215936,"internally_reserved":0,"allocated":3903488,"data_stored":2503584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":372898247},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757601,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218928,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":215678,"kb_avail":93526224,"statfs":{"total":95995035648,"available":95770853376,"internally_reserved":0,"allocated":3313664,"data_stored":1917968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":220854727},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019241,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313276,"kb_used_data":3248,"kb_used_omap":1,"kb_used_meta":310014,"kb_avail":93431876,"statfs":{"total":95995035648,"available":95674241024,"internally_reserved":0,"allocated":3325952,"data_stored":1922450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":317454790},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468126,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313844,"kb_used_data":7332,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93431308,"statfs":{"total":95995035648,"available":95673659392,"internally_reserved":0,"allocated":7507968,"data_stored":6112046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":313846948},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631655,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408324,"kb_used_data":7348,"kb_used_omap":8,"kb_used_meta":400951,"kb_avail":93336828,"statfs":{"total":95995035648,"available":95576911872,"internally_reserved":0,"allocated":7524352,"data_stored":6117280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":410574815},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795178,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":521840,"kb_used_data":197224,"kb_used_omap":7,"kb_used_meta":324600,"kb_avail":93223312,"statfs":{"total":95995035648,"available":95460671488,"internally_reserved":0,"allocated":201957376,"data_stored":200557783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":332391026},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958708,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292268,"kb_used_data":7904,"kb_used_omap":7,"kb_used_meta":284344,"kb_avail":93452884,"statfs":{"total":95995035648,"available":95695753216,"internally_reserved":0,"allocated":8093696,"data_stored":6694319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":291168743},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:55:01.442 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:01.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:01.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:01.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:01.655 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:01.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:01.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:01.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:01.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:01.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:01.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:55:01.952 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:55:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:55:01] "GET /metrics HTTP/1.1" 200 36444 "" "Prometheus/2.51.0" 2024-09-13T14:55:02.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:02.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:02.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:02.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:02.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:02 smithi154 ceph-mon[102314]: pgmap v801: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:02.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:02 smithi154 ceph-mon[107038]: pgmap v801: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:02.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:02.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:02.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:02.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:02 smithi172 ceph-mon[100696]: pgmap v801: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:02.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:02.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:02.591 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:55:02.591 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:02.599 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:55:02.599 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:02.615 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:55:02.615 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:02.631 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:55:02.631 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:02.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:02.648 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:55:02.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:02.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:02.665 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:55:02.665 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:02.681 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:55:02.682 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:02.698 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:55:02.699 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:02.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:02.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:02.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:02.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:02.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:02.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:03.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:03.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:03.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:03.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:03.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:03.282 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:03.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:03.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/514761733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/514761733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/514761733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:03.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:03.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:03.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:03.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:03.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:03.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:03.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:03.914 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:03.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:04.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:04.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:04.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:04.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:04.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:04.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:04.348 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:04.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:04.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:04.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:55:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:04 smithi154 ceph-mon[102314]: pgmap v802: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:04 smithi154 ceph-mon[107038]: pgmap v802: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:04 smithi172 ceph-mon[100696]: pgmap v802: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:04.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:04.539 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:04.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:04.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:04.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:04.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:04.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:04.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:04.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:05.014 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:05.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:05.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:05.165 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-13T14:55:05.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:05.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:05.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:05.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:05.420 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:05.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:05 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3827288151' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:05.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:05 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:05.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:05 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3827288151' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:05.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:05 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:05 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3827288151' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:05 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T14:55:05.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:05.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:05.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:55:05.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:05.672 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:05.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:05.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:05.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:05.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:05.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:05.988 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:05.988 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:06.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:06.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:06.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:06.179 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:06.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:06 smithi154 ceph-mon[102314]: pgmap v803: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:06.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:06 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:55:06.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:06 smithi154 ceph-mon[102314]: osdmap e264: 8 total, 8 up, 7 in 2024-09-13T14:55:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:06 smithi154 ceph-mon[107038]: pgmap v803: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:06 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:55:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:06 smithi154 ceph-mon[107038]: osdmap e264: 8 total, 8 up, 7 in 2024-09-13T14:55:06.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:06.307 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:06.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:06.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:06.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:06 smithi172 ceph-mon[100696]: pgmap v803: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:06 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T14:55:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:06 smithi172 ceph-mon[100696]: osdmap e264: 8 total, 8 up, 7 in 2024-09-13T14:55:06.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:06.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:55:06.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:06.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:06.809 DEBUG:teuthology.orchestra.run.smithi154:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-13T14:55:06.812 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T14:55:06.812 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T14:55:06.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:06.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:06.884 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:hints = 1 2024-09-13T14:55:06.884 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T14:55:06.884 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object prefix: benchmark_data_smithi154_225924 2024-09-13T14:55:06.884 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:06.885 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 0 0 0 0 0 0 - 0 2024-09-13T14:55:06.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:06.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:06.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:06.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:07.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:07.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:07.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:07.195 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:07.198 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:07.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:07.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:07.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:07.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:07.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:07 smithi154 ceph-mon[102314]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:55:07.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:07 smithi154 ceph-mon[107038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:55:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:07 smithi172 ceph-mon[100696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T14:55:07.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:07.582 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:07.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:07.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:07.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:07.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:07.884 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 1 16 3264 3248 202.971 203 0.0102754 0.0049109 2024-09-13T14:55:07.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:07.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:07.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:08.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:08.123 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:08.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:08.252 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:08.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:08.382 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:08 smithi154 ceph-mon[102314]: pgmap v805: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:08.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:08 smithi154 ceph-mon[107038]: pgmap v805: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:08.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:08 smithi172 ceph-mon[100696]: pgmap v805: 89 pgs: 89 active+clean; 4.6 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-13T14:55:08.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:08.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:08.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:08.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:08.783 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:08.885 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 2 16 6232 6216 194.228 185.5 0.00381982 0.00513642 2024-09-13T14:55:08.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:08.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:08.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:09.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:09.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:09.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:09.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:09.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:09.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:09.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:09.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:09.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:09.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:09.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:09.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:09.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:09.850 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:09.885 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 3 16 8977 8961 186.668 171.562 0.00358282 0.00534586 2024-09-13T14:55:09.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:09.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:09.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:10.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:10.104 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:10.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:10.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:10.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:10.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:10.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:10.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:10 smithi154 ceph-mon[102314]: pgmap v806: 89 pgs: 89 active+clean; 81 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 7.6 MiB/s wr, 121 op/s 2024-09-13T14:55:10.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:10 smithi154 ceph-mon[107038]: pgmap v806: 89 pgs: 89 active+clean; 81 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 7.6 MiB/s wr, 121 op/s 2024-09-13T14:55:10.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:10 smithi172 ceph-mon[100696]: pgmap v806: 89 pgs: 89 active+clean; 81 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail; 7.6 MiB/s wr, 121 op/s 2024-09-13T14:55:10.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:10.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:10.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:10.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:10.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:10.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 4 16 10933 10917 170.545 122.25 0.00768198 0.00585351 2024-09-13T14:55:10.909 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:10.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:11.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:11.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:11.154 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:11.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:11.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:11.319 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:11.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:11.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:11 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:11.496 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:11 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:11.497 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:11 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:55:11.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:11.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:11.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:11.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:11.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:11.885 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 5 16 13382 13366 167.046 153.062 0.00430214 0.00597759 2024-09-13T14:55:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:11.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:11.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:11.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:12.022 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:12.089 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:55:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:55:11] "GET /metrics HTTP/1.1" 200 36444 "" "Prometheus/2.51.0" 2024-09-13T14:55:12.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:12.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:12.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.379 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:12 smithi154 ceph-mon[102314]: pgmap v807: 89 pgs: 89 active+clean; 376 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 37 MiB/s wr, 594 op/s 2024-09-13T14:55:12.380 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:12 smithi154 ceph-mon[107038]: pgmap v807: 89 pgs: 89 active+clean; 376 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 37 MiB/s wr, 594 op/s 2024-09-13T14:55:12.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:12 smithi172 ceph-mon[100696]: pgmap v807: 89 pgs: 89 active+clean; 376 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 37 MiB/s wr, 594 op/s 2024-09-13T14:55:12.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:12.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:12.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:12.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:12.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:12.885 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 6 16 15518 15502 161.453 133.5 0.0200507 0.00618508 2024-09-13T14:55:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:55:12.926 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:55:12.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:12.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:12.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:13.065 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:55:13.065 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:13.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:13.103 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:55:13.103 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:13.123 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:13.140 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:55:13.141 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:13.177 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:55:13.177 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:13.211 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:55:13.211 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:13.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:13.236 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:13.248 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:55:13.248 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:13.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:13.288 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:55:13.288 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:13.320 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:55:13.321 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:13.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:13.421 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:13 smithi172 ceph-mon[100696]: from='client.55132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:55:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/632860753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3035220272' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:13.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:13.553 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:13.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:13.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:13.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[102314]: from='client.55132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/632860753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3035220272' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[107038]: from='client.55132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/632860753' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3035220272' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:13.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:13.796 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2024-09-13T14:55:11.041874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394336229,"num_objects":5972,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":6319,"num_read_kb":13776,"num_write":7067,"num_write_kb":395457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2378885,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7126,"ondisk_log_size":7126,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":268,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3485580,"kb_used_data":1310668,"kb_used_omap":32,"kb_used_meta":2174815,"kb_avail":652730484,"statfs":{"total":671965249536,"available":668396015616,"internally_reserved":0,"allocated":1342124032,"data_stored":1332265948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2227011120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5944,"num_write_kb":380416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002472"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":410,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143426+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:55:06.143426+0000","last_peered":"2024-09-13T14:55:06.143426+0000","last_clean":"2024-09-13T14:55:06.143426+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:55:06.143426+0000","last_undegraded":"2024-09-13T14:55:06.143426+0000","last_fullsized":"2024-09-13T14:55:06.143426+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":131,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:55:04.237830+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:55:04.237830+0000","last_peered":"2024-09-13T14:55:04.237830+0000","last_clean":"2024-09-13T14:55:04.237830+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:55:04.237830+0000","last_undegraded":"2024-09-13T14:55:04.237830+0000","last_fullsized":"2024-09-13T14:55:04.237830+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":100,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998417+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:54:58.998417+0000","last_peered":"2024-09-13T14:54:58.998417+0000","last_clean":"2024-09-13T14:54:58.998417+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:54:58.998417+0000","last_undegraded":"2024-09-13T14:54:58.998417+0000","last_fullsized":"2024-09-13T14:54:58.998417+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":400,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127183+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:55:06.127183+0000","last_peered":"2024-09-13T14:55:06.127183+0000","last_clean":"2024-09-13T14:55:06.127183+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:55:06.127183+0000","last_undegraded":"2024-09-13T14:55:06.127183+0000","last_fullsized":"2024-09-13T14:55:06.127183+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":400,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127653+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:55:06.127653+0000","last_peered":"2024-09-13T14:55:06.127653+0000","last_clean":"2024-09-13T14:55:06.127653+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:55:06.127653+0000","last_undegraded":"2024-09-13T14:55:06.127653+0000","last_fullsized":"2024-09-13T14:55:06.127653+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":201,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:55:04.237781+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:55:04.237781+0000","last_peered":"2024-09-13T14:55:04.237781+0000","last_clean":"2024-09-13T14:55:04.237781+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:55:04.237781+0000","last_undegraded":"2024-09-13T14:55:04.237781+0000","last_fullsized":"2024-09-13T14:55:04.237781+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":125,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142902+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:55:06.142902+0000","last_peered":"2024-09-13T14:55:06.142902+0000","last_clean":"2024-09-13T14:55:06.142902+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:55:06.142902+0000","last_undegraded":"2024-09-13T14:55:06.142902+0000","last_fullsized":"2024-09-13T14:55:06.142902+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":177,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.678778+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:54:58.678778+0000","last_peered":"2024-09-13T14:54:58.678778+0000","last_clean":"2024-09-13T14:54:58.678778+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:54:58.678778+0000","last_undegraded":"2024-09-13T14:54:58.678778+0000","last_fullsized":"2024-09-13T14:54:58.678778+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"264'557","reported_seq":576,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:09.800992+0000","last_change":"2024-09-13T14:54:52.112411+0000","last_active":"2024-09-13T14:55:09.800992+0000","last_peered":"2024-09-13T14:55:09.800992+0000","last_clean":"2024-09-13T14:55:09.800992+0000","last_became_active":"2024-09-13T14:54:52.112143+0000","last_became_peered":"2024-09-13T14:54:52.112143+0000","last_unstale":"2024-09-13T14:55:09.800992+0000","last_undegraded":"2024-09-13T14:55:09.800992+0000","last_fullsized":"2024-09-13T14:55:09.800992+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:43.729008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36503552,"num_objects":557,"num_object_clones":0,"num_object_copies":1671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":557,"num_write_kb":35648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":335,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143998+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:55:06.143998+0000","last_peered":"2024-09-13T14:55:06.143998+0000","last_clean":"2024-09-13T14:55:06.143998+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:55:06.143998+0000","last_undegraded":"2024-09-13T14:55:06.143998+0000","last_fullsized":"2024-09-13T14:55:06.143998+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":128,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998199+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:54:58.998199+0000","last_peered":"2024-09-13T14:54:58.998199+0000","last_clean":"2024-09-13T14:54:58.998199+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:54:58.998199+0000","last_undegraded":"2024-09-13T14:54:58.998199+0000","last_fullsized":"2024-09-13T14:54:58.998199+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"264'645","reported_seq":664,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:10.463233+0000","last_change":"2024-09-13T14:54:52.111899+0000","last_active":"2024-09-13T14:55:10.463233+0000","last_peered":"2024-09-13T14:55:10.463233+0000","last_clean":"2024-09-13T14:55:10.463233+0000","last_became_active":"2024-09-13T14:54:52.110464+0000","last_became_peered":"2024-09-13T14:54:52.110464+0000","last_unstale":"2024-09-13T14:55:10.463233+0000","last_undegraded":"2024-09-13T14:55:10.463233+0000","last_fullsized":"2024-09-13T14:55:10.463233+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:31.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42270720,"num_objects":645,"num_object_clones":0,"num_object_copies":1935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":645,"num_write_kb":41280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":129,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.010629+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:54:59.010629+0000","last_peered":"2024-09-13T14:54:59.010629+0000","last_clean":"2024-09-13T14:54:59.010629+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:54:59.010629+0000","last_undegraded":"2024-09-13T14:54:59.010629+0000","last_fullsized":"2024-09-13T14:54:59.010629+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":281,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.011045+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:54:59.011045+0000","last_peered":"2024-09-13T14:54:59.011045+0000","last_clean":"2024-09-13T14:54:59.011045+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:54:59.011045+0000","last_undegraded":"2024-09-13T14:54:59.011045+0000","last_fullsized":"2024-09-13T14:54:59.011045+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":19,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127167+0000","last_change":"2024-09-13T14:54:52.113408+0000","last_active":"2024-09-13T14:55:06.127167+0000","last_peered":"2024-09-13T14:55:06.127167+0000","last_clean":"2024-09-13T14:55:06.127167+0000","last_became_active":"2024-09-13T14:54:52.113069+0000","last_became_peered":"2024-09-13T14:54:52.113069+0000","last_unstale":"2024-09-13T14:55:06.127167+0000","last_undegraded":"2024-09-13T14:55:06.127167+0000","last_fullsized":"2024-09-13T14:55:06.127167+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:02.699612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":105,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143090+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:55:06.143090+0000","last_peered":"2024-09-13T14:55:06.143090+0000","last_clean":"2024-09-13T14:55:06.143090+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:55:06.143090+0000","last_undegraded":"2024-09-13T14:55:06.143090+0000","last_fullsized":"2024-09-13T14:55:06.143090+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":61,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.925116+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:54:58.925116+0000","last_peered":"2024-09-13T14:54:58.925116+0000","last_clean":"2024-09-13T14:54:58.925116+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:54:58.925116+0000","last_undegraded":"2024-09-13T14:54:58.925116+0000","last_fullsized":"2024-09-13T14:54:58.925116+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"264'446","reported_seq":464,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:09.312794+0000","last_change":"2024-09-13T14:54:52.116627+0000","last_active":"2024-09-13T14:55:09.312794+0000","last_peered":"2024-09-13T14:55:09.312794+0000","last_clean":"2024-09-13T14:55:09.312794+0000","last_became_active":"2024-09-13T14:54:52.116334+0000","last_became_peered":"2024-09-13T14:54:52.116334+0000","last_unstale":"2024-09-13T14:55:09.312794+0000","last_undegraded":"2024-09-13T14:55:09.312794+0000","last_fullsized":"2024-09-13T14:55:09.312794+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":446,"log_dups_size":0,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:25.091347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":446,"num_write_kb":28544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":117,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142913+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:55:06.142913+0000","last_peered":"2024-09-13T14:55:06.142913+0000","last_clean":"2024-09-13T14:55:06.142913+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:55:06.142913+0000","last_undegraded":"2024-09-13T14:55:06.142913+0000","last_fullsized":"2024-09-13T14:55:06.142913+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":379,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127407+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:55:06.127407+0000","last_peered":"2024-09-13T14:55:06.127407+0000","last_clean":"2024-09-13T14:55:06.127407+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:55:06.127407+0000","last_undegraded":"2024-09-13T14:55:06.127407+0000","last_fullsized":"2024-09-13T14:55:06.127407+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":19,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127598+0000","last_change":"2024-09-13T14:54:52.117749+0000","last_active":"2024-09-13T14:55:06.127598+0000","last_peered":"2024-09-13T14:55:06.127598+0000","last_clean":"2024-09-13T14:55:06.127598+0000","last_became_active":"2024-09-13T14:54:52.117446+0000","last_became_peered":"2024-09-13T14:54:52.117446+0000","last_unstale":"2024-09-13T14:55:06.127598+0000","last_undegraded":"2024-09-13T14:55:06.127598+0000","last_fullsized":"2024-09-13T14:55:06.127598+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:34.278909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":388,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143781+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:55:06.143781+0000","last_peered":"2024-09-13T14:55:06.143781+0000","last_clean":"2024-09-13T14:55:06.143781+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:55:06.143781+0000","last_undegraded":"2024-09-13T14:55:06.143781+0000","last_fullsized":"2024-09-13T14:55:06.143781+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":247,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.011181+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:54:59.011181+0000","last_peered":"2024-09-13T14:54:59.011181+0000","last_clean":"2024-09-13T14:54:59.011181+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:54:59.011181+0000","last_undegraded":"2024-09-13T14:54:59.011181+0000","last_fullsized":"2024-09-13T14:54:59.011181+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"264'465","reported_seq":484,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:09.386183+0000","last_change":"2024-09-13T14:54:52.113650+0000","last_active":"2024-09-13T14:55:09.386183+0000","last_peered":"2024-09-13T14:55:09.386183+0000","last_clean":"2024-09-13T14:55:09.386183+0000","last_became_active":"2024-09-13T14:54:52.113338+0000","last_became_peered":"2024-09-13T14:54:52.113338+0000","last_unstale":"2024-09-13T14:55:09.386183+0000","last_undegraded":"2024-09-13T14:55:09.386183+0000","last_fullsized":"2024-09-13T14:55:09.386183+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:53.903772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30474240,"num_objects":465,"num_object_clones":0,"num_object_copies":1395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":465,"num_write_kb":29760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":148,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.678960+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:54:58.678960+0000","last_peered":"2024-09-13T14:54:58.678960+0000","last_clean":"2024-09-13T14:54:58.678960+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:54:58.678960+0000","last_undegraded":"2024-09-13T14:54:58.678960+0000","last_fullsized":"2024-09-13T14:54:58.678960+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":244,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.011303+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:54:59.011303+0000","last_peered":"2024-09-13T14:54:59.011303+0000","last_clean":"2024-09-13T14:54:59.011303+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:54:59.011303+0000","last_undegraded":"2024-09-13T14:54:59.011303+0000","last_fullsized":"2024-09-13T14:54:59.011303+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"264'407","reported_seq":420,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:08.916360+0000","last_change":"2024-09-13T14:54:52.113259+0000","last_active":"2024-09-13T14:55:08.916360+0000","last_peered":"2024-09-13T14:55:08.916360+0000","last_clean":"2024-09-13T14:55:08.916360+0000","last_became_active":"2024-09-13T14:54:52.112910+0000","last_became_peered":"2024-09-13T14:54:52.112910+0000","last_unstale":"2024-09-13T14:55:08.916360+0000","last_undegraded":"2024-09-13T14:55:08.916360+0000","last_fullsized":"2024-09-13T14:55:08.916360+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:37.906148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26673152,"num_objects":407,"num_object_clones":0,"num_object_copies":1221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":407,"num_write_kb":26048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":240,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.010646+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:54:59.010646+0000","last_peered":"2024-09-13T14:54:59.010646+0000","last_clean":"2024-09-13T14:54:59.010646+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:54:59.010646+0000","last_undegraded":"2024-09-13T14:54:59.010646+0000","last_fullsized":"2024-09-13T14:54:59.010646+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":274,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.011455+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:54:59.011455+0000","last_peered":"2024-09-13T14:54:59.011455+0000","last_clean":"2024-09-13T14:54:59.011455+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:54:59.011455+0000","last_undegraded":"2024-09-13T14:54:59.011455+0000","last_fullsized":"2024-09-13T14:54:59.011455+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"264'397","reported_seq":410,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:08.912639+0000","last_change":"2024-09-13T14:54:52.113761+0000","last_active":"2024-09-13T14:55:08.912639+0000","last_peered":"2024-09-13T14:55:08.912639+0000","last_clean":"2024-09-13T14:55:08.912639+0000","last_became_active":"2024-09-13T14:54:52.113441+0000","last_became_peered":"2024-09-13T14:54:52.113441+0000","last_unstale":"2024-09-13T14:55:08.912639+0000","last_undegraded":"2024-09-13T14:55:08.912639+0000","last_fullsized":"2024-09-13T14:55:08.912639+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:01.375020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":397,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":126,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142229+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:55:06.142229+0000","last_peered":"2024-09-13T14:55:06.142229+0000","last_clean":"2024-09-13T14:55:06.142229+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:55:06.142229+0000","last_undegraded":"2024-09-13T14:55:06.142229+0000","last_fullsized":"2024-09-13T14:55:06.142229+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":235,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.144007+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:55:06.144007+0000","last_peered":"2024-09-13T14:55:06.144007+0000","last_clean":"2024-09-13T14:55:06.144007+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:55:06.144007+0000","last_undegraded":"2024-09-13T14:55:06.144007+0000","last_fullsized":"2024-09-13T14:55:06.144007+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":19,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127537+0000","last_change":"2024-09-13T14:54:52.116764+0000","last_active":"2024-09-13T14:55:06.127537+0000","last_peered":"2024-09-13T14:55:06.127537+0000","last_clean":"2024-09-13T14:55:06.127537+0000","last_became_active":"2024-09-13T14:54:52.116144+0000","last_became_peered":"2024-09-13T14:54:52.116144+0000","last_unstale":"2024-09-13T14:55:06.127537+0000","last_undegraded":"2024-09-13T14:55:06.127537+0000","last_fullsized":"2024-09-13T14:55:06.127537+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:37.832100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":582,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127159+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:55:06.127159+0000","last_peered":"2024-09-13T14:55:06.127159+0000","last_clean":"2024-09-13T14:55:06.127159+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:55:06.127159+0000","last_undegraded":"2024-09-13T14:55:06.127159+0000","last_fullsized":"2024-09-13T14:55:06.127159+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":832,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142832+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:55:06.142832+0000","last_peered":"2024-09-13T14:55:06.142832+0000","last_clean":"2024-09-13T14:55:06.142832+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:55:06.142832+0000","last_undegraded":"2024-09-13T14:55:06.142832+0000","last_fullsized":"2024-09-13T14:55:06.142832+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":346,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142824+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:55:06.142824+0000","last_peered":"2024-09-13T14:55:06.142824+0000","last_clean":"2024-09-13T14:55:06.142824+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:55:06.142824+0000","last_undegraded":"2024-09-13T14:55:06.142824+0000","last_fullsized":"2024-09-13T14:55:06.142824+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":61,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.679193+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:54:58.679193+0000","last_peered":"2024-09-13T14:54:58.679193+0000","last_clean":"2024-09-13T14:54:58.679193+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:54:58.679193+0000","last_undegraded":"2024-09-13T14:54:58.679193+0000","last_fullsized":"2024-09-13T14:54:58.679193+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"264'416","reported_seq":435,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:09.299627+0000","last_change":"2024-09-13T14:54:52.115847+0000","last_active":"2024-09-13T14:55:09.299627+0000","last_peered":"2024-09-13T14:55:09.299627+0000","last_clean":"2024-09-13T14:55:09.299627+0000","last_became_active":"2024-09-13T14:54:52.115534+0000","last_became_peered":"2024-09-13T14:54:52.115534+0000","last_unstale":"2024-09-13T14:55:09.299627+0000","last_undegraded":"2024-09-13T14:55:09.299627+0000","last_fullsized":"2024-09-13T14:55:09.299627+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:05.318760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":427,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142920+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:55:06.142920+0000","last_peered":"2024-09-13T14:55:06.142920+0000","last_clean":"2024-09-13T14:55:06.142920+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:55:06.142920+0000","last_undegraded":"2024-09-13T14:55:06.142920+0000","last_fullsized":"2024-09-13T14:55:06.142920+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":394,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127146+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:55:06.127146+0000","last_peered":"2024-09-13T14:55:06.127146+0000","last_clean":"2024-09-13T14:55:06.127146+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:55:06.127146+0000","last_undegraded":"2024-09-13T14:55:06.127146+0000","last_fullsized":"2024-09-13T14:55:06.127146+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":404,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127746+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:55:06.127746+0000","last_peered":"2024-09-13T14:55:06.127746+0000","last_clean":"2024-09-13T14:55:06.127746+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:55:06.127746+0000","last_undegraded":"2024-09-13T14:55:06.127746+0000","last_fullsized":"2024-09-13T14:55:06.127746+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":19,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127414+0000","last_change":"2024-09-13T14:54:52.114072+0000","last_active":"2024-09-13T14:55:06.127414+0000","last_peered":"2024-09-13T14:55:06.127414+0000","last_clean":"2024-09-13T14:55:06.127414+0000","last_became_active":"2024-09-13T14:54:52.113758+0000","last_became_peered":"2024-09-13T14:54:52.113758+0000","last_unstale":"2024-09-13T14:55:06.127414+0000","last_undegraded":"2024-09-13T14:55:06.127414+0000","last_fullsized":"2024-09-13T14:55:06.127414+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:07.961914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":930,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142639+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:55:06.142639+0000","last_peered":"2024-09-13T14:55:06.142639+0000","last_clean":"2024-09-13T14:55:06.142639+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:55:06.142639+0000","last_undegraded":"2024-09-13T14:55:06.142639+0000","last_fullsized":"2024-09-13T14:55:06.142639+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":385,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143173+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:55:06.143173+0000","last_peered":"2024-09-13T14:55:06.143173+0000","last_clean":"2024-09-13T14:55:06.143173+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:55:06.143173+0000","last_undegraded":"2024-09-13T14:55:06.143173+0000","last_fullsized":"2024-09-13T14:55:06.143173+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":138,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143848+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:55:06.143848+0000","last_peered":"2024-09-13T14:55:06.143848+0000","last_clean":"2024-09-13T14:55:06.143848+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:55:06.143848+0000","last_undegraded":"2024-09-13T14:55:06.143848+0000","last_fullsized":"2024-09-13T14:55:06.143848+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"264'414","reported_seq":427,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:08.909432+0000","last_change":"2024-09-13T14:54:52.113753+0000","last_active":"2024-09-13T14:55:08.909432+0000","last_peered":"2024-09-13T14:55:08.909432+0000","last_clean":"2024-09-13T14:55:08.909432+0000","last_became_active":"2024-09-13T14:54:52.113413+0000","last_became_peered":"2024-09-13T14:54:52.113413+0000","last_unstale":"2024-09-13T14:55:08.909432+0000","last_undegraded":"2024-09-13T14:55:08.909432+0000","last_fullsized":"2024-09-13T14:55:08.909432+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:42.771873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":414,"num_write_kb":26496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":119,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998326+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:54:58.998326+0000","last_peered":"2024-09-13T14:54:58.998326+0000","last_clean":"2024-09-13T14:54:58.998326+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:54:58.998326+0000","last_undegraded":"2024-09-13T14:54:58.998326+0000","last_fullsized":"2024-09-13T14:54:58.998326+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":249,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.011298+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:54:59.011298+0000","last_peered":"2024-09-13T14:54:59.011298+0000","last_clean":"2024-09-13T14:54:59.011298+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:54:59.011298+0000","last_undegraded":"2024-09-13T14:54:59.011298+0000","last_fullsized":"2024-09-13T14:54:59.011298+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":28,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998004+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:54:58.998004+0000","last_peered":"2024-09-13T14:54:58.998004+0000","last_clean":"2024-09-13T14:54:58.998004+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:54:58.998004+0000","last_undegraded":"2024-09-13T14:54:58.998004+0000","last_fullsized":"2024-09-13T14:54:58.998004+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"264'475","reported_seq":488,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:09.227966+0000","last_change":"2024-09-13T14:54:52.107585+0000","last_active":"2024-09-13T14:55:09.227966+0000","last_peered":"2024-09-13T14:55:09.227966+0000","last_clean":"2024-09-13T14:55:09.227966+0000","last_became_active":"2024-09-13T14:54:52.106845+0000","last_became_peered":"2024-09-13T14:54:52.106845+0000","last_unstale":"2024-09-13T14:55:09.227966+0000","last_undegraded":"2024-09-13T14:55:09.227966+0000","last_fullsized":"2024-09-13T14:55:09.227966+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:32.999267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":475,"num_write_kb":30400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":119,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998405+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:54:58.998405+0000","last_peered":"2024-09-13T14:54:58.998405+0000","last_clean":"2024-09-13T14:54:58.998405+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:54:58.998405+0000","last_undegraded":"2024-09-13T14:54:58.998405+0000","last_fullsized":"2024-09-13T14:54:58.998405+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":940,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142952+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:55:06.142952+0000","last_peered":"2024-09-13T14:55:06.142952+0000","last_clean":"2024-09-13T14:55:06.142952+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:55:06.142952+0000","last_undegraded":"2024-09-13T14:55:06.142952+0000","last_fullsized":"2024-09-13T14:55:06.142952+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":46,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127169+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:55:06.127169+0000","last_peered":"2024-09-13T14:55:06.127169+0000","last_clean":"2024-09-13T14:55:06.127169+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:55:06.127169+0000","last_undegraded":"2024-09-13T14:55:06.127169+0000","last_fullsized":"2024-09-13T14:55:06.127169+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"264'491","reported_seq":504,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:09.618899+0000","last_change":"2024-09-13T14:54:52.114535+0000","last_active":"2024-09-13T14:55:09.618899+0000","last_peered":"2024-09-13T14:55:09.618899+0000","last_clean":"2024-09-13T14:55:09.618899+0000","last_became_active":"2024-09-13T14:54:52.114212+0000","last_became_peered":"2024-09-13T14:54:52.114212+0000","last_unstale":"2024-09-13T14:55:09.618899+0000","last_undegraded":"2024-09-13T14:55:09.618899+0000","last_fullsized":"2024-09-13T14:55:09.618899+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":491,"log_dups_size":0,"ondisk_log_size":491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:31.266521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":31424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":392,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142857+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:55:06.142857+0000","last_peered":"2024-09-13T14:55:06.142857+0000","last_clean":"2024-09-13T14:55:06.142857+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:55:06.142857+0000","last_undegraded":"2024-09-13T14:55:06.142857+0000","last_fullsized":"2024-09-13T14:55:06.142857+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":613,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127011+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:55:06.127011+0000","last_peered":"2024-09-13T14:55:06.127011+0000","last_clean":"2024-09-13T14:55:06.127011+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:55:06.127011+0000","last_undegraded":"2024-09-13T14:55:06.127011+0000","last_fullsized":"2024-09-13T14:55:06.127011+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":390,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127355+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:55:06.127355+0000","last_peered":"2024-09-13T14:55:06.127355+0000","last_clean":"2024-09-13T14:55:06.127355+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:55:06.127355+0000","last_undegraded":"2024-09-13T14:55:06.127355+0000","last_fullsized":"2024-09-13T14:55:06.127355+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"264'591","reported_seq":610,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:10.457051+0000","last_change":"2024-09-13T14:54:52.112337+0000","last_active":"2024-09-13T14:55:10.457051+0000","last_peered":"2024-09-13T14:55:10.457051+0000","last_clean":"2024-09-13T14:55:10.457051+0000","last_became_active":"2024-09-13T14:54:52.112057+0000","last_became_peered":"2024-09-13T14:54:52.112057+0000","last_unstale":"2024-09-13T14:55:10.457051+0000","last_undegraded":"2024-09-13T14:55:10.457051+0000","last_fullsized":"2024-09-13T14:55:10.457051+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:13.030605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38731776,"num_objects":591,"num_object_clones":0,"num_object_copies":1773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":591,"num_write_kb":37824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":410,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143153+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:55:06.143153+0000","last_peered":"2024-09-13T14:55:06.143153+0000","last_clean":"2024-09-13T14:55:06.143153+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:55:06.143153+0000","last_undegraded":"2024-09-13T14:55:06.143153+0000","last_fullsized":"2024-09-13T14:55:06.143153+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":841,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.144109+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:55:06.144109+0000","last_peered":"2024-09-13T14:55:06.144109+0000","last_clean":"2024-09-13T14:55:06.144109+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:55:06.144109+0000","last_undegraded":"2024-09-13T14:55:06.144109+0000","last_fullsized":"2024-09-13T14:55:06.144109+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":126,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142132+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:55:06.142132+0000","last_peered":"2024-09-13T14:55:06.142132+0000","last_clean":"2024-09-13T14:55:06.142132+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:55:06.142132+0000","last_undegraded":"2024-09-13T14:55:06.142132+0000","last_fullsized":"2024-09-13T14:55:06.142132+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"264'640","reported_seq":659,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:10.461985+0000","last_change":"2024-09-13T14:54:52.112510+0000","last_active":"2024-09-13T14:55:10.461985+0000","last_peered":"2024-09-13T14:55:10.461985+0000","last_clean":"2024-09-13T14:55:10.461985+0000","last_became_active":"2024-09-13T14:54:52.112262+0000","last_became_peered":"2024-09-13T14:54:52.112262+0000","last_unstale":"2024-09-13T14:55:10.461985+0000","last_undegraded":"2024-09-13T14:55:10.461985+0000","last_fullsized":"2024-09-13T14:55:10.461985+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:57.884110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":640,"num_object_clones":0,"num_object_copies":1920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":384,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127535+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:55:06.127535+0000","last_peered":"2024-09-13T14:55:06.127535+0000","last_clean":"2024-09-13T14:55:06.127535+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:55:06.127535+0000","last_undegraded":"2024-09-13T14:55:06.127535+0000","last_fullsized":"2024-09-13T14:55:06.127535+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":817,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.925193+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:54:58.925193+0000","last_peered":"2024-09-13T14:54:58.925193+0000","last_clean":"2024-09-13T14:54:58.925193+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:54:58.925193+0000","last_undegraded":"2024-09-13T14:54:58.925193+0000","last_fullsized":"2024-09-13T14:54:58.925193+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":251,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.010946+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:54:59.010946+0000","last_peered":"2024-09-13T14:54:59.010946+0000","last_clean":"2024-09-13T14:54:59.010946+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:54:59.010946+0000","last_undegraded":"2024-09-13T14:54:59.010946+0000","last_fullsized":"2024-09-13T14:54:59.010946+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":247,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.011220+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:54:59.011220+0000","last_peered":"2024-09-13T14:54:59.011220+0000","last_clean":"2024-09-13T14:54:59.011220+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:54:59.011220+0000","last_undegraded":"2024-09-13T14:54:59.011220+0000","last_fullsized":"2024-09-13T14:54:59.011220+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":295,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.011289+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:54:59.011289+0000","last_peered":"2024-09-13T14:54:59.011289+0000","last_clean":"2024-09-13T14:54:59.011289+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:54:59.011289+0000","last_undegraded":"2024-09-13T14:54:59.011289+0000","last_fullsized":"2024-09-13T14:54:59.011289+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":389,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142707+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:55:06.142707+0000","last_peered":"2024-09-13T14:55:06.142707+0000","last_clean":"2024-09-13T14:55:06.142707+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:55:06.142707+0000","last_undegraded":"2024-09-13T14:55:06.142707+0000","last_fullsized":"2024-09-13T14:55:06.142707+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":229,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142083+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:55:06.142083+0000","last_peered":"2024-09-13T14:55:06.142083+0000","last_clean":"2024-09-13T14:55:06.142083+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:55:06.142083+0000","last_undegraded":"2024-09-13T14:55:06.142083+0000","last_fullsized":"2024-09-13T14:55:06.142083+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":127,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998573+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:54:58.998573+0000","last_peered":"2024-09-13T14:54:58.998573+0000","last_clean":"2024-09-13T14:54:58.998573+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:54:58.998573+0000","last_undegraded":"2024-09-13T14:54:58.998573+0000","last_fullsized":"2024-09-13T14:54:58.998573+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":366,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142349+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:55:06.142349+0000","last_peered":"2024-09-13T14:55:06.142349+0000","last_clean":"2024-09-13T14:55:06.142349+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:55:06.142349+0000","last_undegraded":"2024-09-13T14:55:06.142349+0000","last_fullsized":"2024-09-13T14:55:06.142349+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":421,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127651+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:55:06.127651+0000","last_peered":"2024-09-13T14:55:06.127651+0000","last_clean":"2024-09-13T14:55:06.127651+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:55:06.127651+0000","last_undegraded":"2024-09-13T14:55:06.127651+0000","last_fullsized":"2024-09-13T14:55:06.127651+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":417,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142520+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:55:06.142520+0000","last_peered":"2024-09-13T14:55:06.142520+0000","last_clean":"2024-09-13T14:55:06.142520+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:55:06.142520+0000","last_undegraded":"2024-09-13T14:55:06.142520+0000","last_fullsized":"2024-09-13T14:55:06.142520+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":157,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.679167+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:54:58.679167+0000","last_peered":"2024-09-13T14:54:58.679167+0000","last_clean":"2024-09-13T14:54:58.679167+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:54:58.679167+0000","last_undegraded":"2024-09-13T14:54:58.679167+0000","last_fullsized":"2024-09-13T14:54:58.679167+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":384,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142608+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:55:06.142608+0000","last_peered":"2024-09-13T14:55:06.142608+0000","last_clean":"2024-09-13T14:55:06.142608+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:55:06.142608+0000","last_undegraded":"2024-09-13T14:55:06.142608+0000","last_fullsized":"2024-09-13T14:55:06.142608+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":401,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127699+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:55:06.127699+0000","last_peered":"2024-09-13T14:55:06.127699+0000","last_clean":"2024-09-13T14:55:06.127699+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:55:06.127699+0000","last_undegraded":"2024-09-13T14:55:06.127699+0000","last_fullsized":"2024-09-13T14:55:06.127699+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":369,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143387+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:55:06.143387+0000","last_peered":"2024-09-13T14:55:06.143387+0000","last_clean":"2024-09-13T14:55:06.143387+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:55:06.143387+0000","last_undegraded":"2024-09-13T14:55:06.143387+0000","last_fullsized":"2024-09-13T14:55:06.143387+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":233,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143379+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:55:06.143379+0000","last_peered":"2024-09-13T14:55:06.143379+0000","last_clean":"2024-09-13T14:55:06.143379+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:55:06.143379+0000","last_undegraded":"2024-09-13T14:55:06.143379+0000","last_fullsized":"2024-09-13T14:55:06.143379+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":245,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.010967+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:54:59.010967+0000","last_peered":"2024-09-13T14:54:59.010967+0000","last_clean":"2024-09-13T14:54:59.010967+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:54:59.010967+0000","last_undegraded":"2024-09-13T14:54:59.010967+0000","last_fullsized":"2024-09-13T14:54:59.010967+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":129,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143288+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:55:06.143288+0000","last_peered":"2024-09-13T14:55:06.143288+0000","last_clean":"2024-09-13T14:55:06.143288+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:55:06.143288+0000","last_undegraded":"2024-09-13T14:55:06.143288+0000","last_fullsized":"2024-09-13T14:55:06.143288+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":394,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127320+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:55:06.127320+0000","last_peered":"2024-09-13T14:55:06.127320+0000","last_clean":"2024-09-13T14:55:06.127320+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:55:06.127320+0000","last_undegraded":"2024-09-13T14:55:06.127320+0000","last_fullsized":"2024-09-13T14:55:06.127320+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.010586+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:54:59.010586+0000","last_peered":"2024-09-13T14:54:59.010586+0000","last_clean":"2024-09-13T14:54:59.010586+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:54:59.010586+0000","last_undegraded":"2024-09-13T14:54:59.010586+0000","last_fullsized":"2024-09-13T14:54:59.010586+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":102,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998390+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:54:58.998390+0000","last_peered":"2024-09-13T14:54:58.998390+0000","last_clean":"2024-09-13T14:54:58.998390+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:54:58.998390+0000","last_undegraded":"2024-09-13T14:54:58.998390+0000","last_fullsized":"2024-09-13T14:54:58.998390+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":243,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.010856+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:54:59.010856+0000","last_peered":"2024-09-13T14:54:59.010856+0000","last_clean":"2024-09-13T14:54:59.010856+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:54:59.010856+0000","last_undegraded":"2024-09-13T14:54:59.010856+0000","last_fullsized":"2024-09-13T14:54:59.010856+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":104,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:58.998509+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:54:58.998509+0000","last_peered":"2024-09-13T14:54:58.998509+0000","last_clean":"2024-09-13T14:54:58.998509+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:54:58.998509+0000","last_undegraded":"2024-09-13T14:54:58.998509+0000","last_fullsized":"2024-09-13T14:54:58.998509+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":387,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.142663+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:55:06.142663+0000","last_peered":"2024-09-13T14:55:06.142663+0000","last_clean":"2024-09-13T14:55:06.142663+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:55:06.142663+0000","last_undegraded":"2024-09-13T14:55:06.142663+0000","last_fullsized":"2024-09-13T14:55:06.142663+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":404,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.143257+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:55:06.143257+0000","last_peered":"2024-09-13T14:55:06.143257+0000","last_clean":"2024-09-13T14:55:06.143257+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:55:06.143257+0000","last_undegraded":"2024-09-13T14:55:06.143257+0000","last_fullsized":"2024-09-13T14:55:06.143257+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":274,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-13T14:54:59.010723+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:54:59.010723+0000","last_peered":"2024-09-13T14:54:59.010723+0000","last_clean":"2024-09-13T14:54:59.010723+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:54:59.010723+0000","last_undegraded":"2024-09-13T14:54:59.010723+0000","last_fullsized":"2024-09-13T14:54:59.010723+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":457,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-09-13T14:55:06.127587+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:55:06.127587+0000","last_peered":"2024-09-13T14:55:06.127587+0000","last_clean":"2024-09-13T14:55:06.127587+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:55:06.127587+0000","last_undegraded":"2024-09-13T14:55:06.127587+0000","last_fullsized":"2024-09-13T14:55:06.127587+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5944,"num_write_kb":380416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1342898176,"data_stored":1342898176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5944,"ondisk_log_size":5944,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234333,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495969,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":445524,"kb_used_data":93640,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93299628,"statfs":{"total":95995035648,"available":95538819072,"internally_reserved":0,"allocated":95887360,"data_stored":94476178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757604,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":340420,"kb_used_data":132924,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93404732,"statfs":{"total":95995035648,"available":95646445568,"internally_reserved":0,"allocated":136114176,"data_stored":134710861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":212466119},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019244,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":459920,"kb_used_data":158088,"kb_used_omap":1,"kb_used_meta":301822,"kb_avail":93285232,"statfs":{"total":95995035648,"available":95524077568,"internally_reserved":0,"allocated":161882112,"data_stored":160470991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309066182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468129,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":670468,"kb_used_data":368124,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93074684,"statfs":{"total":95995035648,"available":95308476416,"internally_reserved":0,"allocated":376958976,"data_stored":375555435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":309587108},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":547668,"kb_used_data":141132,"kb_used_omap":8,"kb_used_meta":406519,"kb_avail":93197484,"statfs":{"total":95995035648,"available":95434223616,"internally_reserved":0,"allocated":144519168,"data_stored":143104477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":416276447},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795181,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":729288,"kb_used_data":408832,"kb_used_omap":7,"kb_used_meta":320440,"kb_avail":93015864,"statfs":{"total":95995035648,"available":95248244736,"internally_reserved":0,"allocated":418643968,"data_stored":417236756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":328131186},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958711,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292292,"kb_used_data":7928,"kb_used_omap":7,"kb_used_meta":284344,"kb_avail":93452860,"statfs":{"total":95995035648,"available":95695728640,"internally_reserved":0,"allocated":8118272,"data_stored":6711250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":291168743},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249298944,"data_stored":249298944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212402176,"data_stored":212402176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":404160512,"data_stored":404160512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":175374336,"data_stored":175374336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":168230912,"data_stored":168230912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":133431296,"data_stored":133431296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:55:13.800 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:55:13.800 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:55:13.801 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-13T14:55:13.801 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:55:13.801 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:55:13.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-13T14:55:13.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:13.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 7 16 17671 17655 157.607 134.562 0.00720466 0.00633775 2024-09-13T14:55:13.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:13.968 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:13.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:13.971 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:55:14.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:14.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:14.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:14.225 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:14.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:14.333 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[102314]: pgmap v808: 89 pgs: 89 active+clean; 577 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-13T14:55:14.333 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:14.333 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:55:14.333 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3035220272' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:14.333 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[102314]: osdmap e265: 8 total, 8 up, 7 in 2024-09-13T14:55:14.334 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[107038]: pgmap v808: 89 pgs: 89 active+clean; 577 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-13T14:55:14.334 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:14.334 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:55:14.334 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3035220272' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:14.334 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:14 smithi154 ceph-mon[107038]: osdmap e265: 8 total, 8 up, 7 in 2024-09-13T14:55:14.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:14.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:14.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:14 smithi172 ceph-mon[100696]: pgmap v808: 89 pgs: 89 active+clean; 577 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-13T14:55:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:14 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:55:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:14 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:55:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3035220272' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:14 smithi172 ceph-mon[100696]: osdmap e265: 8 total, 8 up, 7 in 2024-09-13T14:55:14.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:14.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:14.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:14.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:14.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:14.885 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 8 16 19930 19914 155.553 141.188 0.00399533 0.00642234 2024-09-13T14:55:14.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:14.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:15.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:15.027 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:15.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:15.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:15.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:15.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:15.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:15.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.421 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:15.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:15.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:15.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:15.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:15.680 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:15.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:15.810 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:15.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 9 16 22019 22003 152.775 130.562 0.00535939 0.00653909 2024-09-13T14:55:15.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:15.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:15.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:16.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:16.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:16.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:16.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:16.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.350 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:16.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:16.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:16 smithi154 ceph-mon[102314]: pgmap v810: 89 pgs: 89 active+clean; 972 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-13T14:55:16.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:16 smithi154 ceph-mon[107038]: pgmap v810: 89 pgs: 89 active+clean; 972 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-13T14:55:16.480 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:16.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.612 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:16.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:16.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:16.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:55:16.736 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:16 smithi172 ceph-mon[100696]: pgmap v810: 89 pgs: 89 active+clean; 972 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-13T14:55:16.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:16.866 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:16.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:16.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:16.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 10 16 23516 23500 146.854 93.5625 0.00636312 0.00680204 2024-09-13T14:55:16.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:17.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:17.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:17.410 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:17.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:17.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:17.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:17.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:17.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:17 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:17.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:17 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:17.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:17 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:17.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:17 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:17 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:17 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:17.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:17.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:17.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 11 16 25347 25331 143.906 114.438 0.00190118 0.00693205 2024-09-13T14:55:17.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:17.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:17.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:18.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:18.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:18.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:18.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:18.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:18.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:18.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:18.468 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:18.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:18.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:18.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:18 smithi154 ceph-mon[102314]: pgmap v811: 89 pgs: 89 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-09-13T14:55:18.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:18 smithi154 ceph-mon[107038]: pgmap v811: 89 pgs: 89 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-09-13T14:55:18.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:18 smithi172 ceph-mon[100696]: pgmap v811: 89 pgs: 89 active+clean; 1.2 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 126 MiB/s wr, 2.02k op/s 2024-09-13T14:55:18.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:18.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:18.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 12 16 27011 26995 140.58 104 0.00769605 0.00710751 2024-09-13T14:55:18.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:18.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:18.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:19.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:19.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:19.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:19.266 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:19.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:19.396 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:19.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:19.541 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:19.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:19.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:19.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:19.804 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:19.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 13 16 28454 28438 136.703 90.1875 0.00846653 0.00730954 2024-09-13T14:55:19.906 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:19.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:19.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:20.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:20.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:20.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:20.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:20.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:20.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:20.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:20.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:20.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:20.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:20.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:20 smithi154 ceph-mon[102314]: pgmap v812: 89 pgs: 89 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:55:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:20 smithi154 ceph-mon[107038]: pgmap v812: 89 pgs: 89 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:55:20.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:20 smithi172 ceph-mon[100696]: pgmap v812: 89 pgs: 89 active+clean; 1.4 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-09-13T14:55:20.751 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:20.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:20.877 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:20.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:20.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 14 16 30432 30416 135.768 123.625 0.0219976 0.00735581 2024-09-13T14:55:20.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:21.007 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:21.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:21.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:21.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:21.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:21.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:21.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:21.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:21.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:21.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:21.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:21.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:21.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:21.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:21.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 15 16 32171 32155 133.961 108.688 0.0107499 0.00745836 2024-09-13T14:55:21.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:22.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:22.011 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:22.058 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:55:22.058 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:55:22.058 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:55:22.058 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:55:22.058 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:55:22.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:22.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:22.209 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:55:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:55:21] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-13T14:55:22.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.278 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.410 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:22.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:22 smithi154 ceph-mon[102314]: pgmap v813: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T14:55:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:22 smithi154 ceph-mon[107038]: pgmap v813: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T14:55:22.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:22.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:22.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:22 smithi172 ceph-mon[100696]: pgmap v813: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T14:55:22.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:22.791 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:22.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 16 16 33941 33925 132.502 110.625 0.00725895 0.00754122 2024-09-13T14:55:22.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:22.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:22.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:23.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:23.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:23.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:23.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:23.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:23.349 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:23.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:55:23.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:23.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:55:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:23.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:23.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:23.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:23.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:23.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:23.886 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 17 16 35557 35541 130.649 101 0.00860047 0.00764812 2024-09-13T14:55:23.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:23.916 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:23.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:24.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:24.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:24.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:24.187 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:24.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:24.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:24.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:24.427 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:24.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:24.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:24.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:24.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:24 smithi154 ceph-mon[102314]: pgmap v814: 89 pgs: 89 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:55:24.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:24 smithi154 ceph-mon[107038]: pgmap v814: 89 pgs: 89 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:55:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:24 smithi172 ceph-mon[100696]: pgmap v814: 89 pgs: 89 active+clean; 1.8 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-13T14:55:24.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:24.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:24.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:24.887 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 18 16 37268 37252 129.331 106.938 0.0107897 0.00772574 2024-09-13T14:55:24.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:24.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:24.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:25.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.076 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:25.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:25.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:25.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:25.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:25.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:25.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:25.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:25.624 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:25.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.764 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:25.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:25.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:25.887 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 19 16 39007 38991 128.244 108.688 0.00840618 0.00779158 2024-09-13T14:55:25.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:25.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:26.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:26.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:26.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:26.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:26.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:26.306 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:26.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:26.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:26.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:26.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:26.593 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:26.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:26.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:26.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:26 smithi154 ceph-mon[102314]: pgmap v815: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-09-13T14:55:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:26 smithi154 ceph-mon[107038]: pgmap v815: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-09-13T14:55:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:26 smithi172 ceph-mon[100696]: pgmap v815: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-09-13T14:55:26.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:26.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:26.887 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:55:26.886585+0000 min lat: 0.000986333 max lat: 0.294749 avg lat: 0.00787758 2024-09-13T14:55:26.887 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:26.887 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 20 16 40296 40280 125.859 80.5625 0.162685 0.00787758 2024-09-13T14:55:26.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:26.962 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:26.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:27.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:27.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:27.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:27.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:27.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:27.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:27.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:27.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:27.612 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:27.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:27.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:27.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:27.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:27.887 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 21 16 41401 41385 123.154 69.0625 0.00650526 0.00811473 2024-09-13T14:55:27.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:27.997 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:27.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:28.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:28.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:28.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:28.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:28.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:28.405 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:28.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:28.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:28.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:28.665 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:28 smithi154 ceph-mon[102314]: pgmap v816: 89 pgs: 89 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-13T14:55:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:28 smithi154 ceph-mon[107038]: pgmap v816: 89 pgs: 89 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-13T14:55:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:28 smithi172 ceph-mon[100696]: pgmap v816: 89 pgs: 89 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-13T14:55:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:28.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:28.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:28.887 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 22 16 42787 42771 121.494 86.625 0.010032 0.00822482 2024-09-13T14:55:28.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:28.938 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:28.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:29.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:29.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:29.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:29.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:29.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:29.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:29.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:29.487 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:29.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:29.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:29.700 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:29.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:29.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:29.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:29.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 23 16 44314 44298 120.357 95.4375 0.0105359 0.00830276 2024-09-13T14:55:29.912 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:29.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:30.014 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:30.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:30.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:30.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:30.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:30.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:30.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:30.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:30.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:30.578 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:30.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:30 smithi154 ceph-mon[102314]: pgmap v817: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:55:30.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:30 smithi154 ceph-mon[107038]: pgmap v817: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:55:30.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:30.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:30 smithi172 ceph-mon[100696]: pgmap v817: 89 pgs: 89 active+clean; 2.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T14:55:30.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:30.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:30.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 24 16 45720 45704 119.004 87.875 0.00514031 0.00839677 2024-09-13T14:55:30.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:30.985 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:30.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:31.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:31.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:31.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:31.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:31.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:31.388 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:31.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:31.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:31.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:31.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:31.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:31.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:31.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 25 16 46876 46860 117.133 72.25 0.00801201 0.00853 2024-09-13T14:55:31.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:31.923 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:31.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:32.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:32.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:32.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:32.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:32.190 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:55:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:55:31] "GET /metrics HTTP/1.1" 200 36509 "" "Prometheus/2.51.0" 2024-09-13T14:55:32.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:32.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:32.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:32.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:32.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:32.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:32.648 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:32 smithi154 ceph-mon[102314]: pgmap v818: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:55:32.648 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:32 smithi154 ceph-mon[107038]: pgmap v818: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:55:32.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:32.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:32.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:32 smithi172 ceph-mon[100696]: pgmap v818: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-13T14:55:32.761 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:55:32.762 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:55:32.762 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:55:32.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:32.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:32.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:32.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 26 16 48121 48105 115.621 77.8125 0.0181626 0.00864333 2024-09-13T14:55:32.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:33.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:33.119 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:33.142 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:33.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:33.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:33.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:33.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:33.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:33.548 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:33.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:33.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:33.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:33.800 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:33.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 27 16 49227 49211 113.899 69.125 0.00610064 0.0087742 2024-09-13T14:55:33.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:33.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:33.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:34.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:34.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:34.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:34.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:34.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:34.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:34.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:34.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:34.467 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:34.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:34.597 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:34.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:34 smithi154 ceph-mon[102314]: pgmap v819: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T14:55:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:34 smithi154 ceph-mon[107038]: pgmap v819: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T14:55:34.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:34 smithi172 ceph-mon[100696]: pgmap v819: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T14:55:34.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:34.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:34.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 28 16 50730 50714 113.185 93.9375 0.00521248 0.00882881 2024-09-13T14:55:34.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:34.989 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:34.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:35.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:35.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:35.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:35.242 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:35.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:35.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:35.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:35.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:35.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:35.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:35.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:35.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:35.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 29 16 52145 52129 112.332 88.4375 0.00700852 0.00889642 2024-09-13T14:55:35.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:35.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:35.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:36.064 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:36.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:36.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:36.216 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:36.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:36.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:36.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:36.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:36 smithi154 ceph-mon[102314]: pgmap v820: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T14:55:36.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:36 smithi154 ceph-mon[107038]: pgmap v820: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T14:55:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.490 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:36.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:36.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:36.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:36.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:36 smithi172 ceph-mon[100696]: pgmap v820: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 99 MiB/s wr, 1.59k op/s 2024-09-13T14:55:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:36.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:36.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:36.879 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:36.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:36.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 30 16 53260 53244 110.91 69.6875 0.00828149 0.00901124 2024-09-13T14:55:36.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:37.008 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:37.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:37.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:37.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:37.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:37.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:37.405 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:37.463 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:55:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86673 6.4 6:24d0d169:::benchmark_data_smithi154_175640_object897:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.507975+0000", 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.951510525, 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.107262861999999, 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86673 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507975+0000", 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507975+0000", 2024-09-13T14:55:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507976+0000", 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.502e-06 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507981+0000", 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0769999999999998e-06 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507998+0000", 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6201999999999998e-05 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508005+0000", 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.379e-06 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508217+0000", 2024-09-13T14:55:37.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021226200000000001 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508230+0000", 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2646999999999999e-05 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.032888+0000", 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.524658221999999 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.032900+0000", 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1522000000000001e-05 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793624+0000", 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.76072437800000003 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793635+0000", 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1229999999999999e-05 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.513259+0000", 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7196241109999999 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.515815+0000", 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025560069999999999 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.612791+0000", 2024-09-13T14:55:37.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.096975483000000001 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.615238+0000", 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00244684 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86674 6.4 6:26d83999:::benchmark_data_smithi154_175640_object898:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:55:37.471 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95146838800002, 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.073532165000003, 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86674 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4999999999999998e-07 2024-09-13T14:55:37.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508020+0000", 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1670000000000002e-06 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508026+0000", 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0850000000000002e-06 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508030+0000", 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5229999999999997e-06 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508255+0000", 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022519099999999999 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508264+0000", 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3420000000000008e-06 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036648+0000", 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528384581000001 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036660+0000", 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1253999999999999e-05 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793661+0000", 2024-09-13T14:55:37.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75700164999999997 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793669+0000", 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.524233+0000", 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7305637449999995 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.528990+0000", 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047570019999999998 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.536276+0000", 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072860319999999996 2024-09-13T14:55:37.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.567951+0000", 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031675805000000001 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.569224+0000", 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012729099999999999 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.573311+0000", 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040869019999999999 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.575675+0000", 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023637990000000002 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.581549+0000", 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058741080000000003 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86702 6.4 6:2bd62bb7:::benchmark_data_smithi154_175640_object926:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95139860699999, 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.084438992000003, 2024-09-13T14:55:37.481 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86702 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.89e-07 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508090+0000", 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3240000000000001e-06 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508097+0000", 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3170000000000004e-06 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508103+0000", 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.314e-06 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508291+0000", 2024-09-13T14:55:37.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018816199999999999 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508300+0000", 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2199999999999992e-06 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036687+0000", 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528387514999999 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036695+0000", 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3590000000000008e-06 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793701+0000", 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75700578900000004 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793708+0000", 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3210000000000001e-06 2024-09-13T14:55:37.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.552107+0000", 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7583998209999994 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.556007+0000", 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038999799999999999 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.561750+0000", 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057424920000000001 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.572199+0000", 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010448756999999999 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.583202+0000", 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011003811 2024-09-13T14:55:37.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.584404+0000", 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012016279999999999 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.586113+0000", 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001708839 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.592526+0000", 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064129540000000002 2024-09-13T14:55:37.489 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86716 6.4 6:26c9dec4:::benchmark_data_smithi154_175640_object940:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95137548600002, 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.150778488999997, 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.490 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86716 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.22e-07 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508112+0000", 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6869999999999999e-06 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508117+0000", 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2780000000000001e-06 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508121+0000", 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.155e-06 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508328+0000", 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000206162 2024-09-13T14:55:37.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508336+0000", 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2819999999999996e-06 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036721+0000", 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528384908 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036742+0000", 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1185000000000001e-05 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793733+0000", 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75699062800000005 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793738+0000", 2024-09-13T14:55:37.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8939999999999998e-06 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.615597+0000", 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8218581789999995 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.619094+0000", 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003497432 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.624815+0000", 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057204880000000001 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.647410+0000", 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022595578000000002 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.648555+0000", 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001144897 2024-09-13T14:55:37.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.649148+0000", 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00059299699999999995 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.650609+0000", 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014611839999999999 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.658888+0000", 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082792330000000004 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86721 6.4 6:2594c329:::benchmark_data_smithi154_175640_object945:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508126+0000", 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95135899899998, 2024-09-13T14:55:37.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.183112059999999, 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86721 2024-09-13T14:55:37.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508126+0000", 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508126+0000", 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508127+0000", 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7300000000000002e-07 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508128+0000", 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4479999999999999e-06 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508135+0000", 2024-09-13T14:55:37.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5230000000000001e-06 2024-09-13T14:55:37.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508139+0000", 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2980000000000002e-06 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508364+0000", 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022497700000000001 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508372+0000", 2024-09-13T14:55:37.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2500000000000006e-06 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036772+0000", 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528400153 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036782+0000", 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7280000000000007e-06 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793773+0000", 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75699076899999995 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793780+0000", 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8539999999999999e-06 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.629719+0000", 2024-09-13T14:55:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8359395529999993 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.633166+0000", 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034471340000000001 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.640075+0000", 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069084089999999999 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.659963+0000", 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01988848 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.661225+0000", 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012620699999999999 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.679879+0000", 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018653508999999999 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.681788+0000", 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019092429999999999 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.691238+0000", 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009450389 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.509 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86742 6.4 6:271a548d:::benchmark_data_smithi154_175640_object966:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95134163400002, 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.218343629000003, 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86742 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7000000000000001e-07 2024-09-13T14:55:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508146+0000", 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6649999999999999e-06 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508151+0000", 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4600000000000002e-06 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508155+0000", 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1250000000000003e-06 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508399+0000", 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000243964 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508407+0000", 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9079999999999995e-06 2024-09-13T14:55:37.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036823+0000", 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528415599999999 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036831+0000", 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2940000000000006e-06 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793809+0000", 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75697761200000002 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793816+0000", 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8510000000000001e-06 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.661864+0000", 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8680481600000007 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.666377+0000", 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045136229999999996 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.672872+0000", 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064947770000000002 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.713116+0000", 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.040243565000000002 2024-09-13T14:55:37.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.714917+0000", 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001801265 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.715728+0000", 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081117899999999998 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.717401+0000", 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001672782 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.726487+0000", 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090865289999999994 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86748 6.4 6:218d5a01:::benchmark_data_smithi154_175640_object972:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:55:37.518 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95132542900001, 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.256619432000001, 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86748 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8200000000000001e-07 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508162+0000", 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6789999999999999e-06 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508167+0000", 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0690000000000001e-06 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508171+0000", 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0080000000000004e-06 2024-09-13T14:55:37.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508433+0000", 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026166600000000003 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508440+0000", 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.52e-06 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036906+0000", 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528465809 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036914+0000", 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7260000000000002e-06 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793841+0000", 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75692729299999995 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793847+0000", 2024-09-13T14:55:37.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:37.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4189999999999999e-06 2024-09-13T14:55:37.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.691732+0000", 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8978850040000008 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.696147+0000", 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044142169999999998 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.702810+0000", 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066635089999999998 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.727854+0000", 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025043894000000001 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.752826+0000", 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024971942 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.753744+0000", 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00091785 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.755632+0000", 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018883229999999999 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.764779+0000", 2024-09-13T14:55:37.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009147222 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86764 6.4 6:2ddc6785:::benchmark_data_smithi154_175640_object988:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.951309589, 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.304434260000001, 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.529 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86764 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:55:37.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9299999999999999e-07 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508178+0000", 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6339999999999999e-06 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508183+0000", 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.797e-06 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508187+0000", 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1169999999999997e-06 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508464+0000", 2024-09-13T14:55:37.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000277481 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508472+0000", 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1929999999999993e-06 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036953+0000", 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528480492 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036963+0000", 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0277e-05 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793881+0000", 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.756918337 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793888+0000", 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7190000000000004e-06 2024-09-13T14:55:37.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.729082+0000", 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9351940640000009 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.737635+0000", 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085531549999999998 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.744938+0000", 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007302411 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.766104+0000", 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021166570999999999 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.767670+0000", 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015657780000000001 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.800097+0000", 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032427072000000001 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.803677+0000", 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035797239999999998 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.812610+0000", 2024-09-13T14:55:37.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008933145 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86766 6.4 6:20e2c325:::benchmark_data_smithi154_175640_object990:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95119585800001, 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.350745707000002, 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.538 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86766 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:55:37.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9400000000000001e-07 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508292+0000", 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.215e-06 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508299+0000", 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5699999999999998e-06 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508304+0000", 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9840000000000001e-06 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508500+0000", 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000195916 2024-09-13T14:55:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508508+0000", 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.191e-06 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037024+0000", 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528515762000001 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037031+0000", 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6879999999999999e-06 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793912+0000", 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75688088399999998 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793919+0000", 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6259999999999997e-06 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.771094+0000", 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9771756180000004 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.776563+0000", 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054689760000000004 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.785532+0000", 2024-09-13T14:55:37.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089686279999999993 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.796845+0000", 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011312971 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.798420+0000", 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015745500000000001 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.849219+0000", 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.050798994 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.850702+0000", 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001483053 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.859035+0000", 2024-09-13T14:55:37.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083336869999999993 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86785 6.4 6:2ff89860:::benchmark_data_smithi154_175640_object1009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95112575899998, 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.401965842000003, 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.547 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86785 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7500000000000001e-07 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508362+0000", 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.323e-06 2024-09-13T14:55:37.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508369+0000", 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508374+0000", 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508532+0000", 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000158066 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508541+0000", 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9740000000000008e-06 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037073+0000", 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528531924999999 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037081+0000", 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3380000000000005e-06 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793943+0000", 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75686249800000005 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793950+0000", 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8959999999999997e-06 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.813095+0000", 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.019144845 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.817234+0000", 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041391170000000003 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.822708+0000", 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054735089999999997 2024-09-13T14:55:37.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.860019+0000", 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037311534 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.860598+0000", 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000578781 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.900474+0000", 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039876138999999998 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.902193+0000", 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001718844 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.910326+0000", 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081324269999999994 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.555 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86817 6.4 6:29f76746:::benchmark_data_smithi154_175640_object1041:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508444+0000", 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.95104096699998, 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.427115067999999, 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86817 2024-09-13T14:55:37.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508444+0000", 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508444+0000", 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508445+0000", 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0999999999999998e-07 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508448+0000", 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.416e-06 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508458+0000", 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4469999999999995e-06 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508465+0000", 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3610000000000003e-06 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508567+0000", 2024-09-13T14:55:37.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010213900000000001 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508576+0000", 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3329999999999997e-06 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037114+0000", 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528537827000001 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037121+0000", 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0129999999999996e-06 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793974+0000", 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75685360199999996 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793981+0000", 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5799999999999997e-06 2024-09-13T14:55:37.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.829986+0000", 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.036004924 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.836767+0000", 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067812799999999998 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.842365+0000", 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055977359999999999 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.860883+0000", 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018518458000000002 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.862019+0000", 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011355339999999999 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.920345+0000", 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058326714000000002 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.924394+0000", 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004048856 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.935560+0000", 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011165237999999999 2024-09-13T14:55:37.564 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86824 6.4 6:26c7fd74:::benchmark_data_smithi154_175640_object1048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.849029+0000", 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.61045623400003, 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.125902050000001, 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.565 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86824 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849029+0000", 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849029+0000", 2024-09-13T14:55:37.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849031+0000", 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.649e-06 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849035+0000", 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8190000000000002e-06 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849045+0000", 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.047e-05 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849050+0000", 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1139999999999997e-06 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849111+0000", 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0603999999999998e-05 2024-09-13T14:55:37.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849127+0000", 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6089000000000001e-05 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037169+0000", 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.188041861999999 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037178+0000", 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7080000000000003e-06 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794006+0000", 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75682800400000005 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794012+0000", 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6549999999999998e-06 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.865282+0000", 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.071270213 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.870806+0000", 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055233510000000001 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.876156+0000", 2024-09-13T14:55:37.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053498440000000003 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.911402+0000", 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035246016999999998 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.912156+0000", 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00075456099999999995 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.957551+0000", 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.045395104999999998 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.960921+0000", 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00337002 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.974931+0000", 2024-09-13T14:55:37.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014009964999999999 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86833 6.4 6:299146a5:::benchmark_data_smithi154_175640_object1057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.876366+0000", 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.58311966299999, 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.111824777000002, 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.574 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86833 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876366+0000", 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876366+0000", 2024-09-13T14:55:37.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876367+0000", 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.375e-06 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876372+0000", 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9640000000000002e-06 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876384+0000", 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2367e-05 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876391+0000", 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7819999999999996e-06 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876457+0000", 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5437000000000004e-05 2024-09-13T14:55:37.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876468+0000", 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.152e-05 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037217+0000", 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.160749016 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037225+0000", 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0350000000000001e-06 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794036+0000", 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75681060600000005 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794042+0000", 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5740000000000001e-06 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.881409+0000", 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.087366431 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.886690+0000", 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052807449999999999 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.893946+0000", 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072567079999999997 2024-09-13T14:55:37.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.913284+0000", 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019337656000000002 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.917600+0000", 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043160029999999997 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.977875+0000", 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.060274600999999997 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.980107+0000", 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002232394 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.988191+0000", 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080835630000000002 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.582 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86857 6.4 6:28ca1b9c:::benchmark_data_smithi154_175640_object1081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.966732+0000", 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.49275303500002, 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.108482150999997, 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.583 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86857 2024-09-13T14:55:37.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:37 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2840477291' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:37.585 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:37 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2840477291' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.586 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966732+0000", 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966732+0000", 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966734+0000", 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.897e-06 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966741+0000", 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1729999999999998e-06 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966759+0000", 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7082999999999999e-05 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966772+0000", 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2978e-05 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966839+0000", 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7847000000000003e-05 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.589 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966852+0000", 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2519e-05 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037265+0000", 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.070413295000002 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037273+0000", 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8490000000000006e-06 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.590 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794068+0000", 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75679514800000003 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794075+0000", 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6409999999999996e-06 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.936087+0000", 2024-09-13T14:55:37.591 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.14201224 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.941272+0000", 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051847730000000002 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.947691+0000", 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064196419999999997 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.592 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.976256+0000", 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028564672999999999 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.022507+0000", 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046251160999999999 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.064329+0000", 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041822152000000001 2024-09-13T14:55:37.593 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.066135+0000", 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018055149999999999 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.075215+0000", 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090795649999999995 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86871 6.4 6:2f2f4832:::benchmark_data_smithi154_175640_object1095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:59.156758+0000", 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.30272696600002, 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 33.933005207999997, 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86871 2024-09-13T14:55:37.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156758+0000", 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156758+0000", 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.596 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156760+0000", 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4759999999999999e-06 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156764+0000", 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3340000000000003e-06 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156776+0000", 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1487e-05 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.597 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156784+0000", 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0630000000000006e-06 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156838+0000", 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4119999999999997e-05 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156846+0000", 2024-09-13T14:55:37.598 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6590000000000007e-06 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037312+0000", 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 22.880466437999999 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037319+0000", 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3950000000000003e-06 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.599 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794099+0000", 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75677971600000005 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794103+0000", 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0219999999999998e-06 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.600 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.988616+0000", 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.194513259000001 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.993901+0000", 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052842210000000004 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.000598+0000", 2024-09-13T14:55:37.601 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066976589999999999 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.023408+0000", 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.02280983 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.025486+0000", 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020782859999999999 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.078543+0000", 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.053056568999999998 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.080957+0000", 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002413583 2024-09-13T14:55:37.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.089764+0000", 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088070909999999995 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86889 6.4 6:247588cb:::benchmark_data_smithi154_175640_object1113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.582787+0000", 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 304.87669796400002, 2024-09-13T14:55:37.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.52048566299999999, 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86889 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.605 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582787+0000", 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582787+0000", 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582789+0000", 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.251e-06 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.606 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582794+0000", 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0529999999999997e-06 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582808+0000", 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4511999999999999e-05 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582817+0000", 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2819999999999996e-06 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.607 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.007485+0000", 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42466857899999999 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.011316+0000", 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00383096 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.016788+0000", 2024-09-13T14:55:37.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054720619999999998 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.024771+0000", 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079829429999999993 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.025835+0000", 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010641649999999999 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.092991+0000", 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.067156034000000003 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.095426+0000", 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024348059999999999 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.103273+0000", 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078470160000000001 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.610 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86896 6.4 6:21378335:::benchmark_data_smithi154_175640_object1120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.622457+0000", 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 304.83702820799999, 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.49349832399999999, 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.611 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86896 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622457+0000", 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622457+0000", 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622458+0000", 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0839999999999999e-06 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622463+0000", 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5619999999999997e-06 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622473+0000", 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0098000000000001e-05 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622480+0000", 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2049999999999996e-06 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.028350+0000", 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.40586974300000001 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.034726+0000", 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063761699999999996 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.040426+0000", 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056998259999999999 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.077031+0000", 2024-09-13T14:55:37.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036605407 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.090146+0000", 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013115096999999999 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.104571+0000", 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014424655999999999 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.106974+0000", 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024029089999999999 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.115956+0000", 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089815669999999993 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86923 6.4 6:244d3275:::benchmark_data_smithi154_175640_object1147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.721799+0000", 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 304.73768618999998, 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.40723228500000003, 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86923 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.618 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721799+0000", 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721799+0000", 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721802+0000", 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4559999999999999e-06 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721812+0000", 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8789999999999998e-06 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721826+0000", 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4837999999999999e-05 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721837+0000", 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0467e-05 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.046967+0000", 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.325129742 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.051648+0000", 2024-09-13T14:55:37.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046818500000000004 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.058047+0000", 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063980570000000004 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.090506+0000", 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032459384000000001 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.091521+0000", 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010147190000000001 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.117303+0000", 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025781999999999999 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.119836+0000", 2024-09-13T14:55:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025335409999999998 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.129032+0000", 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091953520000000004 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86953 6.4 6:2cb54e5d:::benchmark_data_smithi154_175640_object1177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.805319+0000", 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 304.65416624400001, 2024-09-13T14:55:37.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42385360999999999, 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86953 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805319+0000", 2024-09-13T14:55:37.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805319+0000", 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805321+0000", 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.516e-06 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805327+0000", 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1800000000000001e-06 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805343+0000", 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6021000000000001e-05 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805351+0000", 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0569999999999993e-06 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.129532+0000", 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.32418106200000002 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.134486+0000", 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049543770000000003 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.140485+0000", 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059984080000000002 2024-09-13T14:55:37.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.168220+0000", 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027735471000000001 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.217857+0000", 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049637037000000002 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.219707+0000", 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001849982 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.221214+0000", 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001507017 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.229173+0000", 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079584819999999994 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.630 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:87070 6.4 6:25334854:::benchmark_data_smithi154_175640_object1294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:33.147617+0000", 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 304.31186798099998, 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.41807496500000002, 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:55:37.631 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 87070 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147617+0000", 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147617+0000", 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:55:37.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147619+0000", 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6249999999999999e-06 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147626+0000", 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4359999999999999e-06 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:55:37.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147637+0000", 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1781e-05 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147645+0000", 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9160000000000001e-06 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.428804+0000", 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.28115832400000002 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.433717+0000", 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049131590000000003 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.441378+0000", 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0076612609999999999 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.476105+0000", 2024-09-13T14:55:37.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034727190999999998 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.502269+0000", 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026163842 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.554708+0000", 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052439432000000001 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.557604+0000", 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002895342 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.565692+0000", 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080886559999999996 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:55:37.637 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:55:37.638 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:37.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:37.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:37 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2840477291' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:55:37.780 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:37.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:37.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 31 16 54538 54522 109.909 79.875 0.00896511 0.00909246 2024-09-13T14:55:37.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:37.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:38.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:38.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:38.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:38.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:38.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:38.286 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:38.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:38.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:38.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:38.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:38.669 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:38.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[102314]: pgmap v821: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-13T14:55:38.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:38.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2840477291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:38.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[102314]: osdmap e266: 8 total, 8 up, 7 in 2024-09-13T14:55:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[107038]: pgmap v821: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-13T14:55:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2840477291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:38.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:38 smithi154 ceph-mon[107038]: osdmap e266: 8 total, 8 up, 7 in 2024-09-13T14:55:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:38 smithi172 ceph-mon[100696]: pgmap v821: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-13T14:55:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:38 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2840477291' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:55:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:38 smithi172 ceph-mon[100696]: osdmap e266: 8 total, 8 up, 7 in 2024-09-13T14:55:38.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:38.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:38.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 32 16 55794 55778 108.927 78.5 0.0031312 0.00916218 2024-09-13T14:55:38.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:38.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:38.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:39.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:39.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:39.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:39.213 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:39.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:39.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:39.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:39.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:39.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:39.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:39.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:39.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:39.751 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:39.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:39.888 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 33 16 57095 57079 108.09 81.3125 0.027331 0.00924492 2024-09-13T14:55:39.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:39.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:39.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:40.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:55:40.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T14:55:40.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:40.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T14:55:40.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:40.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:40.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:40.342 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:55:40.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:40.455 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:40.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:40.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:40.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:40.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:40 smithi154 ceph-mon[102314]: pgmap v823: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:55:40.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:40 smithi154 ceph-mon[107038]: pgmap v823: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:55:40.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:40 smithi172 ceph-mon[100696]: pgmap v823: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-13T14:55:40.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:40.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:40.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:40.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 34 16 57863 57847 106.323 48 0.0437134 0.00939282 2024-09-13T14:55:40.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:40.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:41.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:41.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:41.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:41.383 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:41.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:41.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:41.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:41.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:41.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:41.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:41.834 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:41.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:41.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 35 16 59178 59162 105.633 82.1875 0.0223453 0.00946098 2024-09-13T14:55:41.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:41.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:41.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:42.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:42.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:42.151 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:55:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:55:41] "GET /metrics HTTP/1.1" 200 36509 "" "Prometheus/2.51.0" 2024-09-13T14:55:42.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:42.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:42.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:42.427 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:42.449 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:42 smithi154 ceph-mon[102314]: pgmap v824: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:55:42.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:42 smithi154 ceph-mon[107038]: pgmap v824: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:55:42.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:42.551 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:42.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:42.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:42.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:42.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:42.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:42 smithi172 ceph-mon[100696]: pgmap v824: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-13T14:55:42.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:42.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:42.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:42.867 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:42.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 36 16 60370 60354 104.767 74.5 0.0183311 0.00953946 2024-09-13T14:55:42.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:43.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:43.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:43.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:43.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:43.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:43.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:43.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:43.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:43.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:43.543 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:43.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:43.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:43.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:43.831 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:43.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 37 16 61626 61610 104.057 78.5 0.00944185 0.00960387 2024-09-13T14:55:43.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:43.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:55:43.971 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:43.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:43.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:44.012 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:44.012 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:44.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:43 smithi154 ceph-mon[102314]: pgmap v825: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:55:44.013 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:44.013 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:44.013 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:43 smithi154 ceph-mon[107038]: pgmap v825: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:55:44.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:44.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:44.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:43 smithi172 ceph-mon[100696]: pgmap v825: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T14:55:44.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:44.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:44.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:44.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:44.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:44.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:44.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:44.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:44.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:44.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 38 16 62732 62716 103.138 69.125 0.0169488 0.00968919 2024-09-13T14:55:44.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:44.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:44.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:45.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.116 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1493116153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1493116153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1493116153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:55:45.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:45.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:45.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:45.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:45.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:45.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:45.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:45.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:45.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:45.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:45.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 39 16 63790 63774 102.188 66.125 0.015546 0.00978027 2024-09-13T14:55:45.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.010 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:46.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:46.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:46.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:46.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:45 smithi154 ceph-mon[102314]: pgmap v826: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:55:46.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:45 smithi154 ceph-mon[107038]: pgmap v826: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:55:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:45 smithi172 ceph-mon[100696]: pgmap v826: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T14:55:46.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:46.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:46.383 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:46.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:46.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:46.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:46.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:46.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:46.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:46.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:46.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:46.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:55:46.889331+0000 min lat: 0.000986333 max lat: 0.318275 avg lat: 0.00987191 2024-09-13T14:55:46.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:55:46.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 40 16 64820 64804 101.243 64.375 0.0130051 0.00987191 2024-09-13T14:55:46.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:46.971 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:55:46.971 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:55:46.971 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:55:46.971 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:55:46.972 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:55:46.973 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:55:46.973 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:55:47.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:47.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:47.131 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:47.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:47.256 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:47.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:47.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:47.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:47.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:47.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:47.657 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:47.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:47.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:47.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 41 16 66076 66060 100.688 78.5 0.0142665 0.00992568 2024-09-13T14:55:47.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:47.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:47.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:48.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:48.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:48.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:48.208 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:48.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:48.333 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:48.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:48.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:48.463 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:48 smithi154 ceph-mon[102314]: pgmap v827: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:55:48.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:48 smithi154 ceph-mon[107038]: pgmap v827: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:55:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:48 smithi172 ceph-mon[100696]: pgmap v827: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T14:55:48.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:48.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:48.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:48.745 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:48.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:48.873 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:48.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:48.889 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 42 16 67306 67290 100.121 76.875 0.0178499 0.00998182 2024-09-13T14:55:48.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:49.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:49.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:49.173 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:49.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:49.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:49.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:49.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:49.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:49.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:49.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:49.731 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:49.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:55:49.856 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:49.890 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 43 16 68427 68411 99.4218 70.0625 0.0177024 0.0100521 2024-09-13T14:55:49.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:49.969 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:49.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:49.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:50.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:50.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:50.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:50 smithi154 ceph-mon[102314]: pgmap v828: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:55:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:50 smithi154 ceph-mon[107038]: pgmap v828: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:55:50.230 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:50.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:50.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:50.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:50.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:50 smithi172 ceph-mon[100696]: pgmap v828: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T14:55:50.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:50.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:50.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:50.752 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:50.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:50.890 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 44 16 69319 69303 98.4292 55.75 0.0106327 0.0101537 2024-09-13T14:55:50.892 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:50.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:50.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:51.021 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:51.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:51.132 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T14:55:51.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:51.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:51.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:51.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:51.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:51.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/162911760' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:55:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:51 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:55:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/162911760' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:55:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:51 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:55:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/162911760' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:55:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:51 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T14:55:51.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:51.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:51.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:51.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:51.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:51.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:51.890 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 45 16 70418 70402 97.7682 68.6875 0.00901709 0.0102225 2024-09-13T14:55:51.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:51.921 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:51.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:52.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:52.050 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:52.135 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:55:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:55:51] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-13T14:55:52.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:52.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:52.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:52.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:52.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:52.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:52 smithi154 ceph-mon[102314]: pgmap v829: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:55:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:52 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:55:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:52 smithi154 ceph-mon[102314]: osdmap e267: 8 total, 8 up, 7 in 2024-09-13T14:55:52.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:52 smithi154 ceph-mon[107038]: pgmap v829: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:55:52.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:52 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:55:52.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:52 smithi154 ceph-mon[107038]: osdmap e267: 8 total, 8 up, 7 in 2024-09-13T14:55:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:52 smithi172 ceph-mon[100696]: pgmap v829: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-13T14:55:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:52 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T14:55:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:52 smithi172 ceph-mon[100696]: osdmap e267: 8 total, 8 up, 7 in 2024-09-13T14:55:52.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:52.582 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:52.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:52.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:52.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:52.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:52.890 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 46 16 71444 71428 97.0367 64.125 0.0366545 0.0102982 2024-09-13T14:55:52.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:52.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:52.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:53.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:53.088 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:53.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:53.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:53.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:53.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:53.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:53.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:53.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:53.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:53.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:53.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:53.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:53.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:53.890 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 47 16 72590 72574 96.4959 71.625 0.0313558 0.010357 2024-09-13T14:55:53.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:53.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:54.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:54.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:54.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:54.185 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:54.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:54.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:54.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:54.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:54.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:54 smithi154 ceph-mon[102314]: pgmap v831: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-13T14:55:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:54 smithi154 ceph-mon[107038]: pgmap v831: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-13T14:55:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:54 smithi172 ceph-mon[100696]: pgmap v831: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-13T14:55:54.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:54.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:54.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T14:55:54.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:54.684 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:55:54.684 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:55:54.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:54.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:54.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:54.891 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 48 16 73691 73675 95.9174 68.8125 0.0301764 0.010419 2024-09-13T14:55:54.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:54.997 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:54.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:55.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:55.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:55.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:55.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:55.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:55.395 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:55.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:55.522 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:55.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T14:55:55.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:55.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:55.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:55.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:55.892 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 49 16 74772 74756 95.3373 67.5625 0.0211806 0.0104831 2024-09-13T14:55:55.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:55.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:55.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:56.067 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:56.074 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:56.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:56.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:56.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:56.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:56.403 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:56 smithi154 ceph-mon[102314]: pgmap v832: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:55:56.403 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:56.403 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:56 smithi154 ceph-mon[107038]: pgmap v832: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:55:56.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:56 smithi172 ceph-mon[100696]: pgmap v832: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-13T14:55:56.505 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:56.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:56.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:56.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:56.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:56.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:56.892 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 50 16 75758 75742 94.663 61.625 0.0244157 0.0105575 2024-09-13T14:55:56.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:56.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:56.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:57.064 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:57.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:57.175 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:55:57.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:57.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:57.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:57.363 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:57.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/128511021' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:57.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:57 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:57.451 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/128511021' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:57.451 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:57 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:57.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/128511021' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:57 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:55:57.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:57.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:57.626 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:57.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:57.768 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:57.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:57.892 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 51 16 76792 76776 94.074 64.625 0.0127195 0.0106243 2024-09-13T14:55:57.895 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:57.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:57.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:58.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:58.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:58.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:58.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:55:58.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:55:58.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:58.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[102314]: pgmap v833: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:55:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[102314]: osdmap e268: 8 total, 8 up, 7 in 2024-09-13T14:55:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[107038]: pgmap v833: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:55:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[107038]: osdmap e268: 8 total, 8 up, 7 in 2024-09-13T14:55:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:58 smithi172 ceph-mon[100696]: pgmap v833: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T14:55:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:58 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:55:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:55:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:58 smithi172 ceph-mon[100696]: osdmap e268: 8 total, 8 up, 7 in 2024-09-13T14:55:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:55:58.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:58.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:58.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:58.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:58.810 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:55:58.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:58.856 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:58.894 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 52 16 77780 77764 93.449 61.75 0.0220156 0.0106947 2024-09-13T14:55:58.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:55:58.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:58.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:55:59.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:55:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:59.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:55:59.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:55:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:59.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:55:59.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:55:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:59.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:55:59.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:55:59.545 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.545 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.545 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:59.545 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:59.546 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:59.546 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:55:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:55:59.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:55:59.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:55:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:55:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:55:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:55:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:55:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:55:59.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:59.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:55:59.894 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 53 16 78772 78756 92.8555 62 0.0081641 0.0107643 2024-09-13T14:55:59.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:55:59.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:55:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:55:59.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:00.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:00.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:00.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:00.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:00.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:00.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:00 smithi154 ceph-mon[102314]: pgmap v835: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-13T14:56:00.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:00 smithi154 ceph-mon[107038]: pgmap v835: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-13T14:56:00.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:00.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:00.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:00.638 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:00.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:00 smithi172 ceph-mon[100696]: pgmap v835: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-13T14:56:00.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:00.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:00.890 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:00.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:00.894 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 54 16 79684 79668 92.1915 57 0.00312752 0.0108302 2024-09-13T14:56:00.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:01.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:01.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.154 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:01.282 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:01.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:56:01.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:01.408 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:01.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:01.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:01.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:01.650 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3332641128' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:56:01.650 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:01 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:56:01.650 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3332641128' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:56:01.650 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:01 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:56:01.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:01.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3332641128' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:56:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:01 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T14:56:01.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:01.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:01.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:01.867 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T14:56:01.894 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 55 16 80687 80671 91.655 62.6875 0.00941009 0.0109051 2024-09-13T14:56:01.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:01.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:01.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:56:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:56:01] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:56:02.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:02.077 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:02.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:02.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:02.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:02.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:02.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:02.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:02.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:02.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:02.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:02 smithi154 ceph-mon[102314]: pgmap v836: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T14:56:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:02 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:56:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:02 smithi154 ceph-mon[102314]: osdmap e269: 8 total, 8 up, 7 in 2024-09-13T14:56:02.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:02 smithi154 ceph-mon[107038]: pgmap v836: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T14:56:02.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:02 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:56:02.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:02 smithi154 ceph-mon[107038]: osdmap e269: 8 total, 8 up, 7 in 2024-09-13T14:56:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:02 smithi172 ceph-mon[100696]: pgmap v836: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-13T14:56:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:02 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T14:56:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:02 smithi172 ceph-mon[100696]: osdmap e269: 8 total, 8 up, 7 in 2024-09-13T14:56:02.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:02.790 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:02.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T14:56:02.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:02.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 56 16 81595 81579 91.0305 56.75 0.0212023 0.0109807 2024-09-13T14:56:02.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:02.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:03.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:03.043 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:03.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:03.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:03.209 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:56:03.209 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:03.242 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:56:03.243 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:03.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.289 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:56:03.289 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:03.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:03.325 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:56:03.325 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:03.359 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:56:03.360 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:03.395 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:56:03.395 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:03.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:03.420 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:03.426 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:56:03.426 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:03.458 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:56:03.458 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:03.459 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3131265063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:03.459 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3131265063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:03.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:03.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:03.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:03.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3131265063' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:03.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:03.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:03.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 57 16 82393 82377 90.3086 49.875 0.0173184 0.011066 2024-09-13T14:56:03.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:03.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:03.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:04.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:04.183 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:04.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.244 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:04.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:04.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:04.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:04.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:04.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.497 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.627 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:04 smithi154 ceph-mon[102314]: pgmap v838: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T14:56:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:04 smithi154 ceph-mon[107038]: pgmap v838: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T14:56:04.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:04 smithi172 ceph-mon[100696]: pgmap v838: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T14:56:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:04.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:04.862 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:04.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:04.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 58 16 83361 83345 89.7946 60.5 0.0148201 0.0111297 2024-09-13T14:56:04.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:04.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:04.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.045 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:05.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:05.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:05.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:05.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:05.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:05.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.421 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:05.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:05.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:05.548 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:05.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:05.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:05.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:05.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:05.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 59 16 84370 84354 89.3414 63.0625 0.0171227 0.011187 2024-09-13T14:56:05.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:05.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:05.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:05.990 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:56:05.990 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:56:05.991 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:56:05.991 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:56:05.991 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:56:06.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.092 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:06.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:06.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:06.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:06.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:06.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:06.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:06.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:06.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:06.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:06.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:06 smithi154 ceph-mon[102314]: pgmap v839: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T14:56:06.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:06 smithi154 ceph-mon[107038]: pgmap v839: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T14:56:06.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:06 smithi172 ceph-mon[100696]: pgmap v839: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-13T14:56:06.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:06.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:06.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:06.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:56:06.895142+0000 min lat: 0.000986333 max lat: 0.318275 avg lat: 0.0112521 2024-09-13T14:56:06.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:56:06.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 60 16 85304 85288 88.8253 58.375 0.0101184 0.0112521 2024-09-13T14:56:06.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:07.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:07.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:07.131 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:07.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:07.256 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:07.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:07.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:07.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:07.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:07.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:07.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:07.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:07.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:07.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:07.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 61 16 86104 86088 88.1888 50 0.00889347 0.0113335 2024-09-13T14:56:07.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:07.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:08.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:08.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:08.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:08.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:08.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.291 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:08.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:08.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:08.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:08.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:08.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:08 smithi154 ceph-mon[102314]: pgmap v840: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:56:08.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:08 smithi154 ceph-mon[107038]: pgmap v840: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:56:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:08 smithi172 ceph-mon[100696]: pgmap v840: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-13T14:56:08.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:08.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:08.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 62 16 86970 86954 87.6394 54.125 0.0106785 0.0114047 2024-09-13T14:56:08.924 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:08.948 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:08.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:09.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:09.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:09.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:09.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.352 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:09.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:09.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:09.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:09.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:09.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:09.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:09.872 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:09.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:09.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 63 16 87865 87849 87.1361 55.9375 0.0330323 0.0114703 2024-09-13T14:56:09.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:10.011 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:10.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:10.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:10.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:10.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:10.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:10.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:10.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:10.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:10.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:10.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:10 smithi154 ceph-mon[102314]: pgmap v841: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:56:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:10 smithi154 ceph-mon[107038]: pgmap v841: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:56:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:10 smithi172 ceph-mon[100696]: pgmap v841: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T14:56:10.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:10.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:10.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 64 16 88697 88681 86.5871 52 0.00765238 0.0115426 2024-09-13T14:56:10.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:10.972 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:10.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.096 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:11.221 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:11.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:11.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:11.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:11.749 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:11.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:11.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:11.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:11.895 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 65 16 89704 89688 86.2232 62.9375 0.0219021 0.0115921 2024-09-13T14:56:11.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:56:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:56:11] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:56:11.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:12.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:12.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:12.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:12.168 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:12.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.311 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:12.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:12.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T14:56:12.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:12.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:12.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:12.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:12.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:12 smithi154 ceph-mon[102314]: pgmap v842: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-13T14:56:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:12 smithi154 ceph-mon[107038]: pgmap v842: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-13T14:56:12.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:12 smithi172 ceph-mon[100696]: pgmap v842: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-13T14:56:12.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.808 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:12.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:12.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 66 16 90639 90623 85.8022 58.4375 0.020873 0.0116497 2024-09-13T14:56:12.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:12.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:12.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:13.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:13.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:13.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:13.236 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:13.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:13.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:13.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:13.515 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:13.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:13.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:13.645 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:13.645 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:13.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:56:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:13.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:13.763 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:13.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:13.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:13.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:13.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 67 16 91501 91485 85.3256 53.875 0.0134475 0.011714 2024-09-13T14:56:13.934 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:13.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:14.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:14.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:14.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:14.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:14.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:14.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:14.426 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T14:56:14.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:14.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:14.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:14.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:14 smithi154 ceph-mon[102314]: pgmap v843: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 58 MiB/s wr, 930 op/s 2024-09-13T14:56:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/289942257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:14 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:14.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:14 smithi154 ceph-mon[107038]: pgmap v843: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 58 MiB/s wr, 930 op/s 2024-09-13T14:56:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/289942257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:14 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:14.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:14 smithi172 ceph-mon[100696]: pgmap v843: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 58 MiB/s wr, 930 op/s 2024-09-13T14:56:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/289942257' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:14 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T14:56:14.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:14.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:14.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 68 16 92263 92247 84.7712 47.625 0.0225509 0.0117907 2024-09-13T14:56:14.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:14.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:14.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:15.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:15.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:15.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:15.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:15.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.421 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:15.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:15.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:15.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:15.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:15.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:15.712 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:15 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:56:15.712 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:15 smithi154 ceph-mon[102314]: osdmap e270: 8 total, 8 up, 7 in 2024-09-13T14:56:15.712 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:15 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:56:15.713 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:15 smithi154 ceph-mon[107038]: osdmap e270: 8 total, 8 up, 7 in 2024-09-13T14:56:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:15 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T14:56:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:15 smithi172 ceph-mon[100696]: osdmap e270: 8 total, 8 up, 7 in 2024-09-13T14:56:15.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:15.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:15.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 69 16 93161 93145 84.356 56.125 0.0140875 0.0118485 2024-09-13T14:56:15.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:15.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:15.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:16.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:16.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:16.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:16.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:16.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:16.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:16.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T14:56:16.455 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:56:16.455 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:16.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:16.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:16.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:16.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:16 smithi154 ceph-mon[102314]: pgmap v845: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1787769300' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:16 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:16 smithi154 ceph-mon[107038]: pgmap v845: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:56:16.669 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1787769300' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:16.670 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:16 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:16 smithi172 ceph-mon[100696]: pgmap v845: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-13T14:56:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1787769300' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:16 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:56:16.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:16.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:16.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:16.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:16.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 70 16 93998 93982 83.8981 52.3125 0.0202875 0.011914 2024-09-13T14:56:16.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:16.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:17.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:17.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:17.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:17.188 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:17.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:17.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:17.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:17.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:17.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:17.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:17.647 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:17 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:17.648 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:17 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:17.648 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:17 smithi154 ceph-mon[102314]: osdmap e271: 8 total, 8 up, 7 in 2024-09-13T14:56:17.648 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:17 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:17.648 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:17 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:17.648 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:17 smithi154 ceph-mon[107038]: osdmap e271: 8 total, 8 up, 7 in 2024-09-13T14:56:17.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:17 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:17 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:56:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:17 smithi172 ceph-mon[100696]: osdmap e271: 8 total, 8 up, 7 in 2024-09-13T14:56:17.752 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:17.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:17.880 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:17.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:17.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 71 16 94938 94922 83.5439 58.75 0.0113293 0.0119644 2024-09-13T14:56:17.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:18.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:18.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:18.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:18.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:18.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:18.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:18.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:18.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:18.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:18.561 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:18 smithi154 ceph-mon[107038]: pgmap v847: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-13T14:56:18.562 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:18 smithi154 ceph-mon[102314]: pgmap v847: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-13T14:56:18.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:18.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:18 smithi172 ceph-mon[100696]: pgmap v847: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 56 MiB/s wr, 892 op/s 2024-09-13T14:56:18.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:18.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:18.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 72 16 95862 95846 83.1856 57.75 0.00935741 0.0120161 2024-09-13T14:56:18.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:18.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:18.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:19.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:19.096 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:19.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:19.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:19.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:19.373 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:19.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:56:19.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:19.539 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:19.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:19.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:19.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:19.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:19.871 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:19.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:19.896 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 73 16 96700 96684 82.7635 52.375 0.025971 0.0120776 2024-09-13T14:56:19.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:20.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:20.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:20.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:20.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:20.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:20.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:20.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:20.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:20.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:20.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:20.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:20.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:20 smithi154 ceph-mon[102314]: pgmap v848: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-13T14:56:20.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:20 smithi154 ceph-mon[107038]: pgmap v848: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-13T14:56:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:20 smithi172 ceph-mon[100696]: pgmap v848: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-13T14:56:20.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:20.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:20.897 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 74 16 97597 97581 82.4026 56.0625 0.0102158 0.0121298 2024-09-13T14:56:20.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:20.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:20.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:21.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:21.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:21.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:21.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:21.233 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:21.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:21.361 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:21.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:21.486 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:21.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:21.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:21.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:21.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:21.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:21.897 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 75 16 98442 98426 82.0081 52.8125 0.0239855 0.012188 2024-09-13T14:56:21.902 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:21.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:22.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:22.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:22.052 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:56:22.052 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:22.091 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:56:22.092 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:22.130 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:56:22.131 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:22.131 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:56:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:56:21] "GET /metrics HTTP/1.1" 200 36513 "" "Prometheus/2.51.0" 2024-09-13T14:56:22.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.173 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:22.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:22.176 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:56:22.176 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:22.209 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:56:22.209 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:22.241 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:56:22.242 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:22.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:22.278 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:56:22.278 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:22.312 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:56:22.312 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:22.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:22.429 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:22.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:22.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:22.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:22.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:22.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:22.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:22.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:22.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:22.871 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:22 smithi154 ceph-mon[102314]: pgmap v849: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 858 op/s 2024-09-13T14:56:22.871 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:22 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2359344746' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:22.871 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:22 smithi154 ceph-mon[107038]: pgmap v849: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 858 op/s 2024-09-13T14:56:22.872 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:22 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2359344746' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:22.897 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 76 16 99291 99275 81.6272 53.0625 0.0138668 0.012245 2024-09-13T14:56:22.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:22.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:22 smithi172 ceph-mon[100696]: pgmap v849: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 858 op/s 2024-09-13T14:56:22.993 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:22 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2359344746' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:22.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:22.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:23.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:23.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:23.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:23.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:23.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:23.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:23.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:23.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:23.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:23.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:23.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:23.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:23.894 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:23.894 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:23.897 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 77 16 100152 100136 81.2659 53.8125 0.0161146 0.0122989 2024-09-13T14:56:23.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:23.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:23.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:24.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:24.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:24.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:24.252 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:24.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:24.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:24.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:24.509 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T14:56:24.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:24.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:24.761 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:24 smithi154 ceph-mon[102314]: pgmap v850: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 893 op/s 2024-09-13T14:56:24.761 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2095898044' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:56:24.761 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:24 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:56:24.761 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:24 smithi154 ceph-mon[107038]: pgmap v850: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 893 op/s 2024-09-13T14:56:24.761 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2095898044' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:56:24.762 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:24 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:56:24.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:24.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:24.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:56:24.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:24.897 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 78 16 100962 100946 80.873 50.625 0.0110965 0.0123585 2024-09-13T14:56:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:24.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:24.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:24 smithi172 ceph-mon[100696]: pgmap v850: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 56 MiB/s wr, 893 op/s 2024-09-13T14:56:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2095898044' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:56:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:24 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T14:56:25.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:25.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:25.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:25.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:25.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:25.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:25.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:25.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:25.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:25.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:25.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:25.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:25.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:25 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:56:25.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:25 smithi154 ceph-mon[102314]: osdmap e272: 8 total, 8 up, 7 in 2024-09-13T14:56:25.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:25 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:56:25.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:25 smithi154 ceph-mon[107038]: osdmap e272: 8 total, 8 up, 7 in 2024-09-13T14:56:25.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:25.771 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:25.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T14:56:25.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:25.898 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 79 16 101801 101785 80.5121 52.4375 0.0134714 0.012415 2024-09-13T14:56:25.936 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:25.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:25 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T14:56:25.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:25 smithi172 ceph-mon[100696]: osdmap e272: 8 total, 8 up, 7 in 2024-09-13T14:56:26.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:26.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:26.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:26.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:26.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:26.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:26.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:26.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:26.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:26.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:26.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:26.728 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:26.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:26.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:26.898 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T14:56:26.897960+0000 min lat: 0.000986333 max lat: 0.318275 avg lat: 0.0124722 2024-09-13T14:56:26.898 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T14:56:26.898 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 80 16 102612 102596 80.1392 50.6875 0.0120302 0.0124722 2024-09-13T14:56:26.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:26.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:26 smithi154 ceph-mon[102314]: pgmap v852: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 58 MiB/s wr, 922 op/s 2024-09-13T14:56:26.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:26 smithi154 ceph-mon[107038]: pgmap v852: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 58 MiB/s wr, 922 op/s 2024-09-13T14:56:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:26 smithi172 ceph-mon[100696]: pgmap v852: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 58 MiB/s wr, 922 op/s 2024-09-13T14:56:26.991 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:26.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:27.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:27.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:27.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:27.243 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:27.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:27.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:27.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:27.494 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:27.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:27.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:27.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:27.763 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:27.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:27.898 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 81 16 103292 103276 79.6745 42.5 0.0470367 0.0125461 2024-09-13T14:56:27.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:27.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:27.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:28.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:28.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:28.126 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:28.137 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:28.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:28.175 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:28.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:28.302 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:28.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:28.428 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:28.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:28.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:28.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:28.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:28.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:28.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:28.898 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 82 16 104091 104075 79.3119 49.9375 0.0101885 0.0126024 2024-09-13T14:56:28.913 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:28 smithi154 ceph-mon[107038]: pgmap v853: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:56:28.913 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:28.914 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:28 smithi154 ceph-mon[102314]: pgmap v853: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:56:28.914 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:28.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:28.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:28.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:28 smithi172 ceph-mon[100696]: pgmap v853: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-13T14:56:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:29.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:29.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:29.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:29.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:29.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:29.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:29.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:29.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:29.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:29.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:29.677 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:29.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:29.801 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:29.898 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 83 16 104957 104941 79.0084 54.125 0.0152468 0.0126519 2024-09-13T14:56:29.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:29.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:29.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:30.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:30.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:30.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:30.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:30.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:30.366 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:30.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:30.505 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:30.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:30.629 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:30.730 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:30 smithi154 ceph-mon[102314]: pgmap v854: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 53 MiB/s wr, 848 op/s 2024-09-13T14:56:30.730 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:30 smithi154 ceph-mon[107038]: pgmap v854: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 53 MiB/s wr, 848 op/s 2024-09-13T14:56:30.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:30.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:30.844 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:30.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:30.902 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 84 16 105786 105770 78.6816 51.8125 0.0246009 0.0127029 2024-09-13T14:56:30.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:30.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:30 smithi172 ceph-mon[100696]: pgmap v854: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 53 MiB/s wr, 848 op/s 2024-09-13T14:56:31.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:31.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:31.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:31.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:31.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:31.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:31.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:31.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:31.551 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:56:31.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:31.594 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:31.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:31.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:31.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:31 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2204232600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:31 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2204232600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:31.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:31.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:31.902 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 85 16 106529 106513 78.3022 46.4375 0.0177502 0.0127648 2024-09-13T14:56:31.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:31.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:31 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2204232600' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:31.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:31.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:32.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:32.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:32.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:56:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:56:31] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:56:32.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:32.255 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:32.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:32.382 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:32.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:32.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:32.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:32.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:32.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[102314]: pgmap v855: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:56:32.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2204232600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[102314]: osdmap e273: 8 total, 8 up, 7 in 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[107038]: pgmap v855: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2204232600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:32 smithi154 ceph-mon[107038]: osdmap e273: 8 total, 8 up, 7 in 2024-09-13T14:56:32.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:32.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:32.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:32.902 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 86 16 107194 107178 77.875 41.5625 0.115814 0.0128266 2024-09-13T14:56:32.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:32.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:32 smithi172 ceph-mon[100696]: pgmap v855: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 53 MiB/s wr, 841 op/s 2024-09-13T14:56:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:32 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:56:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:32 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:56:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2204232600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:32 smithi172 ceph-mon[100696]: osdmap e273: 8 total, 8 up, 7 in 2024-09-13T14:56:33.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:33.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:33.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:33.218 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:33.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:33.344 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:33.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:33.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:33.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:33.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:33.615 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:33.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:33.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:33.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:33.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:33.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:33.902 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 87 16 108006 107990 77.5632 50.75 0.0112374 0.0128866 2024-09-13T14:56:33.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:34.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:34.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:34.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:34.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:34.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:34.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:34.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:34.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:34.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:34.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:34.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:34.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:34.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:34.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:34.891 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:34 smithi154 ceph-mon[102314]: pgmap v857: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 52 MiB/s wr, 829 op/s 2024-09-13T14:56:34.891 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:34 smithi154 ceph-mon[107038]: pgmap v857: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 52 MiB/s wr, 829 op/s 2024-09-13T14:56:34.902 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 88 16 108799 108783 77.2449 49.5625 0.0266056 0.0129397 2024-09-13T14:56:34.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:34.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:34 smithi172 ceph-mon[100696]: pgmap v857: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 52 MiB/s wr, 829 op/s 2024-09-13T14:56:35.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:35.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:35.139 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:35.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:35.263 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:35.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:35.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:35.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:35.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:35.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:35.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:35.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:35.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:35.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:35.902 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 89 16 109626 109610 76.9576 51.6875 0.0194038 0.0129885 2024-09-13T14:56:35.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:35.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:36.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:36.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:36.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:36.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:36.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:36.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:56:36.285 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:36.286 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:36.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:36.320 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:36.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:36.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:36.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:36.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:36.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:36.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:36.806 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:36 smithi154 ceph-mon[102314]: pgmap v858: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 799 op/s 2024-09-13T14:56:36.806 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1646064478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:36.806 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:36 smithi154 ceph-mon[107038]: pgmap v858: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 799 op/s 2024-09-13T14:56:36.807 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1646064478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:36.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:36.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total time run: 90.0127 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total writes made: 110261 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Write size: 65536 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object size: 65536 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Bandwidth (MB/sec): 76.5593 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Bandwidth: 32.3576 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max bandwidth (MB/sec): 203 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min bandwidth (MB/sec): 41.5625 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average IOPS: 1224 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev IOPS: 517.721 2024-09-13T14:56:36.903 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max IOPS: 3248 2024-09-13T14:56:36.904 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min IOPS: 665 2024-09-13T14:56:36.904 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average Latency(s): 0.0130569 2024-09-13T14:56:36.904 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Latency(s): 0.0122449 2024-09-13T14:56:36.904 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max latency(s): 0.318275 2024-09-13T14:56:36.904 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min latency(s): 0.000986333 2024-09-13T14:56:36.907 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T14:56:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:56:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:56:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:56:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:56:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:56:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:36.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:36.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:36 smithi172 ceph-mon[100696]: pgmap v858: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 799 op/s 2024-09-13T14:56:36.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1646064478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:36.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:36.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:37.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.122 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:37.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:37.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:37.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:37.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:37.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:37.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:37.769 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:37.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:37.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:37.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:37.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:38.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:38.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:38.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.187 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:38.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:38.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:38.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:38.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:38.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:56:38.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:38.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:38.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:38.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:38 smithi154 ceph-mon[102314]: pgmap v859: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 49 MiB/s wr, 782 op/s 2024-09-13T14:56:38.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:38 smithi154 ceph-mon[107038]: pgmap v859: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 49 MiB/s wr, 782 op/s 2024-09-13T14:56:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:38.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:38.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:38.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:38.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:38.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:38.978 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:38.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:38 smithi172 ceph-mon[100696]: pgmap v859: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 49 MiB/s wr, 782 op/s 2024-09-13T14:56:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:39.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:39.031 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:39.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:39.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:39.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.304 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:39.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:39.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:39.430 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:39.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:39.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:39.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:39.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:39.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:56:39.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:39.838 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:39.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:39.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:40.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:40.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:40.126 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:40.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:40.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:40.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:40.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:40.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:40.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:40.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:40.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:40.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:40.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:40.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:40.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:40.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:40.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:40 smithi154 ceph-mon[102314]: pgmap v860: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 46 MiB/s wr, 782 op/s 2024-09-13T14:56:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:40 smithi154 ceph-mon[107038]: pgmap v860: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 46 MiB/s wr, 782 op/s 2024-09-13T14:56:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:40 smithi172 ceph-mon[100696]: pgmap v860: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 46 MiB/s wr, 782 op/s 2024-09-13T14:56:41.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:41.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:41.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:41.188 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:41.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:41.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:41.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:41.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:41.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:41.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:41.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:41.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:41.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:41.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:41.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:41.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:42.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:42.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:42.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:42.130 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:56:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:56:41] "GET /metrics HTTP/1.1" 200 36512 "" "Prometheus/2.51.0" 2024-09-13T14:56:42.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:42.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:42.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:42.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:42.427 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:42.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:42.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:42.577 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T14:56:42.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:42.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:42.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:42.841 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:42.908 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:42 smithi154 ceph-mon[102314]: pgmap v861: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 39 MiB/s wr, 765 op/s 2024-09-13T14:56:42.908 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1094103837' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:56:42.908 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:42 smithi154 ceph-mon[107038]: pgmap v861: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 39 MiB/s wr, 765 op/s 2024-09-13T14:56:42.908 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1094103837' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:56:42.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:42.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:42.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:42 smithi172 ceph-mon[100696]: pgmap v861: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 39 MiB/s wr, 765 op/s 2024-09-13T14:56:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1094103837' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T14:56:43.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:43.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:43.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:43.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:43.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:43.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:43.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:43.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:43.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:43.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:43.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:43.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1094103837' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[102314]: osdmap e274: 8 total, 8 up, 7 in 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1094103837' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[107038]: osdmap e274: 8 total, 8 up, 7 in 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:56:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:43.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:43.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:43.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:43.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:43.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:43.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1094103837' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T14:56:43.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:43 smithi172 ceph-mon[100696]: osdmap e274: 8 total, 8 up, 7 in 2024-09-13T14:56:43.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:56:43.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:44.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:44.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:44.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.361 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:44.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:44.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.505 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:44.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:44.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T14:56:44.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:44.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:44.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:44.784 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:44.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:44.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:44.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:44.921 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:44 smithi154 ceph-mon[102314]: pgmap v863: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 29 MiB/s wr, 762 op/s 2024-09-13T14:56:44.921 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:44 smithi154 ceph-mon[107038]: pgmap v863: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 29 MiB/s wr, 762 op/s 2024-09-13T14:56:44.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:44.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:44 smithi172 ceph-mon[100696]: pgmap v863: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 29 MiB/s wr, 762 op/s 2024-09-13T14:56:45.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:45.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:45.126 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:45.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:45.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:45.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:45.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:45.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:45.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:45.581 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:56:45.582 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:56:45.582 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:56:45.582 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:56:45.582 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:56:45.606 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:56:45.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:45.681 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:45 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1438779785' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:45.681 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:45 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:45.681 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:45 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1438779785' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:45.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:45 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:45.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:45.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:45.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:45.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:45.938 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:45.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:45.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:45 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1438779785' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:45.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:45 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:56:46.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:46.065 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:46.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:46.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:46.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:46.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:46.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:46.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:46.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.624 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:46.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:46.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[102314]: pgmap v864: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 20 MiB/s wr, 838 op/s 2024-09-13T14:56:46.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:46.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:46.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[102314]: osdmap e275: 8 total, 8 up, 7 in 2024-09-13T14:56:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[107038]: pgmap v864: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 20 MiB/s wr, 838 op/s 2024-09-13T14:56:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:46.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:46 smithi154 ceph-mon[107038]: osdmap e275: 8 total, 8 up, 7 in 2024-09-13T14:56:46.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:46.749 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:46.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:46.877 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:46.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:46.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:46 smithi172 ceph-mon[100696]: pgmap v864: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 20 MiB/s wr, 838 op/s 2024-09-13T14:56:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:46 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:46 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:56:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:46 smithi172 ceph-mon[100696]: osdmap e275: 8 total, 8 up, 7 in 2024-09-13T14:56:47.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:47.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:47.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:47.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:47.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:47.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:47.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:47.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.554 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:47.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:47.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:47.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:56:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:47.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:47.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:47.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:47.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:47.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:47.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:48.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:48.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:48.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:48.217 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:48.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:48.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:48.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:48.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:48.564 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:48.655 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:48 smithi154 ceph-mon[102314]: pgmap v866: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.0 MiB/s wr, 784 op/s 2024-09-13T14:56:48.655 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:48 smithi154 ceph-mon[107038]: pgmap v866: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.0 MiB/s wr, 784 op/s 2024-09-13T14:56:48.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:48.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:48.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:48.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:48.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:48.942 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:48.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:48.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:48 smithi172 ceph-mon[100696]: pgmap v866: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.0 MiB/s wr, 784 op/s 2024-09-13T14:56:49.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:49.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:49.127 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:56:49.128 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:56:49.162 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:56:49.163 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:56:49.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:49.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:56:49.192 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:49.194 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:56:49.194 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:56:49.234 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:56:49.234 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:56:49.263 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:56:49.263 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:56:49.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:49.297 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:56:49.297 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:56:49.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:49.332 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:56:49.332 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:56:49.367 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:56:49.368 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:56:49.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:49.443 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:49.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:49.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:49.663 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:49.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:49.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:49 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2359379440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:49.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:49 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2359379440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:49.725 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:49.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:49.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:49.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:49.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:49.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:49.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:49.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:49 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2359379440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:56:50.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:50.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:50.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:50.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:50.334 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:50.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:50.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:50.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:50.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:50.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:50.683 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:50.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:50.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:50.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:50.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:50.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:51.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:51.039 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:50 smithi154 ceph-mon[102314]: pgmap v867: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 72 KiB/s wr, 797 op/s 2024-09-13T14:56:51.039 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:50 smithi154 ceph-mon[107038]: pgmap v867: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 72 KiB/s wr, 797 op/s 2024-09-13T14:56:51.056 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:51.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:51.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:51.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:50 smithi172 ceph-mon[100696]: pgmap v867: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 72 KiB/s wr, 797 op/s 2024-09-13T14:56:51.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:51.309 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:51.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:51.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:51.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:51.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:51.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:51.725 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:51.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:51.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:51.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:51.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:51.996 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:56:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:56:51] "GET /metrics HTTP/1.1" 200 36516 "" "Prometheus/2.51.0" 2024-09-13T14:56:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:52.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:52.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:52.127 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:52.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:52.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:56:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.279 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:52.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:52.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:52.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:52.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:52.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:52.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:52.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:52.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:52.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:52.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:52.893 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T14:56:52.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:52.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:52.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:53.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:53.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:53.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:53.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:52 smithi154 ceph-mon[102314]: pgmap v868: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 810 op/s 2024-09-13T14:56:53.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:52 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2529409497' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:56:53.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:52 smithi154 ceph-mon[107038]: pgmap v868: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 810 op/s 2024-09-13T14:56:53.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:52 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2529409497' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:56:53.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:53.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:52 smithi172 ceph-mon[100696]: pgmap v868: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 810 op/s 2024-09-13T14:56:53.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:52 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2529409497' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T14:56:53.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:53.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:53.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:53.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:53.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:53.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:53.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:53.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:53.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:53.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:53.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:54.059 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:54.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:54.186 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:54.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:54.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:53 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2529409497' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:56:54.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:53 smithi154 ceph-mon[102314]: osdmap e276: 8 total, 8 up, 7 in 2024-09-13T14:56:54.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:53 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2529409497' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:56:54.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:53 smithi154 ceph-mon[107038]: osdmap e276: 8 total, 8 up, 7 in 2024-09-13T14:56:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:53 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2529409497' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T14:56:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:53 smithi172 ceph-mon[100696]: osdmap e276: 8 total, 8 up, 7 in 2024-09-13T14:56:54.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:54.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:54.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:54.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:54.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:54.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.737 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:54.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:54.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:54.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:54.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:54.988 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:54.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:55.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:55.097 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:54 smithi154 ceph-mon[102314]: pgmap v870: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 718 op/s 2024-09-13T14:56:55.097 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:54 smithi154 ceph-mon[107038]: pgmap v870: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 718 op/s 2024-09-13T14:56:55.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:55.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:54 smithi172 ceph-mon[100696]: pgmap v870: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 718 op/s 2024-09-13T14:56:55.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:55.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:55.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.508 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:55.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:55.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:55.633 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:55.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T14:56:55.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:55.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:55.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:55.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:55.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:56.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:56.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:56.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:56.159 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:56.194 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:56.195 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:56.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.285 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:56.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:56.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:56.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:56.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:56.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:56.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:56.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:56.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:56.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:56:56.943 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:56.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:56.970 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:56:56.987 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:56:57.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:57.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:57.136 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:56 smithi154 ceph-mon[102314]: pgmap v871: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 838 op/s 2024-09-13T14:56:57.136 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:56 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1983458180' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:57.136 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:56 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:57.136 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:56 smithi154 ceph-mon[107038]: pgmap v871: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 838 op/s 2024-09-13T14:56:57.136 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:56 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1983458180' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:57.136 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:56 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:57.151 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:56:57.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:57.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:56 smithi172 ceph-mon[100696]: pgmap v871: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 838 op/s 2024-09-13T14:56:57.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:56 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1983458180' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:57.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:56 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:56:57.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:57.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:57.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:57.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:57.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:57.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:57.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:57.705 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:57.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:57.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:57.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:57.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:57.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:57.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:58.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:58.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:58.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:57 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:57 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:57 smithi154 ceph-mon[102314]: osdmap e277: 8 total, 8 up, 7 in 2024-09-13T14:56:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:57 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:57 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:57 smithi154 ceph-mon[107038]: osdmap e277: 8 total, 8 up, 7 in 2024-09-13T14:56:58.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:57 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:56:58.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:57 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:56:58.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:57 smithi172 ceph-mon[100696]: osdmap e277: 8 total, 8 up, 7 in 2024-09-13T14:56:58.242 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:58.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:58.367 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:58.469 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:58.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:58.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:56:58.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:56:58.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:56:58.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:56:58.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:58.907 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:58.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:59.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:59.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:56:59.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:59.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:59.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:58 smithi172 ceph-mon[100696]: pgmap v873: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 802 op/s 2024-09-13T14:56:59.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:56:59.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:59.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:56:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.304 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:59.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:56:59.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:56:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:59.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:56:59.453 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:56:59.453 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:56:59.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:58 smithi154 ceph-mon[102314]: pgmap v873: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 802 op/s 2024-09-13T14:56:59.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:56:59.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:59.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:58 smithi154 ceph-mon[107038]: pgmap v873: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 802 op/s 2024-09-13T14:56:59.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:56:59.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:56:59.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:56:59.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:56:59.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:56:59.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:56:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:56:59.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:56:59.842 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:56:59.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:56:59.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:56:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:56:59.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:00.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.096 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:00.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:00.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:57:00.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:00.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:00.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:00.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:00.371 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:56:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:00.372 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:56:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:00.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:56:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:57:00.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:00.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:00.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:00.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:00.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:00.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:00.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:00.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:01.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:01.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:01.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:01.132 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:01.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:01.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:00 smithi154 ceph-mon[102314]: pgmap v874: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-13T14:57:01.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:01 smithi154 ceph-mon[107038]: pgmap v874: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-13T14:57:01.258 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:00 smithi172 ceph-mon[100696]: pgmap v874: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-13T14:57:01.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:01.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:01.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:01.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:01.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:01.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:01.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:01.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:01.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:01.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:01.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:57:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:57:01] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:57:01.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:01.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:02.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:02.096 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:02.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:02.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:02.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:02 smithi172 ceph-mon[100696]: pgmap v875: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 728 op/s 2024-09-13T14:57:02.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:02.329 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:02 smithi154 ceph-mon[102314]: pgmap v875: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 728 op/s 2024-09-13T14:57:02.329 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:02 smithi154 ceph-mon[107038]: pgmap v875: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 728 op/s 2024-09-13T14:57:02.351 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:02.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:02.475 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:02.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:02.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:02.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.745 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:02.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:02.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:02.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:02.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:02.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:03.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:03.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:03.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:03.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:03.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:03.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:03.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:03.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:03.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:03.643 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:03.644 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:03.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:03.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:03.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:03.903 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:03.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:04.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.037 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:04.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:04.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:04.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:04.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:04 smithi172 ceph-mon[100696]: pgmap v876: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:57:04.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:57:04.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:04.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:04 smithi154 ceph-mon[107038]: pgmap v876: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:57:04.298 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:04 smithi154 ceph-mon[102314]: pgmap v876: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 748 op/s 2024-09-13T14:57:04.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:04.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:04.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:04.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:04.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:04.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:04.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:04.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:04.873 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:04.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:04.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:05.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:05.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:57:05.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:05.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:05.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:05.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:05.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:05.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:05.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:05.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:05.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:05.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:05.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:05.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:05.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:05.914 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:05.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:06.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:06.043 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:06.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:06.159 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T14:57:06.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:06.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:06.309 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:57:06.310 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:06.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:06 smithi154 ceph-mon[107038]: pgmap v877: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 730 op/s 2024-09-13T14:57:06.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/477005235' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:57:06.312 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:06 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:57:06.312 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:06 smithi154 ceph-mon[102314]: pgmap v877: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 730 op/s 2024-09-13T14:57:06.313 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/477005235' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:57:06.313 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:06 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:57:06.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:06.355 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:57:06.355 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:06.392 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:06.392 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:57:06.393 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:06.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:06.431 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:57:06.432 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:06.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:06.465 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:57:06.465 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:06 smithi172 ceph-mon[100696]: pgmap v877: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 730 op/s 2024-09-13T14:57:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/477005235' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:57:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:06 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T14:57:06.502 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:57:06.502 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:06.533 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:57:06.533 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:06.562 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:57:06.562 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:06.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:06.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:06.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:06.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:06.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:06.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:06.888 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:06.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:06.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:07.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:07.084 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T14:57:07.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:57:07.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:07.165 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:07.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:07.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:07.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:07.410 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:07 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:57:07.410 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:07 smithi154 ceph-mon[102314]: osdmap e278: 8 total, 8 up, 7 in 2024-09-13T14:57:07.410 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3033248044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:07.410 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:07 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:57:07.410 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:07 smithi154 ceph-mon[107038]: osdmap e278: 8 total, 8 up, 7 in 2024-09-13T14:57:07.410 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3033248044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:07.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:07.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:07 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T14:57:07.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:07 smithi172 ceph-mon[100696]: osdmap e278: 8 total, 8 up, 7 in 2024-09-13T14:57:07.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3033248044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:07.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:07.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:07.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:07.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:07.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:07.814 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:07.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:07.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:07.956 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:08.003 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:08.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:08.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:08.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:08.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:08.261 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:08.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:08.390 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:08.474 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:08 smithi154 ceph-mon[102314]: pgmap v879: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 726 op/s 2024-09-13T14:57:08.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:08 smithi154 ceph-mon[107038]: pgmap v879: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 726 op/s 2024-09-13T14:57:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:08 smithi172 ceph-mon[100696]: pgmap v879: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 726 op/s 2024-09-13T14:57:08.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:08.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:08.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:08.678 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:08.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:08.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:08.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:08.949 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:08.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:09.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:09.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:09.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:09.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:09.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86743 6.e 6:7360c465:::benchmark_data_smithi154_175640_object967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:56.640282+0000", 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 432.59488022099998, 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89457589000000004, 2024-09-13T14:57:09.245 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86743 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640282+0000", 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640282+0000", 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640280+0000", 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640288+0000", 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6480000000000003e-06 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640305+0000", 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6552e-05 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640311+0000", 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8510000000000001e-06 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640368+0000", 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6291e-05 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640375+0000", 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8040000000000002e-06 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.520758+0000", 2024-09-13T14:57:09.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.88038205700000005 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.520867+0000", 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010984000000000001 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,5", 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.520948+0000", 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0227000000000003e-05 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529430+0000", 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084824119999999999 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529509+0000", 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9183000000000004e-05 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534788+0000", 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052784499999999996 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534831+0000", 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3852e-05 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534858+0000", 2024-09-13T14:57:09.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6291999999999998e-05 2024-09-13T14:57:09.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86745 6.e 6:7d92d165:::benchmark_data_smithi154_175640_object969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T14:57:09.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:56.661715+0000", 2024-09-13T14:57:09.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 432.573446671, 2024-09-13T14:57:09.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87325762500000004, 2024-09-13T14:57:09.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86745 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661715+0000", 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661715+0000", 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661713+0000", 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661724+0000", 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0847e-05 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661745+0000", 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0888000000000001e-05 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661755+0000", 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5030000000000003e-06 2024-09-13T14:57:09.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661810+0000", 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5704000000000001e-05 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661820+0000", 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4339999999999998e-06 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.523654+0000", 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86183458800000001 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.523785+0000", 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013034099999999999 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,5", 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.524120+0000", 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033531499999999998 2024-09-13T14:57:09.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529542+0000", 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054218490000000003 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529572+0000", 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0568000000000003e-05 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534919+0000", 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053466360000000001 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534955+0000", 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5871000000000003e-05 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534973+0000", 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8207999999999999e-05 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86780 6.f 6:f38bec78:::benchmark_data_smithi154_175640_object1004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.535104+0000", 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 431.70005761200002, 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.269125445, 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.262 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86780 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535104+0000", 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535104+0000", 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535103+0000", 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535110+0000", 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7750000000000004e-06 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535127+0000", 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7268000000000001e-05 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535137+0000", 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3610000000000007e-06 2024-09-13T14:57:09.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547418+0000", 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012281337999999999 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547433+0000", 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4694e-05 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.781430+0000", 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2339976349999999 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.782221+0000", 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079034699999999995 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.790514+0000", 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082936369999999995 2024-09-13T14:57:09.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.790589+0000", 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4864999999999996e-05 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.790690+0000", 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101266 2024-09-13T14:57:09.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800544+0000", 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098536419999999993 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800588+0000", 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4236000000000001e-05 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800624+0000", 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5914000000000002e-05 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800689+0000", 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4478000000000002e-05 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804230+0000", 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003541317 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86786 6.f 6:f6949069:::benchmark_data_smithi154_175640_object1010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.542654+0000", 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 431.692507648, 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.265779684, 2024-09-13T14:57:09.271 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86786 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542654+0000", 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542654+0000", 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542653+0000", 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542662+0000", 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0550000000000005e-06 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542682+0000", 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0769999999999999e-05 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542691+0000", 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8799999999999997e-06 2024-09-13T14:57:09.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547451+0000", 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047592829999999996 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547455+0000", 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3610000000000003e-06 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.784425+0000", 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2369699439999999 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.785202+0000", 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077720700000000003 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.798177+0000", 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012974741999999999 2024-09-13T14:57:09.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.798247+0000", 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0505000000000003e-05 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.798343+0000", 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5259000000000004e-05 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804963+0000", 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066199359999999999 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804974+0000", 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1936999999999999e-05 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.805006+0000", 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2030999999999999e-05 2024-09-13T14:57:09.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.805045+0000", 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8080999999999997e-05 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.808434+0000", 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033895520000000001 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86796 6.f 6:f0712b29:::benchmark_data_smithi154_175640_object1020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.589684+0000", 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 431.64547855799998, 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.262880338, 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86796 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589684+0000", 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589684+0000", 2024-09-13T14:57:09.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589682+0000", 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589689+0000", 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6080000000000001e-06 2024-09-13T14:57:09.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589699+0000", 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.066e-05 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589705+0000", 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7490000000000004e-06 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589749+0000", 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4161e-05 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589757+0000", 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9030000000000004e-06 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804277+0000", 2024-09-13T14:57:09.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.214520332 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804917+0000", 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063914099999999997 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.820581+0000", 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015664843000000001 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.820661+0000", 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0003e-05 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.820759+0000", 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7700000000000003e-05 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835612+0000", 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014852868999999999 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848754+0000", 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013141664000000001 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848813+0000", 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8989000000000001e-05 2024-09-13T14:57:09.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848868+0000", 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852564+0000", 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036962660000000001 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.289 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86800 6.f 6:f5a03e3c:::benchmark_data_smithi154_175640_object1024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:57:09.289 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.668107+0000", 2024-09-13T14:57:09.289 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 431.567054869, 2024-09-13T14:57:09.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.188158123, 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86800 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668107+0000", 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668107+0000", 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668106+0000", 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668111+0000", 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3040000000000004e-06 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668121+0000", 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9229999999999997e-06 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668127+0000", 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9379999999999997e-06 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668170+0000", 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3164000000000002e-05 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668185+0000", 2024-09-13T14:57:09.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4419e-05 2024-09-13T14:57:09.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.808500+0000", 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1403150879999999 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.809148+0000", 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064803699999999996 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835647+0000", 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026499338000000001 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835732+0000", 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.509e-05 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835828+0000", 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6115999999999999e-05 2024-09-13T14:57:09.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848685+0000", 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012856572 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852637+0000", 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039521770000000003 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852744+0000", 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106524 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852809+0000", 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4993999999999995e-05 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856265+0000", 2024-09-13T14:57:09.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034566649999999998 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86801 6.f 6:fed2b0bf:::benchmark_data_smithi154_175640_object1025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.746781+0000", 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 431.48838097599997, 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.112958592, 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.304 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86801 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746781+0000", 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746781+0000", 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746780+0000", 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746787+0000", 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3760000000000002e-06 2024-09-13T14:57:09.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746801+0000", 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4178999999999999e-05 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746807+0000", 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0020000000000003e-06 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746869+0000", 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1486000000000001e-05 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746882+0000", 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3108999999999999e-05 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.812349+0000", 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0654668549999999 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.813209+0000", 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086008199999999997 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.845548+0000", 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032339097999999997 2024-09-13T14:57:09.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.845678+0000", 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129651 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.845862+0000", 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000184444 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856280+0000", 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010418487000000001 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856339+0000", 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8145e-05 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856384+0000", 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5139999999999998e-05 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856422+0000", 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7910000000000001e-05 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.859740+0000", 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033180459999999998 2024-09-13T14:57:09.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110738 6.e 6:780e6055:::benchmark_data_smithi154_175640_object24962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.142686+0000", 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 292.09247607100002, 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.041401764, 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110738 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142686+0000", 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142686+0000", 2024-09-13T14:57:09.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142685+0000", 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142690+0000", 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000003e-06 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142700+0000", 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0345e-05 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142707+0000", 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9650000000000002e-06 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142751+0000", 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4088999999999997e-05 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142759+0000", 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.294e-06 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.172515+0000", 2024-09-13T14:57:09.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.029756407 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.172632+0000", 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117419 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.173099+0000", 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046696500000000001 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179720+0000", 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066204250000000001 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179777+0000", 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7538999999999999e-05 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.183999+0000", 2024-09-13T14:57:09.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042218719999999998 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.184046+0000", 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7150000000000001e-05 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.184088+0000", 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1390000000000002e-05 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.320 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110739 6.e 6:78427618:::benchmark_data_smithi154_175640_object24963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 292.088430824, 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0540792329999999, 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.321 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110739 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:57:09.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146734+0000", 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8319999999999999e-06 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146743+0000", 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4460000000000001e-06 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146749+0000", 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3790000000000003e-06 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146788+0000", 2024-09-13T14:57:09.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9172999999999999e-05 2024-09-13T14:57:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146795+0000", 2024-09-13T14:57:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3289999999999999e-06 2024-09-13T14:57:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.175941+0000", 2024-09-13T14:57:09.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0291466950000001 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.176007+0000", 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5673999999999995e-05 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.176071+0000", 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4245999999999994e-05 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.181605+0000", 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055336969999999997 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.194960+0000", 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013354428 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200751+0000", 2024-09-13T14:57:09.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057913399999999999 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200782+0000", 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1278000000000001e-05 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200811+0000", 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8384000000000001e-05 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.329 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110766 6.e 6:7c2939fb:::benchmark_data_smithi154_175640_object24990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.287081+0000", 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 291.94808142099998, 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92337370100000005, 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110766 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.330 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287081+0000", 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287081+0000", 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287078+0000", 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287089+0000", 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0464e-05 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287110+0000", 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1403000000000002e-05 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287119+0000", 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2359999999999993e-06 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287185+0000", 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5074999999999996e-05 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287193+0000", 2024-09-13T14:57:09.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7730000000000005e-06 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.177878+0000", 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89068479700000003 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.177987+0000", 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108955 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.178102+0000", 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114571 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.186553+0000", 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084510200000000001 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200852+0000", 2024-09-13T14:57:09.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014299335999999999 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210402+0000", 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095498179999999998 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210431+0000", 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9479999999999999e-05 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210454+0000", 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3074e-05 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110767 6.e 6:7ea46229:::benchmark_data_smithi154_175640_object24991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.297777+0000", 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 291.93738540300001, 2024-09-13T14:57:09.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91276756699999995, 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110767 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297777+0000", 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297777+0000", 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297774+0000", 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297785+0000", 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0249e-05 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297802+0000", 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7652999999999999e-05 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297810+0000", 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9200000000000004e-06 2024-09-13T14:57:09.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297866+0000", 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5358000000000001e-05 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297874+0000", 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3610000000000001e-06 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179816+0000", 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.88194239600000002 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179923+0000", 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106557 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.180036+0000", 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113286 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.191600+0000", 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011563982 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200934+0000", 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093336400000000007 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210492+0000", 2024-09-13T14:57:09.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009557879 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210522+0000", 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0074e-05 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210544+0000", 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2561e-05 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40265 7.1 7:8a6771d1:::benchmark_data_smithi154_225924_object40264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.707202+0000", 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.527960542, 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.19042437100000001, 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40265 2024-09-13T14:57:09.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707202+0000", 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707202+0000", 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707201+0000", 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707233+0000", 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2341000000000001e-05 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707243+0000", 2024-09-13T14:57:09.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2119999999999992e-06 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707248+0000", 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7559999999999996e-06 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708726+0000", 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014780259999999999 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708933+0000", 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000206381 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.709077+0000", 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014402600000000001 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712231+0000", 2024-09-13T14:57:09.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031540700000000001 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713576+0000", 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013451660000000001 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897560+0000", 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18398365 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897600+0000", 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0713999999999997e-05 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897626+0000", 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5602999999999999e-05 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.351 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40266 7.1 7:810b8261:::benchmark_data_smithi154_225924_object40265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.52790689699999, 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.190473265, 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40266 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:57:09.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:09.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707290+0000", 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5580999999999997e-05 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707301+0000", 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0767e-05 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707304+0000", 2024-09-13T14:57:09.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5310000000000002e-06 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710646+0000", 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033420049999999999 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710809+0000", 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162168 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710955+0000", 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014668400000000001 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713638+0000", 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026823720000000001 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.714885+0000", 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001247623 2024-09-13T14:57:09.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897664+0000", 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18277848399999999 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897707+0000", 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3087999999999998e-05 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897728+0000", 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1569999999999998e-05 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40271 7.1 7:8fdb6a0e:::benchmark_data_smithi154_225924_object40270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.708865+0000", 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.52629751400001, 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.188944634, 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.359 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40271 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708865+0000", 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708865+0000", 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708863+0000", 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708962+0000", 2024-09-13T14:57:09.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8227999999999995e-05 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708974+0000", 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2170999999999999e-05 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708981+0000", 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3599999999999998e-06 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712315+0000", 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033341109999999998 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712442+0000", 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012720099999999999 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712587+0000", 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014424199999999999 2024-09-13T14:57:09.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.714890+0000", 2024-09-13T14:57:09.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002303027 2024-09-13T14:57:09.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.714932+0000", 2024-09-13T14:57:09.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2055000000000003e-05 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897762+0000", 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18283049900000001 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897790+0000", 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7555e-05 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897809+0000", 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9446999999999999e-05 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40273 7.1 7:8f2b799d:::benchmark_data_smithi154_225924_object40272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.710113+0000", 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.52504938200001, 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18851839400000001, 2024-09-13T14:57:09.367 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40273 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710113+0000", 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710113+0000", 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710111+0000", 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710163+0000", 2024-09-13T14:57:09.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1301000000000002e-05 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710177+0000", 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3839000000000001e-05 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710184+0000", 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2880000000000004e-06 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713700+0000", 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035157890000000001 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713804+0000", 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104338 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713931+0000", 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012671700000000001 2024-09-13T14:57:09.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.715112+0000", 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011810830000000001 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.715121+0000", 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8300000000000002e-06 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898553+0000", 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18343242800000001 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898597+0000", 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4369999999999997e-05 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898631+0000", 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3698000000000002e-05 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40289 7.1 7:8e157fd9:::benchmark_data_smithi154_225924_object40288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.719678+0000", 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.515483753, 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.179173417, 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.374 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40289 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719678+0000", 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719678+0000", 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719677+0000", 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719727+0000", 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0504999999999998e-05 2024-09-13T14:57:09.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719738+0000", 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0623e-05 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719743+0000", 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1170000000000003e-06 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719778+0000", 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5201999999999999e-05 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719881+0000", 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102693 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719978+0000", 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7448000000000006e-05 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.721474+0000", 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014957880000000001 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.722455+0000", 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00098106600000000006 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898774+0000", 2024-09-13T14:57:09.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.17631929199999999 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898811+0000", 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6491e-05 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898852+0000", 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0799e-05 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.380 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:53103 7.1 7:856a4192:::benchmark_data_smithi154_225924_object53102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:36.663289+0000", 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.571873385000004, 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.124667057, 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.381 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 53103 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663289+0000", 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663289+0000", 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663287+0000", 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663328+0000", 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0697e-05 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663341+0000", 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3220999999999999e-05 2024-09-13T14:57:09.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663349+0000", 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5129999999999999e-06 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663396+0000", 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7114000000000003e-05 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663525+0000", 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128932 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.663665+0000", 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014061100000000001 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.665629+0000", 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019636829999999999 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.666211+0000", 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00058172200000000001 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.787862+0000", 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.121651158 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.787916+0000", 2024-09-13T14:57:09.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4097000000000001e-05 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:36.787956+0000", 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9759000000000003e-05 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:86045 7.1 7:85073c31:::benchmark_data_smithi154_225924_object86044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:56:07.743900+0000", 2024-09-13T14:57:09.387 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.491262589999998, 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10203804599999999, 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86045 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.743900+0000", 2024-09-13T14:57:09.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.743900+0000", 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.743898+0000", 2024-09-13T14:57:09.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.743954+0000", 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5624999999999997e-05 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.743968+0000", 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4134000000000001e-05 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.743976+0000", 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0989999999999999e-06 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.744019+0000", 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2929999999999997e-05 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.744113+0000", 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4611000000000006e-05 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.744221+0000", 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107119 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.746740+0000", 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025199599999999999 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.746823+0000", 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2882999999999999e-05 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.845832+0000", 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.099008503999999997 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.845893+0000", 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0846e-05 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:07.845938+0000", 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4838e-05 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.394 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:88208 7.1 7:8d69ea98:::benchmark_data_smithi154_225924_object88207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:56:10.297362+0000", 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.937799988000002, 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10633068699999999, 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 88208 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297362+0000", 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297362+0000", 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297361+0000", 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:57:09.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297411+0000", 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0322999999999999e-05 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297422+0000", 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1001e-05 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297428+0000", 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3979999999999998e-06 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297460+0000", 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2591e-05 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297557+0000", 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6504999999999998e-05 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.297648+0000", 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1374999999999998e-05 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.299817+0000", 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002169102 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300074+0000", 2024-09-13T14:57:09.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000256655 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.403615+0000", 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103540679 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.403652+0000", 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7277000000000002e-05 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.403693+0000", 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0769999999999998e-05 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:88210 7.1 7:8f42e2b4:::benchmark_data_smithi154_225924_object88209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:56:10.300787+0000", 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.934375543000002, 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.103001911, 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:09.401 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 88210 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300787+0000", 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:09.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300787+0000", 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300786+0000", 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300838+0000", 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2302999999999997e-05 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300847+0000", 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3219999999999999e-06 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300852+0000", 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.118e-06 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.300886+0000", 2024-09-13T14:57:09.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4023999999999998e-05 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.301025+0000", 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000138307 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.301163+0000", 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013845599999999999 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:57:09.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.303362+0000", 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021991290000000002 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.303570+0000", 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020789499999999999 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.403733+0000", 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.100162427 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.403765+0000", 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2064999999999999e-05 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:56:10.403788+0000", 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.37e-05 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:09.407 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:57:09.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:09.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:09.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:09.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:09.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.701 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:09.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:09.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:09.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:09.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:09.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:09.950 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:09.951 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:09.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:09.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:09.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:09.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:09.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:10.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:10.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:10.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:10.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:10.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:10.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:10.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:10.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:10.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:57:10.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:10.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:10.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:10.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:10.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:10.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:10.912 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:10.962 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:10.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:10 smithi172 ceph-mon[100696]: pgmap v880: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:57:11.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:11.069 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:10 smithi154 ceph-mon[102314]: pgmap v880: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:57:11.069 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:10 smithi154 ceph-mon[107038]: pgmap v880: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:57:11.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:11.169 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:11.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:11.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:11.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:11.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:11.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:11.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:11.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:11.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:11.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:11.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:11.865 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:11.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:11.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:11.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:57:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:57:11] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:57:12.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:12.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:12.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:12.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:12.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:12.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:12.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:12.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:12.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:12.670 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:57:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:57:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:137471 7.b e278/260 7:de8b7ea7:::benchmark_data_smithi154_225924_object27207:head v 278'8454, mlcod=278'8451)", 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:12.664002+0000", 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0051426900000000001, 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051751749999999997, 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.664002+0000", 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.664002+0000", 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.664002+0000", 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.664005+0000", 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.495e-06 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.664011+0000", 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9959999999999999e-06 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.664014+0000", 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2930000000000001e-06 2024-09-13T14:57:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.666526+0000", 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025118839999999998 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.666540+0000", 2024-09-13T14:57:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3760999999999999e-05 2024-09-13T14:57:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:137476 7.0 7:05f947eb:::benchmark_data_smithi154_225924_object27212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:12.667189+0000", 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0019557559999999999, 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020520500000000001, 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 137476 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:57:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667189+0000", 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667189+0000", 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667187+0000", 2024-09-13T14:57:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667195+0000", 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1110000000000002e-06 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667212+0000", 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7521e-05 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667221+0000", 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5290000000000008e-06 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667276+0000", 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5353999999999999e-05 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667408+0000", 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013179399999999999 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:57:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:12.667485+0000", 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7169000000000006e-05 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 2 2024-09-13T14:57:12.680 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:57:12.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:12.694 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:12.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:12.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:12.743 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:57:12.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:12.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:12.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:12.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:12 smithi154 ceph-mon[102314]: pgmap v881: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:57:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/289126551' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:12 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:12 smithi154 ceph-mon[107038]: pgmap v881: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:57:12.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/289126551' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.973 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:12 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:12 smithi172 ceph-mon[100696]: pgmap v881: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 715 op/s 2024-09-13T14:57:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/289126551' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:12.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:12 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:57:13.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:13.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:13.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:13.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:13.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:13.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:57:13.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:13.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:13.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:13.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:13.597 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:13.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:13.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:13.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:13.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:13.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[102314]: osdmap e279: 8 total, 8 up, 7 in 2024-09-13T14:57:13.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[107038]: osdmap e279: 8 total, 8 up, 7 in 2024-09-13T14:57:13.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:13.988 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:13.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:13 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:13 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:57:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:13 smithi172 ceph-mon[100696]: osdmap e279: 8 total, 8 up, 7 in 2024-09-13T14:57:13.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:14.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:14.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:14.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:14.261 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:14.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:14.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:14.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:14.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:14.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:14.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:14.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:14.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:14.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:14.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:14.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:14.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:15.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:15.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:15.145 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:14 smithi172 ceph-mon[100696]: pgmap v883: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 654 op/s 2024-09-13T14:57:15.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:15.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:15.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:14 smithi154 ceph-mon[102314]: pgmap v883: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 654 op/s 2024-09-13T14:57:15.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:14 smithi154 ceph-mon[107038]: pgmap v883: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 654 op/s 2024-09-13T14:57:15.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:15.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:15.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.487 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:15.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:15.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:15.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:15.636 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:57:15.636 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:57:15.636 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:57:15.636 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:57:15.636 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:57:15.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:15.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:15.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:15.885 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:15.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:15.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:15.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:16.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:16.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:16.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:16.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:16.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:15 smithi154 ceph-mon[102314]: from='client.55216 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:16.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:15 smithi154 ceph-mon[107038]: from='client.55216 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:15 smithi172 ceph-mon[100696]: from='client.55216 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:16.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:16.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:16.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:16.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:57:16.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:16.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:16.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:16.560 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:16.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:16.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:16.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:16.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:16.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:16.869 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:57:16.872 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:57:16.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:16.979 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:16 smithi154 ceph-mon[102314]: pgmap v884: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 814 op/s 2024-09-13T14:57:16.980 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:16 smithi154 ceph-mon[107038]: pgmap v884: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 814 op/s 2024-09-13T14:57:16.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:16.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:16 smithi172 ceph-mon[100696]: pgmap v884: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 814 op/s 2024-09-13T14:57:17.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:17.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:17.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:17.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:17.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.387 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:17.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:17.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:17.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:17.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:17.676 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:17.692 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:57:17.692 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:17.735 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:57:17.735 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:17.777 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:57:17.777 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:17.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:17.817 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:57:17.818 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:17.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:17.854 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:57:17.854 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:17.855 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:17 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/198036751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:17.855 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:17 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/198036751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:17.888 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:57:17.888 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:17.920 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:57:17.920 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:17.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:17.959 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:57:17.959 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:17.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:17.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:17 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/198036751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:18.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:18.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:18.178 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":883,"stamp":"2024-09-13T14:57:13.083420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5545072637,"num_objects":84567,"num_object_clones":0,"num_object_copies":253701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84567,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":137108,"num_write_kb":7071746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2378885,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124667,"ondisk_log_size":124667,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19074808,"kb_used_data":16383612,"kb_used_omap":32,"kb_used_meta":2691103,"kb_avail":637141256,"statfs":{"total":671965249536,"available":652432646144,"internally_reserved":0,"allocated":16776818688,"data_stored":16766555793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2755690032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,8,8,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-343212032,"num_objects":-5237,"num_object_clones":0,"num_object_copies":-15711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002075"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":439,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.159968+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:57:06.159968+0000","last_peered":"2024-09-13T14:57:06.159968+0000","last_clean":"2024-09-13T14:57:06.159968+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:57:06.159968+0000","last_undegraded":"2024-09-13T14:57:06.159968+0000","last_fullsized":"2024-09-13T14:57:06.159968+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":145,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008688+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:56:57.008688+0000","last_peered":"2024-09-13T14:56:57.008688+0000","last_clean":"2024-09-13T14:56:57.008688+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:56:57.008688+0000","last_undegraded":"2024-09-13T14:56:57.008688+0000","last_fullsized":"2024-09-13T14:56:57.008688+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":115,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997354+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:56:56.997354+0000","last_peered":"2024-09-13T14:56:56.997354+0000","last_clean":"2024-09-13T14:56:56.997354+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:56:56.997354+0000","last_undegraded":"2024-09-13T14:56:56.997354+0000","last_fullsized":"2024-09-13T14:56:56.997354+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":429,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.171549+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:57:06.171549+0000","last_peered":"2024-09-13T14:57:06.171549+0000","last_clean":"2024-09-13T14:57:06.171549+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:57:06.171549+0000","last_undegraded":"2024-09-13T14:57:06.171549+0000","last_fullsized":"2024-09-13T14:57:06.171549+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":429,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.172120+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:57:06.172120+0000","last_peered":"2024-09-13T14:57:06.172120+0000","last_clean":"2024-09-13T14:57:06.172120+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:57:06.172120+0000","last_undegraded":"2024-09-13T14:57:06.172120+0000","last_fullsized":"2024-09-13T14:57:06.172120+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":215,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.010739+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:56:57.010739+0000","last_peered":"2024-09-13T14:56:57.010739+0000","last_clean":"2024-09-13T14:56:57.010739+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:56:57.010739+0000","last_undegraded":"2024-09-13T14:56:57.010739+0000","last_fullsized":"2024-09-13T14:56:57.010739+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":154,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.156119+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:57:06.156119+0000","last_peered":"2024-09-13T14:57:06.156119+0000","last_clean":"2024-09-13T14:57:06.156119+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:57:06.156119+0000","last_undegraded":"2024-09-13T14:57:06.156119+0000","last_fullsized":"2024-09-13T14:57:06.156119+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":192,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997842+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:56:56.997842+0000","last_peered":"2024-09-13T14:56:56.997842+0000","last_clean":"2024-09-13T14:56:56.997842+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:56:56.997842+0000","last_undegraded":"2024-09-13T14:56:56.997842+0000","last_fullsized":"2024-09-13T14:56:56.997842+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"278'8339","reported_seq":8387,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:09.971338+0000","last_change":"2024-09-13T14:54:52.112411+0000","last_active":"2024-09-13T14:57:09.971338+0000","last_peered":"2024-09-13T14:57:09.971338+0000","last_clean":"2024-09-13T14:57:09.971338+0000","last_became_active":"2024-09-13T14:54:52.112143+0000","last_became_peered":"2024-09-13T14:54:52.112143+0000","last_unstale":"2024-09-13T14:57:09.971338+0000","last_undegraded":"2024-09-13T14:57:09.971338+0000","last_fullsized":"2024-09-13T14:57:09.971338+0000","mapping_epoch":260,"log_start":"265'1800","ondisk_log_start":"265'1800","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6539,"log_dups_size":0,"ondisk_log_size":6539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:43.729008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345702400,"num_objects":5275,"num_object_clones":0,"num_object_copies":15825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8339,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":366,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.776676+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:57:12.776676+0000","last_peered":"2024-09-13T14:57:12.776676+0000","last_clean":"2024-09-13T14:57:12.776676+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:57:12.776676+0000","last_undegraded":"2024-09-13T14:57:12.776676+0000","last_fullsized":"2024-09-13T14:57:12.776676+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":143,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:57:04.656775+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:57:04.656775+0000","last_peered":"2024-09-13T14:57:04.656775+0000","last_clean":"2024-09-13T14:57:04.656775+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:57:04.656775+0000","last_undegraded":"2024-09-13T14:57:04.656775+0000","last_fullsized":"2024-09-13T14:57:04.656775+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"278'8422","reported_seq":8468,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:09.633094+0000","last_change":"2024-09-13T14:54:52.111899+0000","last_active":"2024-09-13T14:57:09.633094+0000","last_peered":"2024-09-13T14:57:09.633094+0000","last_clean":"2024-09-13T14:57:09.633094+0000","last_became_active":"2024-09-13T14:54:52.110464+0000","last_became_peered":"2024-09-13T14:54:52.110464+0000","last_unstale":"2024-09-13T14:57:09.633094+0000","last_undegraded":"2024-09-13T14:57:09.633094+0000","last_fullsized":"2024-09-13T14:57:09.633094+0000","mapping_epoch":260,"log_start":"265'1900","ondisk_log_start":"265'1900","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:31.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348651520,"num_objects":5320,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":144,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:57:04.268242+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:57:04.268242+0000","last_peered":"2024-09-13T14:57:04.268242+0000","last_clean":"2024-09-13T14:57:04.268242+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:57:04.268242+0000","last_undegraded":"2024-09-13T14:57:04.268242+0000","last_fullsized":"2024-09-13T14:57:04.268242+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":296,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.010800+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:56:57.010800+0000","last_peered":"2024-09-13T14:56:57.010800+0000","last_clean":"2024-09-13T14:56:57.010800+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:56:57.010800+0000","last_undegraded":"2024-09-13T14:56:57.010800+0000","last_fullsized":"2024-09-13T14:56:57.010800+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"278'8728","reported_seq":8775,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:11.943085+0000","last_change":"2024-09-13T14:54:52.113408+0000","last_active":"2024-09-13T14:57:11.943085+0000","last_peered":"2024-09-13T14:57:11.943085+0000","last_clean":"2024-09-13T14:57:11.943085+0000","last_became_active":"2024-09-13T14:54:52.113069+0000","last_became_peered":"2024-09-13T14:54:52.113069+0000","last_unstale":"2024-09-13T14:57:11.943085+0000","last_undegraded":"2024-09-13T14:57:11.943085+0000","last_fullsized":"2024-09-13T14:57:11.943085+0000","mapping_epoch":260,"log_start":"265'1200","ondisk_log_start":"265'1200","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7528,"log_dups_size":0,"ondisk_log_size":7528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:02.699612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345636864,"num_objects":5274,"num_object_clones":0,"num_object_copies":15822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8728,"num_write_kb":448064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":119,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997505+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:56:56.997505+0000","last_peered":"2024-09-13T14:56:56.997505+0000","last_clean":"2024-09-13T14:56:56.997505+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:56:56.997505+0000","last_undegraded":"2024-09-13T14:56:56.997505+0000","last_fullsized":"2024-09-13T14:56:56.997505+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":76,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.163003+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:57:06.163003+0000","last_peered":"2024-09-13T14:57:06.163003+0000","last_clean":"2024-09-13T14:57:06.163003+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:57:06.163003+0000","last_undegraded":"2024-09-13T14:57:06.163003+0000","last_fullsized":"2024-09-13T14:57:06.163003+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"278'8634","reported_seq":8684,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.775832+0000","last_change":"2024-09-13T14:54:52.116627+0000","last_active":"2024-09-13T14:57:12.775832+0000","last_peered":"2024-09-13T14:57:12.775832+0000","last_clean":"2024-09-13T14:57:12.775832+0000","last_became_active":"2024-09-13T14:54:52.116334+0000","last_became_peered":"2024-09-13T14:54:52.116334+0000","last_unstale":"2024-09-13T14:57:12.775832+0000","last_undegraded":"2024-09-13T14:57:12.775832+0000","last_fullsized":"2024-09-13T14:57:12.775832+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8634,"log_dups_size":0,"ondisk_log_size":8634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:25.091347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340656128,"num_objects":5198,"num_object_clones":0,"num_object_copies":15594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8634,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":131,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997323+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:56:56.997323+0000","last_peered":"2024-09-13T14:56:56.997323+0000","last_clean":"2024-09-13T14:56:56.997323+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:56:56.997323+0000","last_undegraded":"2024-09-13T14:56:56.997323+0000","last_fullsized":"2024-09-13T14:56:56.997323+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":408,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.173266+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:57:06.173266+0000","last_peered":"2024-09-13T14:57:06.173266+0000","last_clean":"2024-09-13T14:57:06.173266+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:57:06.173266+0000","last_undegraded":"2024-09-13T14:57:06.173266+0000","last_fullsized":"2024-09-13T14:57:06.173266+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"278'8443","reported_seq":8491,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:11.978019+0000","last_change":"2024-09-13T14:54:52.117749+0000","last_active":"2024-09-13T14:57:11.978019+0000","last_peered":"2024-09-13T14:57:11.978019+0000","last_clean":"2024-09-13T14:57:11.978019+0000","last_became_active":"2024-09-13T14:54:52.117446+0000","last_became_peered":"2024-09-13T14:54:52.117446+0000","last_unstale":"2024-09-13T14:57:11.978019+0000","last_undegraded":"2024-09-13T14:57:11.978019+0000","last_fullsized":"2024-09-13T14:57:11.978019+0000","mapping_epoch":260,"log_start":"264'900","ondisk_log_start":"264'900","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:34.278909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337444864,"num_objects":5149,"num_object_clones":0,"num_object_copies":15447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":434944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":416,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.155274+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:57:06.155274+0000","last_peered":"2024-09-13T14:57:06.155274+0000","last_clean":"2024-09-13T14:57:06.155274+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:57:06.155274+0000","last_undegraded":"2024-09-13T14:57:06.155274+0000","last_fullsized":"2024-09-13T14:57:06.155274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":262,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008699+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:56:57.008699+0000","last_peered":"2024-09-13T14:56:57.008699+0000","last_clean":"2024-09-13T14:56:57.008699+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:56:57.008699+0000","last_undegraded":"2024-09-13T14:56:57.008699+0000","last_fullsized":"2024-09-13T14:56:57.008699+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"279'8659","reported_seq":8709,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.805103+0000","last_change":"2024-09-13T14:54:52.113650+0000","last_active":"2024-09-13T14:57:12.805103+0000","last_peered":"2024-09-13T14:57:12.805103+0000","last_clean":"2024-09-13T14:57:12.805103+0000","last_became_active":"2024-09-13T14:54:52.113338+0000","last_became_peered":"2024-09-13T14:54:52.113338+0000","last_unstale":"2024-09-13T14:57:12.805103+0000","last_undegraded":"2024-09-13T14:57:12.805103+0000","last_fullsized":"2024-09-13T14:57:12.805103+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8659,"log_dups_size":0,"ondisk_log_size":8659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:53.903772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343867392,"num_objects":5247,"num_object_clones":0,"num_object_copies":15741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8659,"num_write_kb":444992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":163,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.998036+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:56:56.998036+0000","last_peered":"2024-09-13T14:56:56.998036+0000","last_clean":"2024-09-13T14:56:56.998036+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:56:56.998036+0000","last_undegraded":"2024-09-13T14:56:56.998036+0000","last_fullsized":"2024-09-13T14:56:56.998036+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":259,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008878+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:56:57.008878+0000","last_peered":"2024-09-13T14:56:57.008878+0000","last_clean":"2024-09-13T14:56:57.008878+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:56:57.008878+0000","last_undegraded":"2024-09-13T14:56:57.008878+0000","last_fullsized":"2024-09-13T14:56:57.008878+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"278'8415","reported_seq":8444,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:08.961090+0000","last_change":"2024-09-13T14:54:52.113259+0000","last_active":"2024-09-13T14:57:08.961090+0000","last_peered":"2024-09-13T14:57:08.961090+0000","last_clean":"2024-09-13T14:57:08.961090+0000","last_became_active":"2024-09-13T14:54:52.112910+0000","last_became_peered":"2024-09-13T14:54:52.112910+0000","last_unstale":"2024-09-13T14:57:08.961090+0000","last_undegraded":"2024-09-13T14:57:08.961090+0000","last_fullsized":"2024-09-13T14:57:08.961090+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8415,"log_dups_size":0,"ondisk_log_size":8415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:37.906148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351797272,"num_objects":5369,"num_object_clones":0,"num_object_copies":16107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8415,"num_write_kb":441025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":255,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.010833+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:56:57.010833+0000","last_peered":"2024-09-13T14:56:57.010833+0000","last_clean":"2024-09-13T14:56:57.010833+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:56:57.010833+0000","last_undegraded":"2024-09-13T14:56:57.010833+0000","last_fullsized":"2024-09-13T14:56:57.010833+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":289,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.009121+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:56:57.009121+0000","last_peered":"2024-09-13T14:56:57.009121+0000","last_clean":"2024-09-13T14:56:57.009121+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:56:57.009121+0000","last_undegraded":"2024-09-13T14:56:57.009121+0000","last_fullsized":"2024-09-13T14:56:57.009121+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"278'8352","reported_seq":8380,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:08.922131+0000","last_change":"2024-09-13T14:54:52.113761+0000","last_active":"2024-09-13T14:57:08.922131+0000","last_peered":"2024-09-13T14:57:08.922131+0000","last_clean":"2024-09-13T14:57:08.922131+0000","last_became_active":"2024-09-13T14:54:52.113441+0000","last_became_peered":"2024-09-13T14:54:52.113441+0000","last_unstale":"2024-09-13T14:57:08.922131+0000","last_undegraded":"2024-09-13T14:57:08.922131+0000","last_fullsized":"2024-09-13T14:57:08.922131+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:01.375020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340000768,"num_objects":5188,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":155,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.158204+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:57:06.158204+0000","last_peered":"2024-09-13T14:57:06.158204+0000","last_clean":"2024-09-13T14:57:06.158204+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:57:06.158204+0000","last_undegraded":"2024-09-13T14:57:06.158204+0000","last_fullsized":"2024-09-13T14:57:06.158204+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":266,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.776760+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:57:12.776760+0000","last_peered":"2024-09-13T14:57:12.776760+0000","last_clean":"2024-09-13T14:57:12.776760+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:57:12.776760+0000","last_undegraded":"2024-09-13T14:57:12.776760+0000","last_fullsized":"2024-09-13T14:57:12.776760+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"278'8405","reported_seq":8452,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:11.986508+0000","last_change":"2024-09-13T14:54:52.116764+0000","last_active":"2024-09-13T14:57:11.986508+0000","last_peered":"2024-09-13T14:57:11.986508+0000","last_clean":"2024-09-13T14:57:11.986508+0000","last_became_active":"2024-09-13T14:54:52.116144+0000","last_became_peered":"2024-09-13T14:54:52.116144+0000","last_unstale":"2024-09-13T14:57:11.986508+0000","last_undegraded":"2024-09-13T14:57:11.986508+0000","last_fullsized":"2024-09-13T14:57:11.986508+0000","mapping_epoch":260,"log_start":"264'900","ondisk_log_start":"264'900","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:37.832100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335609856,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8405,"num_write_kb":432832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":611,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.171709+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:57:06.171709+0000","last_peered":"2024-09-13T14:57:06.171709+0000","last_clean":"2024-09-13T14:57:06.171709+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:57:06.171709+0000","last_undegraded":"2024-09-13T14:57:06.171709+0000","last_fullsized":"2024-09-13T14:57:06.171709+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":846,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997361+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:56:56.997361+0000","last_peered":"2024-09-13T14:56:56.997361+0000","last_clean":"2024-09-13T14:56:56.997361+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:56:56.997361+0000","last_undegraded":"2024-09-13T14:56:56.997361+0000","last_fullsized":"2024-09-13T14:56:56.997361+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":375,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.161027+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:57:06.161027+0000","last_peered":"2024-09-13T14:57:06.161027+0000","last_clean":"2024-09-13T14:57:06.161027+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:57:06.161027+0000","last_undegraded":"2024-09-13T14:57:06.161027+0000","last_fullsized":"2024-09-13T14:57:06.161027+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":76,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.998447+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:56:56.998447+0000","last_peered":"2024-09-13T14:56:56.998447+0000","last_clean":"2024-09-13T14:56:56.998447+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:56:56.998447+0000","last_undegraded":"2024-09-13T14:56:56.998447+0000","last_fullsized":"2024-09-13T14:56:56.998447+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"278'8435","reported_seq":8485,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.776230+0000","last_change":"2024-09-13T14:54:52.115847+0000","last_active":"2024-09-13T14:57:12.776230+0000","last_peered":"2024-09-13T14:57:12.776230+0000","last_clean":"2024-09-13T14:57:12.776230+0000","last_became_active":"2024-09-13T14:54:52.115534+0000","last_became_peered":"2024-09-13T14:54:52.115534+0000","last_unstale":"2024-09-13T14:57:12.776230+0000","last_undegraded":"2024-09-13T14:57:12.776230+0000","last_fullsized":"2024-09-13T14:57:12.776230+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8435,"log_dups_size":0,"ondisk_log_size":8435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:05.318760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336265216,"num_objects":5131,"num_object_clones":0,"num_object_copies":15393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8435,"num_write_kb":434112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":456,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.170154+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:57:06.170154+0000","last_peered":"2024-09-13T14:57:06.170154+0000","last_clean":"2024-09-13T14:57:06.170154+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:57:06.170154+0000","last_undegraded":"2024-09-13T14:57:06.170154+0000","last_fullsized":"2024-09-13T14:57:06.170154+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":423,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.171993+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:57:06.171993+0000","last_peered":"2024-09-13T14:57:06.171993+0000","last_clean":"2024-09-13T14:57:06.171993+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:57:06.171993+0000","last_undegraded":"2024-09-13T14:57:06.171993+0000","last_fullsized":"2024-09-13T14:57:06.171993+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":433,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.172330+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:57:06.172330+0000","last_peered":"2024-09-13T14:57:06.172330+0000","last_clean":"2024-09-13T14:57:06.172330+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:57:06.172330+0000","last_undegraded":"2024-09-13T14:57:06.172330+0000","last_fullsized":"2024-09-13T14:57:06.172330+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"278'8697","reported_seq":8745,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:11.898612+0000","last_change":"2024-09-13T14:54:52.114072+0000","last_active":"2024-09-13T14:57:11.898612+0000","last_peered":"2024-09-13T14:57:11.898612+0000","last_clean":"2024-09-13T14:57:11.898612+0000","last_became_active":"2024-09-13T14:54:52.113758+0000","last_became_peered":"2024-09-13T14:54:52.113758+0000","last_unstale":"2024-09-13T14:57:11.898612+0000","last_undegraded":"2024-09-13T14:57:11.898612+0000","last_fullsized":"2024-09-13T14:57:11.898612+0000","mapping_epoch":260,"log_start":"265'1100","ondisk_log_start":"265'1100","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:07.961914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355270656,"num_objects":5421,"num_object_clones":0,"num_object_copies":16263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":451776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":961,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.773951+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:57:12.773951+0000","last_peered":"2024-09-13T14:57:12.773951+0000","last_clean":"2024-09-13T14:57:12.773951+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:57:12.773951+0000","last_undegraded":"2024-09-13T14:57:12.773951+0000","last_fullsized":"2024-09-13T14:57:12.773951+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":414,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.170394+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:57:06.170394+0000","last_peered":"2024-09-13T14:57:06.170394+0000","last_clean":"2024-09-13T14:57:06.170394+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:57:06.170394+0000","last_undegraded":"2024-09-13T14:57:06.170394+0000","last_fullsized":"2024-09-13T14:57:06.170394+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":167,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.174424+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:57:06.174424+0000","last_peered":"2024-09-13T14:57:06.174424+0000","last_clean":"2024-09-13T14:57:06.174424+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:57:06.174424+0000","last_undegraded":"2024-09-13T14:57:06.174424+0000","last_fullsized":"2024-09-13T14:57:06.174424+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"278'8576","reported_seq":8604,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:08.937422+0000","last_change":"2024-09-13T14:54:52.113753+0000","last_active":"2024-09-13T14:57:08.937422+0000","last_peered":"2024-09-13T14:57:08.937422+0000","last_clean":"2024-09-13T14:57:08.937422+0000","last_became_active":"2024-09-13T14:54:52.113413+0000","last_became_peered":"2024-09-13T14:54:52.113413+0000","last_unstale":"2024-09-13T14:57:08.937422+0000","last_undegraded":"2024-09-13T14:57:08.937422+0000","last_fullsized":"2024-09-13T14:57:08.937422+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:42.771873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358350848,"num_objects":5468,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8576,"num_write_kb":449408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":134,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997276+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:56:56.997276+0000","last_peered":"2024-09-13T14:56:56.997276+0000","last_clean":"2024-09-13T14:56:56.997276+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:56:56.997276+0000","last_undegraded":"2024-09-13T14:56:56.997276+0000","last_fullsized":"2024-09-13T14:56:56.997276+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":264,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008801+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:56:57.008801+0000","last_peered":"2024-09-13T14:56:57.008801+0000","last_clean":"2024-09-13T14:56:57.008801+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:56:57.008801+0000","last_undegraded":"2024-09-13T14:56:57.008801+0000","last_fullsized":"2024-09-13T14:56:57.008801+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":43,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997183+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:56:56.997183+0000","last_peered":"2024-09-13T14:56:56.997183+0000","last_clean":"2024-09-13T14:56:56.997183+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:56:56.997183+0000","last_undegraded":"2024-09-13T14:56:56.997183+0000","last_fullsized":"2024-09-13T14:56:56.997183+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"278'8494","reported_seq":8520,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:09.248092+0000","last_change":"2024-09-13T14:54:52.107585+0000","last_active":"2024-09-13T14:57:09.248092+0000","last_peered":"2024-09-13T14:57:09.248092+0000","last_clean":"2024-09-13T14:57:09.248092+0000","last_became_active":"2024-09-13T14:54:52.106845+0000","last_became_peered":"2024-09-13T14:54:52.106845+0000","last_unstale":"2024-09-13T14:57:09.248092+0000","last_undegraded":"2024-09-13T14:57:09.248092+0000","last_fullsized":"2024-09-13T14:57:09.248092+0000","mapping_epoch":260,"log_start":"264'900","ondisk_log_start":"264'900","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7594,"log_dups_size":0,"ondisk_log_size":7594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:32.999267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352190464,"num_objects":5374,"num_object_clones":0,"num_object_copies":16122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8494,"num_write_kb":443776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997278+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:56:56.997278+0000","last_peered":"2024-09-13T14:56:56.997278+0000","last_clean":"2024-09-13T14:56:56.997278+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:56:56.997278+0000","last_undegraded":"2024-09-13T14:56:56.997278+0000","last_fullsized":"2024-09-13T14:56:56.997278+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":968,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.177298+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:57:06.177298+0000","last_peered":"2024-09-13T14:57:06.177298+0000","last_clean":"2024-09-13T14:57:06.177298+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:57:06.177298+0000","last_undegraded":"2024-09-13T14:57:06.177298+0000","last_fullsized":"2024-09-13T14:57:06.177298+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":75,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.171455+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:57:06.171455+0000","last_peered":"2024-09-13T14:57:06.171455+0000","last_clean":"2024-09-13T14:57:06.171455+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:57:06.171455+0000","last_undegraded":"2024-09-13T14:57:06.171455+0000","last_fullsized":"2024-09-13T14:57:06.171455+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"278'8410","reported_seq":8438,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:09.631877+0000","last_change":"2024-09-13T14:54:52.114535+0000","last_active":"2024-09-13T14:57:09.631877+0000","last_peered":"2024-09-13T14:57:09.631877+0000","last_clean":"2024-09-13T14:57:09.631877+0000","last_became_active":"2024-09-13T14:54:52.114212+0000","last_became_peered":"2024-09-13T14:54:52.114212+0000","last_unstale":"2024-09-13T14:57:09.631877+0000","last_undegraded":"2024-09-13T14:57:09.631877+0000","last_fullsized":"2024-09-13T14:57:09.631877+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8410,"log_dups_size":0,"ondisk_log_size":8410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:31.266521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349831168,"num_objects":5338,"num_object_clones":0,"num_object_copies":16014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8410,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":420,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.176488+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:57:06.176488+0000","last_peered":"2024-09-13T14:57:06.176488+0000","last_clean":"2024-09-13T14:57:06.176488+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:57:06.176488+0000","last_undegraded":"2024-09-13T14:57:06.176488+0000","last_fullsized":"2024-09-13T14:57:06.176488+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":642,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.172200+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:57:06.172200+0000","last_peered":"2024-09-13T14:57:06.172200+0000","last_clean":"2024-09-13T14:57:06.172200+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:57:06.172200+0000","last_undegraded":"2024-09-13T14:57:06.172200+0000","last_fullsized":"2024-09-13T14:57:06.172200+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":419,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.173193+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:57:06.173193+0000","last_peered":"2024-09-13T14:57:06.173193+0000","last_clean":"2024-09-13T14:57:06.173193+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:57:06.173193+0000","last_undegraded":"2024-09-13T14:57:06.173193+0000","last_fullsized":"2024-09-13T14:57:06.173193+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"278'8457","reported_seq":8505,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:09.638976+0000","last_change":"2024-09-13T14:54:52.112337+0000","last_active":"2024-09-13T14:57:09.638976+0000","last_peered":"2024-09-13T14:57:09.638976+0000","last_clean":"2024-09-13T14:57:09.638976+0000","last_became_active":"2024-09-13T14:54:52.112057+0000","last_became_peered":"2024-09-13T14:54:52.112057+0000","last_unstale":"2024-09-13T14:57:09.638976+0000","last_undegraded":"2024-09-13T14:57:09.638976+0000","last_fullsized":"2024-09-13T14:57:09.638976+0000","mapping_epoch":260,"log_start":"265'1900","ondisk_log_start":"265'1900","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6557,"log_dups_size":0,"ondisk_log_size":6557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:13.030605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351993856,"num_objects":5371,"num_object_clones":0,"num_object_copies":16113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8457,"num_write_kb":442496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":439,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.156359+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:57:06.156359+0000","last_peered":"2024-09-13T14:57:06.156359+0000","last_clean":"2024-09-13T14:57:06.156359+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:57:06.156359+0000","last_undegraded":"2024-09-13T14:57:06.156359+0000","last_fullsized":"2024-09-13T14:57:06.156359+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":870,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.174205+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:57:06.174205+0000","last_peered":"2024-09-13T14:57:06.174205+0000","last_clean":"2024-09-13T14:57:06.174205+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:57:06.174205+0000","last_undegraded":"2024-09-13T14:57:06.174205+0000","last_fullsized":"2024-09-13T14:57:06.174205+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":155,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.157800+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:57:06.157800+0000","last_peered":"2024-09-13T14:57:06.157800+0000","last_clean":"2024-09-13T14:57:06.157800+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:57:06.157800+0000","last_undegraded":"2024-09-13T14:57:06.157800+0000","last_fullsized":"2024-09-13T14:57:06.157800+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"278'8519","reported_seq":8567,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:09.562923+0000","last_change":"2024-09-13T14:54:52.112510+0000","last_active":"2024-09-13T14:57:09.562923+0000","last_peered":"2024-09-13T14:57:09.562923+0000","last_clean":"2024-09-13T14:57:09.562923+0000","last_became_active":"2024-09-13T14:54:52.112262+0000","last_became_peered":"2024-09-13T14:54:52.112262+0000","last_unstale":"2024-09-13T14:57:09.562923+0000","last_undegraded":"2024-09-13T14:57:09.562923+0000","last_fullsized":"2024-09-13T14:57:09.562923+0000","mapping_epoch":260,"log_start":"265'1900","ondisk_log_start":"265'1900","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:57.884110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347013120,"num_objects":5295,"num_object_clones":0,"num_object_copies":15885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":442048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":413,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.172031+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:57:06.172031+0000","last_peered":"2024-09-13T14:57:06.172031+0000","last_clean":"2024-09-13T14:57:06.172031+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:57:06.172031+0000","last_undegraded":"2024-09-13T14:57:06.172031+0000","last_fullsized":"2024-09-13T14:57:06.172031+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":832,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.165610+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:57:06.165610+0000","last_peered":"2024-09-13T14:57:06.165610+0000","last_clean":"2024-09-13T14:57:06.165610+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:57:06.165610+0000","last_undegraded":"2024-09-13T14:57:06.165610+0000","last_fullsized":"2024-09-13T14:57:06.165610+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":266,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.010721+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:56:57.010721+0000","last_peered":"2024-09-13T14:56:57.010721+0000","last_clean":"2024-09-13T14:56:57.010721+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:56:57.010721+0000","last_undegraded":"2024-09-13T14:56:57.010721+0000","last_fullsized":"2024-09-13T14:56:57.010721+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":262,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008754+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:56:57.008754+0000","last_peered":"2024-09-13T14:56:57.008754+0000","last_clean":"2024-09-13T14:56:57.008754+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:56:57.008754+0000","last_undegraded":"2024-09-13T14:56:57.008754+0000","last_fullsized":"2024-09-13T14:56:57.008754+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":310,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008850+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:56:57.008850+0000","last_peered":"2024-09-13T14:56:57.008850+0000","last_clean":"2024-09-13T14:56:57.008850+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:56:57.008850+0000","last_undegraded":"2024-09-13T14:56:57.008850+0000","last_fullsized":"2024-09-13T14:56:57.008850+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":418,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.174251+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:57:06.174251+0000","last_peered":"2024-09-13T14:57:06.174251+0000","last_clean":"2024-09-13T14:57:06.174251+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:57:06.174251+0000","last_undegraded":"2024-09-13T14:57:06.174251+0000","last_fullsized":"2024-09-13T14:57:06.174251+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":258,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.155522+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:57:06.155522+0000","last_peered":"2024-09-13T14:57:06.155522+0000","last_clean":"2024-09-13T14:57:06.155522+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:57:06.155522+0000","last_undegraded":"2024-09-13T14:57:06.155522+0000","last_fullsized":"2024-09-13T14:57:06.155522+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":142,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997398+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:56:56.997398+0000","last_peered":"2024-09-13T14:56:56.997398+0000","last_clean":"2024-09-13T14:56:56.997398+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:56:56.997398+0000","last_undegraded":"2024-09-13T14:56:56.997398+0000","last_fullsized":"2024-09-13T14:56:56.997398+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":395,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.155040+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:57:06.155040+0000","last_peered":"2024-09-13T14:57:06.155040+0000","last_clean":"2024-09-13T14:57:06.155040+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:57:06.155040+0000","last_undegraded":"2024-09-13T14:57:06.155040+0000","last_fullsized":"2024-09-13T14:57:06.155040+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":450,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.174709+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:57:06.174709+0000","last_peered":"2024-09-13T14:57:06.174709+0000","last_clean":"2024-09-13T14:57:06.174709+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:57:06.174709+0000","last_undegraded":"2024-09-13T14:57:06.174709+0000","last_fullsized":"2024-09-13T14:57:06.174709+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":446,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.160718+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:57:06.160718+0000","last_peered":"2024-09-13T14:57:06.160718+0000","last_clean":"2024-09-13T14:57:06.160718+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:57:06.160718+0000","last_undegraded":"2024-09-13T14:57:06.160718+0000","last_fullsized":"2024-09-13T14:57:06.160718+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":172,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.998213+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:56:56.998213+0000","last_peered":"2024-09-13T14:56:56.998213+0000","last_clean":"2024-09-13T14:56:56.998213+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:56:56.998213+0000","last_undegraded":"2024-09-13T14:56:56.998213+0000","last_fullsized":"2024-09-13T14:56:56.998213+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":413,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.160933+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:57:06.160933+0000","last_peered":"2024-09-13T14:57:06.160933+0000","last_clean":"2024-09-13T14:57:06.160933+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:57:06.160933+0000","last_undegraded":"2024-09-13T14:57:06.160933+0000","last_fullsized":"2024-09-13T14:57:06.160933+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":430,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.172257+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:57:06.172257+0000","last_peered":"2024-09-13T14:57:06.172257+0000","last_clean":"2024-09-13T14:57:06.172257+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:57:06.172257+0000","last_undegraded":"2024-09-13T14:57:06.172257+0000","last_fullsized":"2024-09-13T14:57:06.172257+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":398,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.175215+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:57:06.175215+0000","last_peered":"2024-09-13T14:57:06.175215+0000","last_clean":"2024-09-13T14:57:06.175215+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:57:06.175215+0000","last_undegraded":"2024-09-13T14:57:06.175215+0000","last_fullsized":"2024-09-13T14:57:06.175215+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":262,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.155379+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:57:06.155379+0000","last_peered":"2024-09-13T14:57:06.155379+0000","last_clean":"2024-09-13T14:57:06.155379+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:57:06.155379+0000","last_undegraded":"2024-09-13T14:57:06.155379+0000","last_fullsized":"2024-09-13T14:57:06.155379+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":260,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008644+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:56:57.008644+0000","last_peered":"2024-09-13T14:56:57.008644+0000","last_clean":"2024-09-13T14:56:57.008644+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:56:57.008644+0000","last_undegraded":"2024-09-13T14:56:57.008644+0000","last_fullsized":"2024-09-13T14:56:57.008644+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":159,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.777932+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:57:12.777932+0000","last_peered":"2024-09-13T14:57:12.777932+0000","last_clean":"2024-09-13T14:57:12.777932+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:57:12.777932+0000","last_undegraded":"2024-09-13T14:57:12.777932+0000","last_fullsized":"2024-09-13T14:57:12.777932+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":422,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.173143+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:57:06.173143+0000","last_peered":"2024-09-13T14:57:06.173143+0000","last_clean":"2024-09-13T14:57:06.173143+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:57:06.173143+0000","last_undegraded":"2024-09-13T14:57:06.173143+0000","last_fullsized":"2024-09-13T14:57:06.173143+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":270,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008679+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:56:57.008679+0000","last_peered":"2024-09-13T14:56:57.008679+0000","last_clean":"2024-09-13T14:56:57.008679+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:56:57.008679+0000","last_undegraded":"2024-09-13T14:56:57.008679+0000","last_fullsized":"2024-09-13T14:56:57.008679+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":117,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997281+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:56:56.997281+0000","last_peered":"2024-09-13T14:56:56.997281+0000","last_clean":"2024-09-13T14:56:56.997281+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:56:56.997281+0000","last_undegraded":"2024-09-13T14:56:56.997281+0000","last_fullsized":"2024-09-13T14:56:56.997281+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":258,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008553+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:56:57.008553+0000","last_peered":"2024-09-13T14:56:57.008553+0000","last_clean":"2024-09-13T14:56:57.008553+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:56:57.008553+0000","last_undegraded":"2024-09-13T14:56:57.008553+0000","last_fullsized":"2024-09-13T14:56:57.008553+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":119,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:56.997434+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:56:56.997434+0000","last_peered":"2024-09-13T14:56:56.997434+0000","last_clean":"2024-09-13T14:56:56.997434+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:56:56.997434+0000","last_undegraded":"2024-09-13T14:56:56.997434+0000","last_fullsized":"2024-09-13T14:56:56.997434+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":416,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.155356+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:57:06.155356+0000","last_peered":"2024-09-13T14:57:06.155356+0000","last_clean":"2024-09-13T14:57:06.155356+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:57:06.155356+0000","last_undegraded":"2024-09-13T14:57:06.155356+0000","last_fullsized":"2024-09-13T14:57:06.155356+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":433,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.155842+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:57:06.155842+0000","last_peered":"2024-09-13T14:57:06.155842+0000","last_clean":"2024-09-13T14:57:06.155842+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:57:06.155842+0000","last_undegraded":"2024-09-13T14:57:06.155842+0000","last_fullsized":"2024-09-13T14:57:06.155842+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":288,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-13T14:56:57.008422+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:56:57.008422+0000","last_peered":"2024-09-13T14:56:57.008422+0000","last_clean":"2024-09-13T14:56:57.008422+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:56:57.008422+0000","last_undegraded":"2024-09-13T14:56:57.008422+0000","last_fullsized":"2024-09-13T14:56:57.008422+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":486,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.174617+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:57:06.174617+0000","last_peered":"2024-09-13T14:57:06.174617+0000","last_clean":"2024-09-13T14:57:06.174617+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:57:06.174617+0000","last_undegraded":"2024-09-13T14:57:06.174617+0000","last_fullsized":"2024-09-13T14:57:06.174617+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5540282392,"num_objects":84539,"num_object_clones":0,"num_object_copies":253617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84539,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135985,"num_write_kb":7056705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16657100800,"data_stored":16657088584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123485,"ondisk_log_size":123485,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234357,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495993,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2131648,"kb_used_data":1725620,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91613504,"statfs":{"total":95995035648,"available":93812228096,"internally_reserved":0,"allocated":1767034880,"data_stored":1765564060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":415758791},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757628,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1999660,"kb_used_data":1707684,"kb_used_omap":1,"kb_used_meta":291966,"kb_avail":91745492,"statfs":{"total":95995035648,"available":93947383808,"internally_reserved":0,"allocated":1748668416,"data_stored":1747209535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":298973639},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019268,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2436408,"kb_used_data":2084528,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91308744,"statfs":{"total":95995035648,"available":93500153856,"internally_reserved":0,"allocated":2134556672,"data_stored":2133089985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468153,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3833900,"kb_used_data":3414948,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":89911252,"statfs":{"total":95995035648,"available":92069122048,"internally_reserved":0,"allocated":3496906752,"data_stored":3495447645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":428993700},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631683,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2796028,"kb_used_data":2335348,"kb_used_omap":8,"kb_used_meta":460663,"kb_avail":90949124,"statfs":{"total":95995035648,"available":93131902976,"internally_reserved":0,"allocated":2391396352,"data_stored":2389926095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":471719903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795205,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2782516,"kb_used_data":2399596,"kb_used_omap":7,"kb_used_meta":382904,"kb_avail":90962636,"statfs":{"total":95995035648,"available":93145739264,"internally_reserved":0,"allocated":2457186304,"data_stored":2455719454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":392094322},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958736,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3094648,"kb_used_data":2715888,"kb_used_omap":7,"kb_used_meta":378744,"kb_avail":90650504,"statfs":{"total":95995035648,"available":92826116096,"internally_reserved":0,"allocated":2781069312,"data_stored":2779599019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":387834343},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2748715008,"data_stored":2748710936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2446528512,"data_stored":2446524440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2369519616,"data_stored":2369519616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3479830528,"data_stored":3479830528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2118385664,"data_stored":2118385664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1741553664,"data_stored":1741553664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1752567808,"data_stored":1752563736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:57:18.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T14:57:18.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:18.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:18.308 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:18.308 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:18.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:18.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:18.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:18.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:18.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:18.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:18.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:57:18.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:18.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:18.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:18.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:18.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:18.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:18 smithi154 ceph-mon[102314]: pgmap v885: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-09-13T14:57:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:18 smithi154 ceph-mon[107038]: pgmap v885: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-09-13T14:57:18.981 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:18 smithi172 ceph-mon[100696]: pgmap v885: 89 pgs: 89 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-09-13T14:57:19.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:57:19.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:19.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:19.135 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:19.155 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:19.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:19.240 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:19.291 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:19.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:19.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:19.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:19.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:19.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.745 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:19.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:19.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:19.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:19.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:19.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:20.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:20.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:20.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:20.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:20.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:20.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:20.444 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:20.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:20.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:20.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:20.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:20.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:20.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:20.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:20.977 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:20.977 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:20.990 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:20 smithi172 ceph-mon[100696]: pgmap v886: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 721 op/s 2024-09-13T14:57:20.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:20.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:21.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:57:21.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:21.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:21.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:20 smithi154 ceph-mon[102314]: pgmap v886: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 721 op/s 2024-09-13T14:57:21.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:20 smithi154 ceph-mon[107038]: pgmap v886: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 721 op/s 2024-09-13T14:57:21.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:21.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:21.408 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:21.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:21.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:21.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:21.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:21.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:57:21.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:21.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:21.899 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:21.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:21.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:21.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:21.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:57:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:57:21] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:57:22.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:22.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:22.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:22.196 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:22.297 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:22.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:22.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:22.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:22.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:22.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:22.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:22.712 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:22.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.840 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:22.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:22.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:22.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:22.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:23.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:23.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:22 smithi172 ceph-mon[100696]: pgmap v887: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 713 op/s 2024-09-13T14:57:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:23.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:23.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:22 smithi154 ceph-mon[102314]: pgmap v887: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 713 op/s 2024-09-13T14:57:23.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:22 smithi154 ceph-mon[107038]: pgmap v887: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 713 op/s 2024-09-13T14:57:23.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:23.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:23.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:23.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.524 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:23.626 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:23.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:23.725 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:23.725 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:23.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:23.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:23.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:23.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:23.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:24.192 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:24.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:57:24.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:24.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:24.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:24.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:24.623 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:24.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:24.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:24.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:24.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:24.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:24.976 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:24.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:24.985 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:24 smithi154 ceph-mon[102314]: pgmap v888: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 752 op/s 2024-09-13T14:57:24.985 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:24 smithi154 ceph-mon[107038]: pgmap v888: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 752 op/s 2024-09-13T14:57:25.032 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:24 smithi172 ceph-mon[100696]: pgmap v888: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 752 op/s 2024-09-13T14:57:25.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:25.110 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:25.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:25.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:25.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:25.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:25.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:25.510 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T14:57:25.511 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T14:57:25.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:25.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:25.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:25.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:25.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:25.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:25.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:25.934 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:57:25.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:26.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:26.050 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:25 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3009227039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:26.050 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:25 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:26.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:26.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:25 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3009227039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:26.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:25 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:26.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:25 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3009227039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:26.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:25 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:26.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:26.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:26.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:26.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:26.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:26.474 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:26.474 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:26.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:26.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:26.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:26.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:26.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:26.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:26.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:26.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:26.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:26.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:27.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:27.056 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:27.114 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:26 smithi172 ceph-mon[100696]: pgmap v889: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-09-13T14:57:27.115 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:26 smithi172 ceph-mon[100696]: from='client.45497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:27.115 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:26 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:27.115 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:26 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T14:57:27.115 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:26 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:27.115 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:26 smithi172 ceph-mon[100696]: osdmap e280: 8 total, 8 up, 7 in 2024-09-13T14:57:27.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:57:27.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:27.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:27.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[102314]: pgmap v889: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-09-13T14:57:27.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[102314]: from='client.45497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:27.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:27.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T14:57:27.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:27.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[102314]: osdmap e280: 8 total, 8 up, 7 in 2024-09-13T14:57:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[107038]: pgmap v889: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-09-13T14:57:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[107038]: from='client.45497 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T14:57:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:57:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T14:57:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:27.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:26 smithi154 ceph-mon[107038]: osdmap e280: 8 total, 8 up, 7 in 2024-09-13T14:57:27.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.324 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:27.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:27.390 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:27.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:27.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:27.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:27.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:27.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:27.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:27.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:27.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:27.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:27.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:27.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:28.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:28.097 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:28.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:28.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:28.368 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:28.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:28.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:28.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:28.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:28.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:28.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:28.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:28.916 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:28.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:29.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:29.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:29.138 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:28 smithi172 ceph-mon[100696]: pgmap v891: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:57:29.140 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:29.140 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:29.153 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:29.153 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:29.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T14:57:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:29.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T14:57:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:28 smithi154 ceph-mon[102314]: pgmap v891: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:57:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:29.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:28 smithi154 ceph-mon[107038]: pgmap v891: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 720 op/s 2024-09-13T14:57:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:29.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:29.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:29.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:29.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:29.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:29.786 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-09-13T14:57:25.088029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4943845373,"num_objects":75393,"num_object_clones":0,"num_object_copies":226179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75393,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":146282,"num_write_kb":7071746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":2378885,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128041,"ondisk_log_size":128041,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17253000,"kb_used_data":14539664,"kb_used_omap":32,"kb_used_meta":2713183,"kb_avail":638963064,"statfs":{"total":671965249536,"available":654298177536,"internally_reserved":0,"allocated":14888615936,"data_stored":14878325230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33232,"internal_metadata":2778299952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-601227264,"num_objects":-9174,"num_object_clones":0,"num_object_copies":-27522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004609"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":441,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.760555+0000","last_change":"2024-09-13T14:49:58.585615+0000","last_active":"2024-09-13T14:57:12.760555+0000","last_peered":"2024-09-13T14:57:12.760555+0000","last_clean":"2024-09-13T14:57:12.760555+0000","last_became_active":"2024-09-13T14:49:58.585252+0000","last_became_peered":"2024-09-13T14:49:58.585252+0000","last_unstale":"2024-09-13T14:57:12.760555+0000","last_undegraded":"2024-09-13T14:57:12.760555+0000","last_fullsized":"2024-09-13T14:57:12.760555+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:28.073679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":147,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.759903+0000","last_change":"2024-09-13T14:48:02.090598+0000","last_active":"2024-09-13T14:57:12.759903+0000","last_peered":"2024-09-13T14:57:12.759903+0000","last_clean":"2024-09-13T14:57:12.759903+0000","last_became_active":"2024-09-13T14:48:02.078150+0000","last_became_peered":"2024-09-13T14:48:02.078150+0000","last_unstale":"2024-09-13T14:57:12.759903+0000","last_undegraded":"2024-09-13T14:57:12.759903+0000","last_fullsized":"2024-09-13T14:57:12.759903+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:41:27.454990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":116,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.168725+0000","last_change":"2024-09-13T14:48:21.095479+0000","last_active":"2024-09-13T14:57:06.168725+0000","last_peered":"2024-09-13T14:57:06.168725+0000","last_clean":"2024-09-13T14:57:06.168725+0000","last_became_active":"2024-09-13T14:48:21.095016+0000","last_became_peered":"2024-09-13T14:48:21.095016+0000","last_unstale":"2024-09-13T14:57:06.168725+0000","last_undegraded":"2024-09-13T14:57:06.168725+0000","last_fullsized":"2024-09-13T14:57:06.168725+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:37.312639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":431,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.766836+0000","last_change":"2024-09-13T14:50:19.801564+0000","last_active":"2024-09-13T14:57:12.766836+0000","last_peered":"2024-09-13T14:57:12.766836+0000","last_clean":"2024-09-13T14:57:12.766836+0000","last_became_active":"2024-09-13T14:50:19.801183+0000","last_became_peered":"2024-09-13T14:50:19.801183+0000","last_unstale":"2024-09-13T14:57:12.766836+0000","last_undegraded":"2024-09-13T14:57:12.766836+0000","last_fullsized":"2024-09-13T14:57:12.766836+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:14.155969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":431,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.768151+0000","last_change":"2024-09-13T14:50:19.802372+0000","last_active":"2024-09-13T14:57:12.768151+0000","last_peered":"2024-09-13T14:57:12.768151+0000","last_clean":"2024-09-13T14:57:12.768151+0000","last_became_active":"2024-09-13T14:50:19.802079+0000","last_became_peered":"2024-09-13T14:50:19.802079+0000","last_unstale":"2024-09-13T14:57:12.768151+0000","last_undegraded":"2024-09-13T14:57:12.768151+0000","last_fullsized":"2024-09-13T14:57:12.768151+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:45.190272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":217,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.759597+0000","last_change":"2024-09-13T14:48:02.075585+0000","last_active":"2024-09-13T14:57:12.759597+0000","last_peered":"2024-09-13T14:57:12.759597+0000","last_clean":"2024-09-13T14:57:12.759597+0000","last_became_active":"2024-09-13T14:48:02.075226+0000","last_became_peered":"2024-09-13T14:48:02.075226+0000","last_unstale":"2024-09-13T14:57:12.759597+0000","last_undegraded":"2024-09-13T14:57:12.759597+0000","last_fullsized":"2024-09-13T14:57:12.759597+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:54.888255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":156,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.751254+0000","last_change":"2024-09-13T14:50:19.805191+0000","last_active":"2024-09-13T14:57:12.751254+0000","last_peered":"2024-09-13T14:57:12.751254+0000","last_clean":"2024-09-13T14:57:12.751254+0000","last_became_active":"2024-09-13T14:50:19.804887+0000","last_became_peered":"2024-09-13T14:50:19.804887+0000","last_unstale":"2024-09-13T14:57:12.751254+0000","last_undegraded":"2024-09-13T14:57:12.751254+0000","last_fullsized":"2024-09-13T14:57:12.751254+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:12.506597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":193,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.162946+0000","last_change":"2024-09-13T14:48:21.109537+0000","last_active":"2024-09-13T14:57:06.162946+0000","last_peered":"2024-09-13T14:57:06.162946+0000","last_clean":"2024-09-13T14:57:06.162946+0000","last_became_active":"2024-09-13T14:48:21.096816+0000","last_became_peered":"2024-09-13T14:48:21.096816+0000","last_unstale":"2024-09-13T14:57:06.162946+0000","last_undegraded":"2024-09-13T14:57:06.162946+0000","last_fullsized":"2024-09-13T14:57:06.162946+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:28:46.661168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"279'8997","reported_seq":9047,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:24.934153+0000","last_change":"2024-09-13T14:54:52.112411+0000","last_active":"2024-09-13T14:57:24.934153+0000","last_peered":"2024-09-13T14:57:24.934153+0000","last_clean":"2024-09-13T14:57:24.934153+0000","last_became_active":"2024-09-13T14:54:52.112143+0000","last_became_peered":"2024-09-13T14:54:52.112143+0000","last_unstale":"2024-09-13T14:57:24.934153+0000","last_undegraded":"2024-09-13T14:57:24.934153+0000","last_fullsized":"2024-09-13T14:57:24.934153+0000","mapping_epoch":260,"log_start":"265'2400","ondisk_log_start":"265'2400","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6597,"log_dups_size":0,"ondisk_log_size":6597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:43.729008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302579712,"num_objects":4617,"num_object_clones":0,"num_object_copies":13851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8997,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":366,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.776676+0000","last_change":"2024-09-13T14:48:02.097287+0000","last_active":"2024-09-13T14:57:12.776676+0000","last_peered":"2024-09-13T14:57:12.776676+0000","last_clean":"2024-09-13T14:57:12.776676+0000","last_became_active":"2024-09-13T14:48:02.085439+0000","last_became_peered":"2024-09-13T14:48:02.085439+0000","last_unstale":"2024-09-13T14:57:12.776676+0000","last_undegraded":"2024-09-13T14:57:12.776676+0000","last_fullsized":"2024-09-13T14:57:12.776676+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:15.483033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":144,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.168802+0000","last_change":"2024-09-13T14:50:37.949934+0000","last_active":"2024-09-13T14:57:06.168802+0000","last_peered":"2024-09-13T14:57:06.168802+0000","last_clean":"2024-09-13T14:57:06.168802+0000","last_became_active":"2024-09-13T14:50:37.949581+0000","last_became_peered":"2024-09-13T14:50:37.949581+0000","last_unstale":"2024-09-13T14:57:06.168802+0000","last_undegraded":"2024-09-13T14:57:06.168802+0000","last_fullsized":"2024-09-13T14:57:06.168802+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:58.870943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"279'9094","reported_seq":9144,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:24.644992+0000","last_change":"2024-09-13T14:54:52.111899+0000","last_active":"2024-09-13T14:57:24.644992+0000","last_peered":"2024-09-13T14:57:24.644992+0000","last_clean":"2024-09-13T14:57:24.644992+0000","last_became_active":"2024-09-13T14:54:52.110464+0000","last_became_peered":"2024-09-13T14:54:52.110464+0000","last_unstale":"2024-09-13T14:57:24.644992+0000","last_undegraded":"2024-09-13T14:57:24.644992+0000","last_fullsized":"2024-09-13T14:57:24.644992+0000","mapping_epoch":260,"log_start":"265'2500","ondisk_log_start":"265'2500","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:24:31.052094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304611328,"num_objects":4648,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9094,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":145,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.148359+0000","last_change":"2024-09-13T14:48:02.107804+0000","last_active":"2024-09-13T14:57:06.148359+0000","last_peered":"2024-09-13T14:57:06.148359+0000","last_clean":"2024-09-13T14:57:06.148359+0000","last_became_active":"2024-09-13T14:48:02.083967+0000","last_became_peered":"2024-09-13T14:48:02.083967+0000","last_unstale":"2024-09-13T14:57:06.148359+0000","last_undegraded":"2024-09-13T14:57:06.148359+0000","last_fullsized":"2024-09-13T14:57:06.148359+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:47.170329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":297,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.147437+0000","last_change":"2024-09-13T14:50:19.795962+0000","last_active":"2024-09-13T14:57:06.147437+0000","last_peered":"2024-09-13T14:57:06.147437+0000","last_clean":"2024-09-13T14:57:06.147437+0000","last_became_active":"2024-09-13T14:50:19.795612+0000","last_became_peered":"2024-09-13T14:50:19.795612+0000","last_unstale":"2024-09-13T14:57:06.147437+0000","last_undegraded":"2024-09-13T14:57:06.147437+0000","last_fullsized":"2024-09-13T14:57:06.147437+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:01:11.089727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"279'9177","reported_seq":9226,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:21.994019+0000","last_change":"2024-09-13T14:54:52.113408+0000","last_active":"2024-09-13T14:57:21.994019+0000","last_peered":"2024-09-13T14:57:21.994019+0000","last_clean":"2024-09-13T14:57:21.994019+0000","last_became_active":"2024-09-13T14:54:52.113069+0000","last_became_peered":"2024-09-13T14:54:52.113069+0000","last_unstale":"2024-09-13T14:57:21.994019+0000","last_undegraded":"2024-09-13T14:57:21.994019+0000","last_fullsized":"2024-09-13T14:57:21.994019+0000","mapping_epoch":260,"log_start":"265'1600","ondisk_log_start":"265'1600","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7577,"log_dups_size":0,"ondisk_log_size":7577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:10:02.699612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316211200,"num_objects":4825,"num_object_clones":0,"num_object_copies":14475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9177,"num_write_kb":448064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":121,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.775811+0000","last_change":"2024-09-13T14:48:21.105750+0000","last_active":"2024-09-13T14:57:12.775811+0000","last_peered":"2024-09-13T14:57:12.775811+0000","last_clean":"2024-09-13T14:57:12.775811+0000","last_became_active":"2024-09-13T14:48:21.095432+0000","last_became_peered":"2024-09-13T14:48:21.095432+0000","last_unstale":"2024-09-13T14:57:12.775811+0000","last_undegraded":"2024-09-13T14:57:12.775811+0000","last_fullsized":"2024-09-13T14:57:12.775811+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:29.863428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":77,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:08.964812+0000","last_change":"2024-09-13T14:50:37.948635+0000","last_active":"2024-09-13T14:57:08.964812+0000","last_peered":"2024-09-13T14:57:08.964812+0000","last_clean":"2024-09-13T14:57:08.964812+0000","last_became_active":"2024-09-13T14:50:37.948378+0000","last_became_peered":"2024-09-13T14:50:37.948378+0000","last_unstale":"2024-09-13T14:57:08.964812+0000","last_undegraded":"2024-09-13T14:57:08.964812+0000","last_fullsized":"2024-09-13T14:57:08.964812+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:04.194850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"279'9078","reported_seq":9128,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:22.796739+0000","last_change":"2024-09-13T14:54:52.116627+0000","last_active":"2024-09-13T14:57:22.796739+0000","last_peered":"2024-09-13T14:57:22.796739+0000","last_clean":"2024-09-13T14:57:22.796739+0000","last_became_active":"2024-09-13T14:54:52.116334+0000","last_became_peered":"2024-09-13T14:54:52.116334+0000","last_unstale":"2024-09-13T14:57:22.796739+0000","last_undegraded":"2024-09-13T14:57:22.796739+0000","last_fullsized":"2024-09-13T14:57:22.796739+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":9078,"log_dups_size":0,"ondisk_log_size":9078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:25.091347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311558144,"num_objects":4754,"num_object_clones":0,"num_object_copies":14262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":133,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.775825+0000","last_change":"2024-09-13T14:50:22.037340+0000","last_active":"2024-09-13T14:57:12.775825+0000","last_peered":"2024-09-13T14:57:12.775825+0000","last_clean":"2024-09-13T14:57:12.775825+0000","last_became_active":"2024-09-13T14:50:22.037056+0000","last_became_peered":"2024-09-13T14:50:22.037056+0000","last_unstale":"2024-09-13T14:57:12.775825+0000","last_undegraded":"2024-09-13T14:57:12.775825+0000","last_fullsized":"2024-09-13T14:57:12.775825+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:36.306475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":410,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.767365+0000","last_change":"2024-09-13T14:40:36.875334+0000","last_active":"2024-09-13T14:57:12.767365+0000","last_peered":"2024-09-13T14:57:12.767365+0000","last_clean":"2024-09-13T14:57:12.767365+0000","last_became_active":"2024-09-13T14:40:36.875023+0000","last_became_peered":"2024-09-13T14:40:36.875023+0000","last_unstale":"2024-09-13T14:57:12.767365+0000","last_undegraded":"2024-09-13T14:57:12.767365+0000","last_fullsized":"2024-09-13T14:57:12.767365+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:48.481880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"279'8895","reported_seq":8945,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:22.001587+0000","last_change":"2024-09-13T14:54:52.117749+0000","last_active":"2024-09-13T14:57:22.001587+0000","last_peered":"2024-09-13T14:57:22.001587+0000","last_clean":"2024-09-13T14:57:22.001587+0000","last_became_active":"2024-09-13T14:54:52.117446+0000","last_became_peered":"2024-09-13T14:54:52.117446+0000","last_unstale":"2024-09-13T14:57:22.001587+0000","last_undegraded":"2024-09-13T14:57:22.001587+0000","last_fullsized":"2024-09-13T14:57:22.001587+0000","mapping_epoch":260,"log_start":"265'1300","ondisk_log_start":"265'1300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:34.278909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307822592,"num_objects":4697,"num_object_clones":0,"num_object_copies":14091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8895,"num_write_kb":434944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":419,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.749163+0000","last_change":"2024-09-13T14:41:41.491211+0000","last_active":"2024-09-13T14:57:12.749163+0000","last_peered":"2024-09-13T14:57:12.749163+0000","last_clean":"2024-09-13T14:57:12.749163+0000","last_became_active":"2024-09-13T14:41:41.490308+0000","last_became_peered":"2024-09-13T14:41:41.490308+0000","last_unstale":"2024-09-13T14:57:12.749163+0000","last_undegraded":"2024-09-13T14:57:12.749163+0000","last_fullsized":"2024-09-13T14:57:12.749163+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:32.503955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":263,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.148129+0000","last_change":"2024-09-13T14:44:54.720533+0000","last_active":"2024-09-13T14:57:06.148129+0000","last_peered":"2024-09-13T14:57:06.148129+0000","last_clean":"2024-09-13T14:57:06.148129+0000","last_became_active":"2024-09-13T14:44:54.717064+0000","last_became_peered":"2024-09-13T14:44:54.717064+0000","last_unstale":"2024-09-13T14:57:06.148129+0000","last_undegraded":"2024-09-13T14:57:06.148129+0000","last_fullsized":"2024-09-13T14:57:06.148129+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:25.546000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"279'9072","reported_seq":9121,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:22.810896+0000","last_change":"2024-09-13T14:54:52.113650+0000","last_active":"2024-09-13T14:57:22.810896+0000","last_peered":"2024-09-13T14:57:22.810896+0000","last_clean":"2024-09-13T14:57:22.810896+0000","last_became_active":"2024-09-13T14:54:52.113338+0000","last_became_peered":"2024-09-13T14:54:52.113338+0000","last_unstale":"2024-09-13T14:57:22.810896+0000","last_undegraded":"2024-09-13T14:57:22.810896+0000","last_fullsized":"2024-09-13T14:57:22.810896+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":9072,"log_dups_size":0,"ondisk_log_size":9072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:53.903772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316801024,"num_objects":4834,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9072,"num_write_kb":444992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":164,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.163163+0000","last_change":"2024-09-13T14:44:54.725392+0000","last_active":"2024-09-13T14:57:06.163163+0000","last_peered":"2024-09-13T14:57:06.163163+0000","last_clean":"2024-09-13T14:57:06.163163+0000","last_became_active":"2024-09-13T14:44:54.725067+0000","last_became_peered":"2024-09-13T14:44:54.725067+0000","last_unstale":"2024-09-13T14:57:06.163163+0000","last_undegraded":"2024-09-13T14:57:06.163163+0000","last_fullsized":"2024-09-13T14:57:06.163163+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:04.127347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":260,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.148129+0000","last_change":"2024-09-13T14:41:09.648738+0000","last_active":"2024-09-13T14:57:06.148129+0000","last_peered":"2024-09-13T14:57:06.148129+0000","last_clean":"2024-09-13T14:57:06.148129+0000","last_became_active":"2024-09-13T14:41:09.648462+0000","last_became_peered":"2024-09-13T14:41:09.648462+0000","last_unstale":"2024-09-13T14:57:06.148129+0000","last_undegraded":"2024-09-13T14:57:06.148129+0000","last_fullsized":"2024-09-13T14:57:06.148129+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:18.241160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"279'9139","reported_seq":9169,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:23.964014+0000","last_change":"2024-09-13T14:54:52.113259+0000","last_active":"2024-09-13T14:57:23.964014+0000","last_peered":"2024-09-13T14:57:23.964014+0000","last_clean":"2024-09-13T14:57:23.964014+0000","last_became_active":"2024-09-13T14:54:52.112910+0000","last_became_peered":"2024-09-13T14:54:52.112910+0000","last_unstale":"2024-09-13T14:57:23.964014+0000","last_undegraded":"2024-09-13T14:57:23.964014+0000","last_fullsized":"2024-09-13T14:57:23.964014+0000","mapping_epoch":260,"log_start":"264'300","ondisk_log_start":"264'300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:37.906148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304349208,"num_objects":4645,"num_object_clones":0,"num_object_copies":13935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9139,"num_write_kb":441025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.147498+0000","last_change":"2024-09-13T14:48:02.085609+0000","last_active":"2024-09-13T14:57:06.147498+0000","last_peered":"2024-09-13T14:57:06.147498+0000","last_clean":"2024-09-13T14:57:06.147498+0000","last_became_active":"2024-09-13T14:48:02.085344+0000","last_became_peered":"2024-09-13T14:48:02.085344+0000","last_unstale":"2024-09-13T14:57:06.147498+0000","last_undegraded":"2024-09-13T14:57:06.147498+0000","last_fullsized":"2024-09-13T14:57:06.147498+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:40.151384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":290,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.148424+0000","last_change":"2024-09-13T14:50:37.952124+0000","last_active":"2024-09-13T14:57:06.148424+0000","last_peered":"2024-09-13T14:57:06.148424+0000","last_clean":"2024-09-13T14:57:06.148424+0000","last_became_active":"2024-09-13T14:50:37.951837+0000","last_became_peered":"2024-09-13T14:50:37.951837+0000","last_unstale":"2024-09-13T14:57:06.148424+0000","last_undegraded":"2024-09-13T14:57:06.148424+0000","last_fullsized":"2024-09-13T14:57:06.148424+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:04:22.390109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"279'8998","reported_seq":9026,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:23.959246+0000","last_change":"2024-09-13T14:54:52.113761+0000","last_active":"2024-09-13T14:57:23.959246+0000","last_peered":"2024-09-13T14:57:23.959246+0000","last_clean":"2024-09-13T14:57:23.959246+0000","last_became_active":"2024-09-13T14:54:52.113441+0000","last_became_peered":"2024-09-13T14:54:52.113441+0000","last_unstale":"2024-09-13T14:57:23.959246+0000","last_undegraded":"2024-09-13T14:57:23.959246+0000","last_fullsized":"2024-09-13T14:57:23.959246+0000","mapping_epoch":260,"log_start":"264'100","ondisk_log_start":"264'100","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:00:01.375020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297664512,"num_objects":4542,"num_object_clones":0,"num_object_copies":13626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8998,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":157,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.763254+0000","last_change":"2024-09-13T14:50:19.808417+0000","last_active":"2024-09-13T14:57:12.763254+0000","last_peered":"2024-09-13T14:57:12.763254+0000","last_clean":"2024-09-13T14:57:12.763254+0000","last_became_active":"2024-09-13T14:50:19.808137+0000","last_became_peered":"2024-09-13T14:50:19.808137+0000","last_unstale":"2024-09-13T14:57:12.763254+0000","last_undegraded":"2024-09-13T14:57:12.763254+0000","last_fullsized":"2024-09-13T14:57:12.763254+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:15.506098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":266,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.776760+0000","last_change":"2024-09-13T14:47:14.129643+0000","last_active":"2024-09-13T14:57:12.776760+0000","last_peered":"2024-09-13T14:57:12.776760+0000","last_clean":"2024-09-13T14:57:12.776760+0000","last_became_active":"2024-09-13T14:47:14.129255+0000","last_became_peered":"2024-09-13T14:47:14.129255+0000","last_unstale":"2024-09-13T14:57:12.776760+0000","last_undegraded":"2024-09-13T14:57:12.776760+0000","last_fullsized":"2024-09-13T14:57:12.776760+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:01.150220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"279'8843","reported_seq":8893,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:22.004603+0000","last_change":"2024-09-13T14:54:52.116764+0000","last_active":"2024-09-13T14:57:22.004603+0000","last_peered":"2024-09-13T14:57:22.004603+0000","last_clean":"2024-09-13T14:57:22.004603+0000","last_became_active":"2024-09-13T14:54:52.116144+0000","last_became_peered":"2024-09-13T14:54:52.116144+0000","last_unstale":"2024-09-13T14:57:22.004603+0000","last_undegraded":"2024-09-13T14:57:22.004603+0000","last_fullsized":"2024-09-13T14:57:22.004603+0000","mapping_epoch":260,"log_start":"265'1300","ondisk_log_start":"265'1300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:37.832100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8843,"num_write_kb":432832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":613,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.774473+0000","last_change":"2024-09-13T14:48:05.133368+0000","last_active":"2024-09-13T14:57:12.774473+0000","last_peered":"2024-09-13T14:57:12.774473+0000","last_clean":"2024-09-13T14:57:12.774473+0000","last_became_active":"2024-09-13T14:48:05.132564+0000","last_became_peered":"2024-09-13T14:48:05.132564+0000","last_unstale":"2024-09-13T14:57:12.774473+0000","last_undegraded":"2024-09-13T14:57:12.774473+0000","last_fullsized":"2024-09-13T14:57:12.774473+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:41.203323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":848,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.775609+0000","last_change":"2024-09-13T14:48:52.025934+0000","last_active":"2024-09-13T14:57:12.775609+0000","last_peered":"2024-09-13T14:57:12.775609+0000","last_clean":"2024-09-13T14:57:12.775609+0000","last_became_active":"2024-09-13T14:48:52.025562+0000","last_became_peered":"2024-09-13T14:48:52.025562+0000","last_unstale":"2024-09-13T14:57:12.775609+0000","last_undegraded":"2024-09-13T14:57:12.775609+0000","last_fullsized":"2024-09-13T14:57:12.775609+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:32:26.990562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":377,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.759452+0000","last_change":"2024-09-13T14:50:19.798517+0000","last_active":"2024-09-13T14:57:12.759452+0000","last_peered":"2024-09-13T14:57:12.759452+0000","last_clean":"2024-09-13T14:57:12.759452+0000","last_became_active":"2024-09-13T14:50:19.798154+0000","last_became_peered":"2024-09-13T14:50:19.798154+0000","last_unstale":"2024-09-13T14:57:12.759452+0000","last_undegraded":"2024-09-13T14:57:12.759452+0000","last_fullsized":"2024-09-13T14:57:12.759452+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:40:18.242169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":77,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.163246+0000","last_change":"2024-09-13T14:50:37.976079+0000","last_active":"2024-09-13T14:57:06.163246+0000","last_peered":"2024-09-13T14:57:06.163246+0000","last_clean":"2024-09-13T14:57:06.163246+0000","last_became_active":"2024-09-13T14:50:37.975745+0000","last_became_peered":"2024-09-13T14:50:37.975745+0000","last_unstale":"2024-09-13T14:57:06.163246+0000","last_undegraded":"2024-09-13T14:57:06.163246+0000","last_fullsized":"2024-09-13T14:57:06.163246+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:32.729239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"279'8897","reported_seq":8947,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:22.815946+0000","last_change":"2024-09-13T14:54:52.115847+0000","last_active":"2024-09-13T14:57:22.815946+0000","last_peered":"2024-09-13T14:57:22.815946+0000","last_clean":"2024-09-13T14:57:22.815946+0000","last_became_active":"2024-09-13T14:54:52.115534+0000","last_became_peered":"2024-09-13T14:54:52.115534+0000","last_unstale":"2024-09-13T14:57:22.815946+0000","last_undegraded":"2024-09-13T14:57:22.815946+0000","last_fullsized":"2024-09-13T14:57:22.815946+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8897,"log_dups_size":0,"ondisk_log_size":8897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:05.318760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305987584,"num_objects":4669,"num_object_clones":0,"num_object_copies":14007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8897,"num_write_kb":434112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":458,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.775436+0000","last_change":"2024-09-13T14:48:02.117375+0000","last_active":"2024-09-13T14:57:12.775436+0000","last_peered":"2024-09-13T14:57:12.775436+0000","last_clean":"2024-09-13T14:57:12.775436+0000","last_became_active":"2024-09-13T14:48:02.098807+0000","last_became_peered":"2024-09-13T14:48:02.098807+0000","last_unstale":"2024-09-13T14:57:12.775436+0000","last_undegraded":"2024-09-13T14:57:12.775436+0000","last_fullsized":"2024-09-13T14:57:12.775436+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:12.664946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":425,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.767375+0000","last_change":"2024-09-13T14:50:19.794534+0000","last_active":"2024-09-13T14:57:12.767375+0000","last_peered":"2024-09-13T14:57:12.767375+0000","last_clean":"2024-09-13T14:57:12.767375+0000","last_became_active":"2024-09-13T14:50:19.794134+0000","last_became_peered":"2024-09-13T14:50:19.794134+0000","last_unstale":"2024-09-13T14:57:12.767375+0000","last_undegraded":"2024-09-13T14:57:12.767375+0000","last_fullsized":"2024-09-13T14:57:12.767375+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:39.221814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":436,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.768420+0000","last_change":"2024-09-13T14:50:07.556661+0000","last_active":"2024-09-13T14:57:12.768420+0000","last_peered":"2024-09-13T14:57:12.768420+0000","last_clean":"2024-09-13T14:57:12.768420+0000","last_became_active":"2024-09-13T14:50:07.556386+0000","last_became_peered":"2024-09-13T14:50:07.556386+0000","last_unstale":"2024-09-13T14:57:12.768420+0000","last_undegraded":"2024-09-13T14:57:12.768420+0000","last_fullsized":"2024-09-13T14:57:12.768420+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:43:30.745246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"279'9163","reported_seq":9213,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:21.937080+0000","last_change":"2024-09-13T14:54:52.114072+0000","last_active":"2024-09-13T14:57:21.937080+0000","last_peered":"2024-09-13T14:57:21.937080+0000","last_clean":"2024-09-13T14:57:21.937080+0000","last_became_active":"2024-09-13T14:54:52.113758+0000","last_became_peered":"2024-09-13T14:54:52.113758+0000","last_unstale":"2024-09-13T14:57:21.937080+0000","last_undegraded":"2024-09-13T14:57:21.937080+0000","last_fullsized":"2024-09-13T14:57:21.937080+0000","mapping_epoch":260,"log_start":"265'1600","ondisk_log_start":"265'1600","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:07.961914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324730880,"num_objects":4955,"num_object_clones":0,"num_object_copies":14865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":451776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":961,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.773951+0000","last_change":"2024-09-13T14:50:19.795396+0000","last_active":"2024-09-13T14:57:12.773951+0000","last_peered":"2024-09-13T14:57:12.773951+0000","last_clean":"2024-09-13T14:57:12.773951+0000","last_became_active":"2024-09-13T14:50:19.794843+0000","last_became_peered":"2024-09-13T14:50:19.794843+0000","last_unstale":"2024-09-13T14:57:12.773951+0000","last_undegraded":"2024-09-13T14:57:12.773951+0000","last_fullsized":"2024-09-13T14:57:12.773951+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:00.377193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":416,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.775661+0000","last_change":"2024-09-13T14:44:54.713730+0000","last_active":"2024-09-13T14:57:12.775661+0000","last_peered":"2024-09-13T14:57:12.775661+0000","last_clean":"2024-09-13T14:57:12.775661+0000","last_became_active":"2024-09-13T14:44:54.713382+0000","last_became_peered":"2024-09-13T14:44:54.713382+0000","last_unstale":"2024-09-13T14:57:12.775661+0000","last_undegraded":"2024-09-13T14:57:12.775661+0000","last_fullsized":"2024-09-13T14:57:12.775661+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:49.948673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":169,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.774977+0000","last_change":"2024-09-13T14:49:58.520413+0000","last_active":"2024-09-13T14:57:12.774977+0000","last_peered":"2024-09-13T14:57:12.774977+0000","last_clean":"2024-09-13T14:57:12.774977+0000","last_became_active":"2024-09-13T14:49:58.520059+0000","last_became_peered":"2024-09-13T14:49:58.520059+0000","last_unstale":"2024-09-13T14:57:12.774977+0000","last_undegraded":"2024-09-13T14:57:12.774977+0000","last_fullsized":"2024-09-13T14:57:12.774977+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:16.403617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"279'9266","reported_seq":9295,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:23.928235+0000","last_change":"2024-09-13T14:54:52.113753+0000","last_active":"2024-09-13T14:57:23.928235+0000","last_peered":"2024-09-13T14:57:23.928235+0000","last_clean":"2024-09-13T14:57:23.928235+0000","last_became_active":"2024-09-13T14:54:52.113413+0000","last_became_peered":"2024-09-13T14:54:52.113413+0000","last_unstale":"2024-09-13T14:57:23.928235+0000","last_undegraded":"2024-09-13T14:57:23.928235+0000","last_fullsized":"2024-09-13T14:57:23.928235+0000","mapping_epoch":260,"log_start":"264'400","ondisk_log_start":"264'400","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:42.771873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313131008,"num_objects":4778,"num_object_clones":0,"num_object_copies":14334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9266,"num_write_kb":449408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":135,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.168746+0000","last_change":"2024-09-13T14:48:21.156737+0000","last_active":"2024-09-13T14:57:06.168746+0000","last_peered":"2024-09-13T14:57:06.168746+0000","last_clean":"2024-09-13T14:57:06.168746+0000","last_became_active":"2024-09-13T14:48:21.099230+0000","last_became_peered":"2024-09-13T14:48:21.099230+0000","last_unstale":"2024-09-13T14:57:06.168746+0000","last_undegraded":"2024-09-13T14:57:06.168746+0000","last_fullsized":"2024-09-13T14:57:06.168746+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:14.195164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":265,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.147988+0000","last_change":"2024-09-13T14:44:54.730518+0000","last_active":"2024-09-13T14:57:06.147988+0000","last_peered":"2024-09-13T14:57:06.147988+0000","last_clean":"2024-09-13T14:57:06.147988+0000","last_became_active":"2024-09-13T14:44:54.730172+0000","last_became_peered":"2024-09-13T14:44:54.730172+0000","last_unstale":"2024-09-13T14:57:06.147988+0000","last_undegraded":"2024-09-13T14:57:06.147988+0000","last_fullsized":"2024-09-13T14:57:06.147988+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:08.172919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":45,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.775907+0000","last_change":"2024-09-13T14:53:14.599369+0000","last_active":"2024-09-13T14:57:12.775907+0000","last_peered":"2024-09-13T14:57:12.775907+0000","last_clean":"2024-09-13T14:57:12.775907+0000","last_became_active":"2024-09-13T14:53:14.599146+0000","last_became_peered":"2024-09-13T14:53:14.599146+0000","last_unstale":"2024-09-13T14:57:12.775907+0000","last_undegraded":"2024-09-13T14:57:12.775907+0000","last_fullsized":"2024-09-13T14:57:12.775907+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:29.506234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"279'9168","reported_seq":9197,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:24.252149+0000","last_change":"2024-09-13T14:54:52.107585+0000","last_active":"2024-09-13T14:57:24.252149+0000","last_peered":"2024-09-13T14:57:24.252149+0000","last_clean":"2024-09-13T14:57:24.252149+0000","last_became_active":"2024-09-13T14:54:52.106845+0000","last_became_peered":"2024-09-13T14:54:52.106845+0000","last_unstale":"2024-09-13T14:57:24.252149+0000","last_undegraded":"2024-09-13T14:57:24.252149+0000","last_fullsized":"2024-09-13T14:57:24.252149+0000","mapping_epoch":260,"log_start":"265'1600","ondisk_log_start":"265'1600","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:21:32.999267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308019200,"num_objects":4700,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":443776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":135,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.168725+0000","last_change":"2024-09-13T14:50:19.811651+0000","last_active":"2024-09-13T14:57:06.168725+0000","last_peered":"2024-09-13T14:57:06.168725+0000","last_clean":"2024-09-13T14:57:06.168725+0000","last_became_active":"2024-09-13T14:50:19.811345+0000","last_became_peered":"2024-09-13T14:50:19.811345+0000","last_unstale":"2024-09-13T14:57:06.168725+0000","last_undegraded":"2024-09-13T14:57:06.168725+0000","last_fullsized":"2024-09-13T14:57:06.168725+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:49.528779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":971,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.772711+0000","last_change":"2024-09-13T14:48:05.137866+0000","last_active":"2024-09-13T14:57:12.772711+0000","last_peered":"2024-09-13T14:57:12.772711+0000","last_clean":"2024-09-13T14:57:12.772711+0000","last_became_active":"2024-09-13T14:48:05.137363+0000","last_became_peered":"2024-09-13T14:48:05.137363+0000","last_unstale":"2024-09-13T14:57:12.772711+0000","last_undegraded":"2024-09-13T14:57:12.772711+0000","last_fullsized":"2024-09-13T14:57:12.772711+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:19.968283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":77,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.766729+0000","last_change":"2024-09-13T14:53:14.548908+0000","last_active":"2024-09-13T14:57:12.766729+0000","last_peered":"2024-09-13T14:57:12.766729+0000","last_clean":"2024-09-13T14:57:12.766729+0000","last_became_active":"2024-09-13T14:53:14.547225+0000","last_became_peered":"2024-09-13T14:53:14.547225+0000","last_unstale":"2024-09-13T14:57:12.766729+0000","last_undegraded":"2024-09-13T14:57:12.766729+0000","last_fullsized":"2024-09-13T14:57:12.766729+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:16:48.104218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"279'9054","reported_seq":9083,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:24.655447+0000","last_change":"2024-09-13T14:54:52.114535+0000","last_active":"2024-09-13T14:57:24.655447+0000","last_peered":"2024-09-13T14:57:24.655447+0000","last_clean":"2024-09-13T14:57:24.655447+0000","last_became_active":"2024-09-13T14:54:52.114212+0000","last_became_peered":"2024-09-13T14:54:52.114212+0000","last_unstale":"2024-09-13T14:57:24.655447+0000","last_undegraded":"2024-09-13T14:57:24.655447+0000","last_fullsized":"2024-09-13T14:57:24.655447+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":9054,"log_dups_size":0,"ondisk_log_size":9054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:31.266521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307625984,"num_objects":4694,"num_object_clones":0,"num_object_copies":14082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":423,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.772622+0000","last_change":"2024-09-13T14:50:22.036572+0000","last_active":"2024-09-13T14:57:12.772622+0000","last_peered":"2024-09-13T14:57:12.772622+0000","last_clean":"2024-09-13T14:57:12.772622+0000","last_became_active":"2024-09-13T14:50:22.036181+0000","last_became_peered":"2024-09-13T14:50:22.036181+0000","last_unstale":"2024-09-13T14:57:12.772622+0000","last_undegraded":"2024-09-13T14:57:12.772622+0000","last_fullsized":"2024-09-13T14:57:12.772622+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:12.445249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":644,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.773581+0000","last_change":"2024-09-13T14:48:02.113507+0000","last_active":"2024-09-13T14:57:12.773581+0000","last_peered":"2024-09-13T14:57:12.773581+0000","last_clean":"2024-09-13T14:57:12.773581+0000","last_became_active":"2024-09-13T14:48:02.085702+0000","last_became_peered":"2024-09-13T14:48:02.085702+0000","last_unstale":"2024-09-13T14:57:12.773581+0000","last_undegraded":"2024-09-13T14:57:12.773581+0000","last_fullsized":"2024-09-13T14:57:12.773581+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:56.487823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":421,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.767279+0000","last_change":"2024-09-13T14:48:21.099614+0000","last_active":"2024-09-13T14:57:12.767279+0000","last_peered":"2024-09-13T14:57:12.767279+0000","last_clean":"2024-09-13T14:57:12.767279+0000","last_became_active":"2024-09-13T14:48:21.099306+0000","last_became_peered":"2024-09-13T14:48:21.099306+0000","last_unstale":"2024-09-13T14:57:12.767279+0000","last_undegraded":"2024-09-13T14:57:12.767279+0000","last_fullsized":"2024-09-13T14:57:12.767279+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:16.307063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"279'9130","reported_seq":9180,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:24.677527+0000","last_change":"2024-09-13T14:54:52.112337+0000","last_active":"2024-09-13T14:57:24.677527+0000","last_peered":"2024-09-13T14:57:24.677527+0000","last_clean":"2024-09-13T14:57:24.677527+0000","last_became_active":"2024-09-13T14:54:52.112057+0000","last_became_peered":"2024-09-13T14:54:52.112057+0000","last_unstale":"2024-09-13T14:57:24.677527+0000","last_undegraded":"2024-09-13T14:57:24.677527+0000","last_fullsized":"2024-09-13T14:57:24.677527+0000","mapping_epoch":260,"log_start":"265'2600","ondisk_log_start":"265'2600","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:05:13.030605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307888128,"num_objects":4698,"num_object_clones":0,"num_object_copies":14094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":442496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":441,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.752650+0000","last_change":"2024-09-13T14:48:21.112535+0000","last_active":"2024-09-13T14:57:12.752650+0000","last_peered":"2024-09-13T14:57:12.752650+0000","last_clean":"2024-09-13T14:57:12.752650+0000","last_became_active":"2024-09-13T14:48:21.099438+0000","last_became_peered":"2024-09-13T14:48:21.099438+0000","last_unstale":"2024-09-13T14:57:12.752650+0000","last_undegraded":"2024-09-13T14:57:12.752650+0000","last_fullsized":"2024-09-13T14:57:12.752650+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:35:59.027735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":872,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.772701+0000","last_change":"2024-09-13T14:46:58.059560+0000","last_active":"2024-09-13T14:57:12.772701+0000","last_peered":"2024-09-13T14:57:12.772701+0000","last_clean":"2024-09-13T14:57:12.772701+0000","last_became_active":"2024-09-13T14:46:58.059242+0000","last_became_peered":"2024-09-13T14:46:58.059242+0000","last_unstale":"2024-09-13T14:57:12.772701+0000","last_undegraded":"2024-09-13T14:57:12.772701+0000","last_fullsized":"2024-09-13T14:57:12.772701+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:34.007219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":157,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.762917+0000","last_change":"2024-09-13T14:50:19.811322+0000","last_active":"2024-09-13T14:57:12.762917+0000","last_peered":"2024-09-13T14:57:12.762917+0000","last_clean":"2024-09-13T14:57:12.762917+0000","last_became_active":"2024-09-13T14:50:19.810476+0000","last_became_peered":"2024-09-13T14:50:19.810476+0000","last_unstale":"2024-09-13T14:57:12.762917+0000","last_undegraded":"2024-09-13T14:57:12.762917+0000","last_fullsized":"2024-09-13T14:57:12.762917+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:44.956575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"279'9188","reported_seq":9238,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:24.562306+0000","last_change":"2024-09-13T14:54:52.112510+0000","last_active":"2024-09-13T14:57:24.562306+0000","last_peered":"2024-09-13T14:57:24.562306+0000","last_clean":"2024-09-13T14:57:24.562306+0000","last_became_active":"2024-09-13T14:54:52.112262+0000","last_became_peered":"2024-09-13T14:54:52.112262+0000","last_unstale":"2024-09-13T14:57:24.562306+0000","last_undegraded":"2024-09-13T14:57:24.562306+0000","last_fullsized":"2024-09-13T14:57:24.562306+0000","mapping_epoch":260,"log_start":"265'2600","ondisk_log_start":"265'2600","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:54:51.092497+0000","last_clean_scrub_stamp":"2024-09-13T14:54:51.092497+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:57.884110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303169536,"num_objects":4626,"num_object_clones":0,"num_object_copies":13878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9188,"num_write_kb":442048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":416,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.768098+0000","last_change":"2024-09-13T14:40:36.877114+0000","last_active":"2024-09-13T14:57:12.768098+0000","last_peered":"2024-09-13T14:57:12.768098+0000","last_clean":"2024-09-13T14:57:12.768098+0000","last_became_active":"2024-09-13T14:40:36.876418+0000","last_became_peered":"2024-09-13T14:40:36.876418+0000","last_unstale":"2024-09-13T14:57:12.768098+0000","last_undegraded":"2024-09-13T14:57:12.768098+0000","last_fullsized":"2024-09-13T14:57:12.768098+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:34.408709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":833,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:08.964936+0000","last_change":"2024-09-13T14:48:05.135133+0000","last_active":"2024-09-13T14:57:08.964936+0000","last_peered":"2024-09-13T14:57:08.964936+0000","last_clean":"2024-09-13T14:57:08.964936+0000","last_became_active":"2024-09-13T14:48:05.134710+0000","last_became_peered":"2024-09-13T14:48:05.134710+0000","last_unstale":"2024-09-13T14:57:08.964936+0000","last_undegraded":"2024-09-13T14:57:08.964936+0000","last_fullsized":"2024-09-13T14:57:08.964936+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:45.153463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":268,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.759632+0000","last_change":"2024-09-13T14:48:21.095252+0000","last_active":"2024-09-13T14:57:12.759632+0000","last_peered":"2024-09-13T14:57:12.759632+0000","last_clean":"2024-09-13T14:57:12.759632+0000","last_became_active":"2024-09-13T14:48:21.094955+0000","last_became_peered":"2024-09-13T14:48:21.094955+0000","last_unstale":"2024-09-13T14:57:12.759632+0000","last_undegraded":"2024-09-13T14:57:12.759632+0000","last_fullsized":"2024-09-13T14:57:12.759632+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:23.167737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":263,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.147963+0000","last_change":"2024-09-13T14:44:54.717891+0000","last_active":"2024-09-13T14:57:06.147963+0000","last_peered":"2024-09-13T14:57:06.147963+0000","last_clean":"2024-09-13T14:57:06.147963+0000","last_became_active":"2024-09-13T14:44:54.717536+0000","last_became_peered":"2024-09-13T14:44:54.717536+0000","last_unstale":"2024-09-13T14:57:06.147963+0000","last_undegraded":"2024-09-13T14:57:06.147963+0000","last_fullsized":"2024-09-13T14:57:06.147963+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:00.449660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":311,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.148274+0000","last_change":"2024-09-13T14:48:02.099214+0000","last_active":"2024-09-13T14:57:06.148274+0000","last_peered":"2024-09-13T14:57:06.148274+0000","last_clean":"2024-09-13T14:57:06.148274+0000","last_became_active":"2024-09-13T14:48:02.078353+0000","last_became_peered":"2024-09-13T14:48:02.078353+0000","last_unstale":"2024-09-13T14:57:06.148274+0000","last_undegraded":"2024-09-13T14:57:06.148274+0000","last_fullsized":"2024-09-13T14:57:06.148274+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:36.637506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14268,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":420,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.772068+0000","last_change":"2024-09-13T14:50:37.955043+0000","last_active":"2024-09-13T14:57:12.772068+0000","last_peered":"2024-09-13T14:57:12.772068+0000","last_clean":"2024-09-13T14:57:12.772068+0000","last_became_active":"2024-09-13T14:50:37.954690+0000","last_became_peered":"2024-09-13T14:50:37.954690+0000","last_unstale":"2024-09-13T14:57:12.772068+0000","last_undegraded":"2024-09-13T14:57:12.772068+0000","last_fullsized":"2024-09-13T14:57:12.772068+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:38.028648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":260,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.760379+0000","last_change":"2024-09-13T14:48:02.106017+0000","last_active":"2024-09-13T14:57:12.760379+0000","last_peered":"2024-09-13T14:57:12.760379+0000","last_clean":"2024-09-13T14:57:12.760379+0000","last_became_active":"2024-09-13T14:48:02.094029+0000","last_became_peered":"2024-09-13T14:48:02.094029+0000","last_unstale":"2024-09-13T14:57:12.760379+0000","last_undegraded":"2024-09-13T14:57:12.760379+0000","last_fullsized":"2024-09-13T14:57:12.760379+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:44.321099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":143,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.168815+0000","last_change":"2024-09-13T14:50:19.811645+0000","last_active":"2024-09-13T14:57:06.168815+0000","last_peered":"2024-09-13T14:57:06.168815+0000","last_clean":"2024-09-13T14:57:06.168815+0000","last_became_active":"2024-09-13T14:50:19.811312+0000","last_became_peered":"2024-09-13T14:50:19.811312+0000","last_unstale":"2024-09-13T14:57:06.168815+0000","last_undegraded":"2024-09-13T14:57:06.168815+0000","last_fullsized":"2024-09-13T14:57:06.168815+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:19.267639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":397,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.746998+0000","last_change":"2024-09-13T14:48:21.097264+0000","last_active":"2024-09-13T14:57:12.746998+0000","last_peered":"2024-09-13T14:57:12.746998+0000","last_clean":"2024-09-13T14:57:12.746998+0000","last_became_active":"2024-09-13T14:48:21.096834+0000","last_became_peered":"2024-09-13T14:48:21.096834+0000","last_unstale":"2024-09-13T14:57:12.746998+0000","last_undegraded":"2024-09-13T14:57:12.746998+0000","last_fullsized":"2024-09-13T14:57:12.746998+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:39.009121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":452,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.768746+0000","last_change":"2024-09-13T14:50:19.795322+0000","last_active":"2024-09-13T14:57:12.768746+0000","last_peered":"2024-09-13T14:57:12.768746+0000","last_clean":"2024-09-13T14:57:12.768746+0000","last_became_active":"2024-09-13T14:50:19.794957+0000","last_became_peered":"2024-09-13T14:50:19.794957+0000","last_unstale":"2024-09-13T14:57:12.768746+0000","last_undegraded":"2024-09-13T14:57:12.768746+0000","last_fullsized":"2024-09-13T14:57:12.768746+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:36.281568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":448,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.764671+0000","last_change":"2024-09-13T14:50:37.949715+0000","last_active":"2024-09-13T14:57:12.764671+0000","last_peered":"2024-09-13T14:57:12.764671+0000","last_clean":"2024-09-13T14:57:12.764671+0000","last_became_active":"2024-09-13T14:50:37.949415+0000","last_became_peered":"2024-09-13T14:50:37.949415+0000","last_unstale":"2024-09-13T14:57:12.764671+0000","last_undegraded":"2024-09-13T14:57:12.764671+0000","last_fullsized":"2024-09-13T14:57:12.764671+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:50:52.980872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.165554+0000","last_change":"2024-09-13T14:48:21.096785+0000","last_active":"2024-09-13T14:57:06.165554+0000","last_peered":"2024-09-13T14:57:06.165554+0000","last_clean":"2024-09-13T14:57:06.165554+0000","last_became_active":"2024-09-13T14:48:21.096381+0000","last_became_peered":"2024-09-13T14:48:21.096381+0000","last_unstale":"2024-09-13T14:57:06.165554+0000","last_undegraded":"2024-09-13T14:57:06.165554+0000","last_fullsized":"2024-09-13T14:57:06.165554+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:22.906525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":415,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.765096+0000","last_change":"2024-09-13T14:50:19.811954+0000","last_active":"2024-09-13T14:57:12.765096+0000","last_peered":"2024-09-13T14:57:12.765096+0000","last_clean":"2024-09-13T14:57:12.765096+0000","last_became_active":"2024-09-13T14:50:19.811654+0000","last_became_peered":"2024-09-13T14:50:19.811654+0000","last_unstale":"2024-09-13T14:57:12.765096+0000","last_undegraded":"2024-09-13T14:57:12.765096+0000","last_fullsized":"2024-09-13T14:57:12.765096+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.046241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":432,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.768299+0000","last_change":"2024-09-13T14:50:19.801579+0000","last_active":"2024-09-13T14:57:12.768299+0000","last_peered":"2024-09-13T14:57:12.768299+0000","last_clean":"2024-09-13T14:57:12.768299+0000","last_became_active":"2024-09-13T14:50:19.801298+0000","last_became_peered":"2024-09-13T14:50:19.801298+0000","last_unstale":"2024-09-13T14:57:12.768299+0000","last_undegraded":"2024-09-13T14:57:12.768299+0000","last_fullsized":"2024-09-13T14:57:12.768299+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:58.369795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":400,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.768855+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T14:57:12.768855+0000","last_peered":"2024-09-13T14:57:12.768855+0000","last_clean":"2024-09-13T14:57:12.768855+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T14:57:12.768855+0000","last_undegraded":"2024-09-13T14:57:12.768855+0000","last_fullsized":"2024-09-13T14:57:12.768855+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":264,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.760249+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T14:57:12.760249+0000","last_peered":"2024-09-13T14:57:12.760249+0000","last_clean":"2024-09-13T14:57:12.760249+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T14:57:12.760249+0000","last_undegraded":"2024-09-13T14:57:12.760249+0000","last_fullsized":"2024-09-13T14:57:12.760249+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":261,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.147302+0000","last_change":"2024-09-13T14:41:09.648308+0000","last_active":"2024-09-13T14:57:06.147302+0000","last_peered":"2024-09-13T14:57:06.147302+0000","last_clean":"2024-09-13T14:57:06.147302+0000","last_became_active":"2024-09-13T14:41:09.647900+0000","last_became_peered":"2024-09-13T14:41:09.647900+0000","last_unstale":"2024-09-13T14:57:06.147302+0000","last_undegraded":"2024-09-13T14:57:06.147302+0000","last_fullsized":"2024-09-13T14:57:06.147302+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:20.994661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":159,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.777932+0000","last_change":"2024-09-13T14:50:22.039397+0000","last_active":"2024-09-13T14:57:12.777932+0000","last_peered":"2024-09-13T14:57:12.777932+0000","last_clean":"2024-09-13T14:57:12.777932+0000","last_became_active":"2024-09-13T14:50:22.038950+0000","last_became_peered":"2024-09-13T14:50:22.038950+0000","last_unstale":"2024-09-13T14:57:12.777932+0000","last_undegraded":"2024-09-13T14:57:12.777932+0000","last_fullsized":"2024-09-13T14:57:12.777932+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:46.862647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":424,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.767179+0000","last_change":"2024-09-13T14:49:58.511168+0000","last_active":"2024-09-13T14:57:12.767179+0000","last_peered":"2024-09-13T14:57:12.767179+0000","last_clean":"2024-09-13T14:57:12.767179+0000","last_became_active":"2024-09-13T14:49:58.510794+0000","last_became_peered":"2024-09-13T14:49:58.510794+0000","last_unstale":"2024-09-13T14:57:12.767179+0000","last_undegraded":"2024-09-13T14:57:12.767179+0000","last_fullsized":"2024-09-13T14:57:12.767179+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:30.344152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":271,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.147911+0000","last_change":"2024-09-13T14:53:14.538979+0000","last_active":"2024-09-13T14:57:06.147911+0000","last_peered":"2024-09-13T14:57:06.147911+0000","last_clean":"2024-09-13T14:57:06.147911+0000","last_became_active":"2024-09-13T14:53:14.538702+0000","last_became_peered":"2024-09-13T14:53:14.538702+0000","last_unstale":"2024-09-13T14:57:06.147911+0000","last_undegraded":"2024-09-13T14:57:06.147911+0000","last_fullsized":"2024-09-13T14:57:06.147911+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:11.741798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":118,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.168710+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T14:57:06.168710+0000","last_peered":"2024-09-13T14:57:06.168710+0000","last_clean":"2024-09-13T14:57:06.168710+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T14:57:06.168710+0000","last_undegraded":"2024-09-13T14:57:06.168710+0000","last_fullsized":"2024-09-13T14:57:06.168710+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":259,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.147225+0000","last_change":"2024-09-13T14:41:41.490555+0000","last_active":"2024-09-13T14:57:06.147225+0000","last_peered":"2024-09-13T14:57:06.147225+0000","last_clean":"2024-09-13T14:57:06.147225+0000","last_became_active":"2024-09-13T14:41:41.490027+0000","last_became_peered":"2024-09-13T14:41:41.490027+0000","last_unstale":"2024-09-13T14:57:06.147225+0000","last_undegraded":"2024-09-13T14:57:06.147225+0000","last_fullsized":"2024-09-13T14:57:06.147225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.235596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":120,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-13T14:57:06.168833+0000","last_change":"2024-09-13T14:48:21.103577+0000","last_active":"2024-09-13T14:57:06.168833+0000","last_peered":"2024-09-13T14:57:06.168833+0000","last_clean":"2024-09-13T14:57:06.168833+0000","last_became_active":"2024-09-13T14:48:21.094232+0000","last_became_peered":"2024-09-13T14:48:21.094232+0000","last_unstale":"2024-09-13T14:57:06.168833+0000","last_undegraded":"2024-09-13T14:57:06.168833+0000","last_fullsized":"2024-09-13T14:57:06.168833+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:46.985901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":418,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.747273+0000","last_change":"2024-09-13T14:50:19.800136+0000","last_active":"2024-09-13T14:57:12.747273+0000","last_peered":"2024-09-13T14:57:12.747273+0000","last_clean":"2024-09-13T14:57:12.747273+0000","last_became_active":"2024-09-13T14:50:19.799894+0000","last_became_peered":"2024-09-13T14:50:19.799894+0000","last_unstale":"2024-09-13T14:57:12.747273+0000","last_undegraded":"2024-09-13T14:57:12.747273+0000","last_fullsized":"2024-09-13T14:57:12.747273+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:45.904097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":435,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.777369+0000","last_change":"2024-09-13T14:50:37.955764+0000","last_active":"2024-09-13T14:57:12.777369+0000","last_peered":"2024-09-13T14:57:12.777369+0000","last_clean":"2024-09-13T14:57:12.777369+0000","last_became_active":"2024-09-13T14:50:37.953985+0000","last_became_peered":"2024-09-13T14:50:37.953985+0000","last_unstale":"2024-09-13T14:57:12.777369+0000","last_undegraded":"2024-09-13T14:57:12.777369+0000","last_fullsized":"2024-09-13T14:57:12.777369+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:33:42.836715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":290,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.759564+0000","last_change":"2024-09-13T14:50:37.947323+0000","last_active":"2024-09-13T14:57:12.759564+0000","last_peered":"2024-09-13T14:57:12.759564+0000","last_clean":"2024-09-13T14:57:12.759564+0000","last_became_active":"2024-09-13T14:50:37.947026+0000","last_became_peered":"2024-09-13T14:50:37.947026+0000","last_unstale":"2024-09-13T14:57:12.759564+0000","last_undegraded":"2024-09-13T14:57:12.759564+0000","last_fullsized":"2024-09-13T14:57:12.759564+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:23.487658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":488,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-13T14:57:12.768638+0000","last_change":"2024-09-13T14:50:07.812184+0000","last_active":"2024-09-13T14:57:12.768638+0000","last_peered":"2024-09-13T14:57:12.768638+0000","last_clean":"2024-09-13T14:57:12.768638+0000","last_became_active":"2024-09-13T14:50:07.556236+0000","last_became_peered":"2024-09-13T14:50:07.556236+0000","last_unstale":"2024-09-13T14:57:12.768638+0000","last_undegraded":"2024-09-13T14:57:12.768638+0000","last_fullsized":"2024-09-13T14:57:12.768638+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:09.465439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4939055128,"num_objects":75365,"num_object_clones":0,"num_object_copies":226095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145159,"num_write_kb":7056705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14762323968,"data_stored":14762311752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126859,"ondisk_log_size":126859,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":17394,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17951,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234360,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495996,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1930824,"kb_used_data":1512512,"kb_used_omap":1,"kb_used_meta":418302,"kb_avail":91814328,"statfs":{"total":95995035648,"available":94017871872,"internally_reserved":0,"allocated":1548812288,"data_stored":1547337495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428341703},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757631,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1784760,"kb_used_data":1496880,"kb_used_omap":1,"kb_used_meta":287870,"kb_avail":91960392,"statfs":{"total":95995035648,"available":94167441408,"internally_reserved":0,"allocated":1532805120,"data_stored":1531342266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294779335},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019271,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2174276,"kb_used_data":1822396,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91570876,"statfs":{"total":95995035648,"available":93768577024,"internally_reserved":0,"allocated":1866133504,"data_stored":1864662844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468156,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3398136,"kb_used_data":2987440,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":90347016,"statfs":{"total":95995035648,"available":92515344384,"internally_reserved":0,"allocated":3059138560,"data_stored":3057675480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":420474020},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":103,"seq":442381631685,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2610248,"kb_used_data":2141248,"kb_used_omap":8,"kb_used_meta":468983,"kb_avail":91134904,"statfs":{"total":95995035648,"available":93322141696,"internally_reserved":0,"allocated":2192637952,"data_stored":2191163722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8225,"internal_metadata":480239583},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795208,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2495616,"kb_used_data":2094648,"kb_used_omap":7,"kb_used_meta":400952,"kb_avail":91249536,"statfs":{"total":95995035648,"available":93439524864,"internally_reserved":0,"allocated":2144919552,"data_stored":2143448729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":410575474},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":93,"seq":399431958738,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2859140,"kb_used_data":2484540,"kb_used_omap":7,"kb_used_meta":374584,"kb_avail":90886012,"statfs":{"total":95995035648,"available":93067276288,"internally_reserved":0,"allocated":2544168960,"data_stored":2542694694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7705,"internal_metadata":383574503},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6111,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2504,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2514030592,"data_stored":2514026520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2132611072,"data_stored":2132607000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2166292480,"data_stored":2166292480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3041656832,"data_stored":3041656832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1853161472,"data_stored":1853161472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1525088256,"data_stored":1525088256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1529483264,"data_stored":1529479192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T14:57:29.790 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T14:57:29.791 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:57:29.791 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-13T14:57:29.791 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T14:57:29.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2 2024-09-13T14:57:29.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:29.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:29.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:57:29.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:29.965 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:29.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:29.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:30.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:30.159 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:30.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:30.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:29 smithi154 systemd[1]: Stopping Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:57:30.225 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:30.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:30.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:30.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[102314]: pgmap v892: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 723 op/s 2024-09-13T14:57:30.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:30.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:30.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[107038]: pgmap v892: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 723 op/s 2024-09-13T14:57:30.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:30.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:30.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:30.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:30 smithi172 ceph-mon[100696]: pgmap v892: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 723 op/s 2024-09-13T14:57:30.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:57:30.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:57:30.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:30.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.636 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:30.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:30.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:30.759 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:30.857 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:57:30.857 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:30.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:30.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:30.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:30.890 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:57:30.891 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:30.928 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T14:57:30.928 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T14:57:30.966 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:57:30.966 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:30.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:30.997 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:57:30.997 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:31.011 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:31.029 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:57:31.029 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:31.063 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:57:31.064 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:31.101 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:57:31.101 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:31.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:31.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:31.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:31.278 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:31.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:31.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3975371807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:31.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-mon[102314]: osd.2 marked itself down and dead 2024-09-13T14:57:31.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3975371807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:31.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-mon[107038]: osd.2 marked itself down and dead 2024-09-13T14:57:31.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[128111]: 2024-09-13T14:57:31.068+0000 7fd9e1bb7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:57:31.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[128111]: 2024-09-13T14:57:31.068+0000 7fd9e1bb7640 -1 osd.2 280 *** Got signal Terminated *** 2024-09-13T14:57:31.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[128111]: 2024-09-13T14:57:31.068+0000 7fd9e1bb7640 -1 osd.2 280 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:57:31.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:31.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:31.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:31.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:31.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:31 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3975371807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:31 smithi172 ceph-mon[100696]: osd.2 marked itself down and dead 2024-09-13T14:57:31.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:31.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:31.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:31.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:31.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:31.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:32.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:32.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:32.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:32.152 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:57:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:57:31] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:57:32.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:31 smithi154 podman[240851]: 2024-09-13 14:57:31.938712142 +0000 UTC m=+1.041132734 container died d44a72dae8d347fd550049824aefe5882b028cac4d099c64f42a5a09b98e4b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T14:57:32.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:32.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:57:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:32.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:32.352 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:32 smithi172 ceph-mon[100696]: pgmap v893: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-09-13T14:57:32.352 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:32 smithi172 ceph-mon[100696]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:57:32.352 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:32 smithi172 ceph-mon[100696]: osdmap e281: 8 total, 7 up, 7 in 2024-09-13T14:57:32.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:32.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:32.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:32.557 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:32.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:32.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:32.721 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:32 smithi154 podman[240851]: 2024-09-13 14:57:32.400514088 +0000 UTC m=+1.502934683 container cleanup d44a72dae8d347fd550049824aefe5882b028cac4d099c64f42a5a09b98e4b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2) 2024-09-13T14:57:32.721 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:32 smithi154 bash[240851]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2 2024-09-13T14:57:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:32 smithi154 ceph-mon[107038]: pgmap v893: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-09-13T14:57:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:32 smithi154 ceph-mon[107038]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:57:32.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:32 smithi154 ceph-mon[107038]: osdmap e281: 8 total, 7 up, 7 in 2024-09-13T14:57:32.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:32 smithi154 ceph-mon[102314]: pgmap v893: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 727 op/s 2024-09-13T14:57:32.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:32 smithi154 ceph-mon[102314]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-13T14:57:32.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:32 smithi154 ceph-mon[102314]: osdmap e281: 8 total, 7 up, 7 in 2024-09-13T14:57:32.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:32.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:32.872 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:32.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:32.951 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:57:32.951 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:57:32.951 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:57:32.951 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:57:32.951 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:57:32.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:32.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.016 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:32 smithi154 podman[240973]: 2024-09-13 14:57:32.760649937 +0000 UTC m=+0.808671086 container remove d44a72dae8d347fd550049824aefe5882b028cac4d099c64f42a5a09b98e4b78 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-13T14:57:33.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.097 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:33.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:33.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:33.474 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:33.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:33.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:33 smithi172 ceph-mon[100696]: osdmap e282: 8 total, 7 up, 7 in 2024-09-13T14:57:33.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:33.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:57:33.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:33.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:33 smithi154 ceph-mon[102314]: osdmap e282: 8 total, 7 up, 7 in 2024-09-13T14:57:33.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:33 smithi154 ceph-mon[107038]: osdmap e282: 8 total, 7 up, 7 in 2024-09-13T14:57:33.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:33.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:33.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:33.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:33.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:33.918 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:57:33.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:34.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:34.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:34.146 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:34.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:34.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:34.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:34.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:34.525 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:34 smithi154 ceph-mon[102314]: pgmap v896: 89 pgs: 5 peering, 5 stale+active+clean, 79 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 617 op/s 2024-09-13T14:57:34.525 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:34 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:57:34.525 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:34 smithi154 ceph-mon[107038]: pgmap v896: 89 pgs: 5 peering, 5 stale+active+clean, 79 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 617 op/s 2024-09-13T14:57:34.525 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:34 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:57:34.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.696 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:34 smithi172 ceph-mon[100696]: pgmap v896: 89 pgs: 5 peering, 5 stale+active+clean, 79 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 617 op/s 2024-09-13T14:57:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:34 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:57:34.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:34.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:34.930 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:34 smithi154 podman[241298]: 2024-09-13 14:57:34.541292691 +0000 UTC m=+0.142602935 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:57:34.931 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:34 smithi154 podman[241298]: 2024-09-13 14:57:34.768604488 +0000 UTC m=+0.369914720 container create bd08bd37a5a93b69c40f3d7d8bd3599eed0a40d03bd26761c1c7f89b2df62037 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-13T14:57:34.954 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:34.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:35.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:35.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:35.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:35.226 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:35.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:35.352 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:35.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:35.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:35.525 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:35.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:35.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:35.667 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:35 smithi154 podman[241298]: 2024-09-13 14:57:35.302495419 +0000 UTC m=+0.903805663 container init bd08bd37a5a93b69c40f3d7d8bd3599eed0a40d03bd26761c1c7f89b2df62037 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-13T14:57:35.668 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:35 smithi154 podman[241298]: 2024-09-13 14:57:35.313842138 +0000 UTC m=+0.915152369 container start bd08bd37a5a93b69c40f3d7d8bd3599eed0a40d03bd26761c1c7f89b2df62037 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-13T14:57:35.668 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:35 smithi154 podman[241298]: 2024-09-13 14:57:35.466180692 +0000 UTC m=+1.067490925 container attach bd08bd37a5a93b69c40f3d7d8bd3599eed0a40d03bd26761c1c7f89b2df62037 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T14:57:35.700 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:57:35.700 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:57:35.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:57:35.700 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [] 2024-09-13T14:57:35.700 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:57:35.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:35.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:35.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:35.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:35.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:35.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:35.914 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:57:35.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3559412354' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:36.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:36.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:36.178 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:36.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3559412354' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.181 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3559412354' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:57:36.280 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:36.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:36.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:57:36.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:36.467 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:36.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:36.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:36.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:36.742 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:36.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:36.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:36.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:36.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:37.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:37.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:37.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[102314]: pgmap v897: 89 pgs: 16 active+undersized, 5 peering, 11 active+undersized+degraded, 57 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 25489/206079 objects degraded (12.369%) 2024-09-13T14:57:37.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 25489/206079 objects degraded (12.369%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:37.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:37.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3559412354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:37.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[102314]: osdmap e283: 8 total, 7 up, 7 in 2024-09-13T14:57:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[107038]: pgmap v897: 89 pgs: 16 active+undersized, 5 peering, 11 active+undersized+degraded, 57 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 25489/206079 objects degraded (12.369%) 2024-09-13T14:57:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 25489/206079 objects degraded (12.369%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:37.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3559412354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:37.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:36 smithi154 ceph-mon[107038]: osdmap e283: 8 total, 7 up, 7 in 2024-09-13T14:57:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:36 smithi172 ceph-mon[100696]: pgmap v897: 89 pgs: 16 active+undersized, 5 peering, 11 active+undersized+degraded, 57 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 714 op/s; 25489/206079 objects degraded (12.369%) 2024-09-13T14:57:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:36 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 25489/206079 objects degraded (12.369%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:36 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3559412354' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:57:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:36 smithi172 ceph-mon[100696]: osdmap e283: 8 total, 7 up, 7 in 2024-09-13T14:57:37.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:37.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:37.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:37.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:37.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:37.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:37.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:37.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:37.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:37.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:37.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:37.955 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:37.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:38.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:38.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:38.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:37 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:57:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:37 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:57:38.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:57:37 smithi154 podman[241298]: 2024-09-13 14:57:37.916217955 +0000 UTC m=+3.517528192 container died bd08bd37a5a93b69c40f3d7d8bd3599eed0a40d03bd26761c1c7f89b2df62037 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-13T14:57:38.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:37 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:57:38.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:38.352 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:38.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:38.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:38.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:38.608 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:38.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T14:57:38.666 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:38.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:38.733 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:38.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T14:57:38.758 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T14:57:38.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:38.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:38.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:38.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:39.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:39.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:39.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:38 smithi172 ceph-mon[100696]: pgmap v899: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 29530/202080 objects degraded (14.613%) 2024-09-13T14:57:39.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:39.246 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:38 smithi154 ceph-mon[102314]: pgmap v899: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 29530/202080 objects degraded (14.613%) 2024-09-13T14:57:39.247 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:38 smithi154 ceph-mon[107038]: pgmap v899: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 29530/202080 objects degraded (14.613%) 2024-09-13T14:57:39.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:39.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:57:39.341 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:57:39.341 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:39.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:39.379 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:57:39.379 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:39.383 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:57:39.420 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:57:39.421 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:39.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:39.452 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:57:39.452 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:39.487 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:57:39.487 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:39.518 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:57:39.518 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:39.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:39.549 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:57:39.549 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.550 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:39.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:39.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:39.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:39.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:39.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:39.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:39.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:39.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:40.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:40.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:40.197 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:39 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3385409512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:40.198 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:39 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3385409512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:40.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:40.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:57:40.239 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:57:40.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:40.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:40.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:39 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3385409512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:40.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:40.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:40.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:40.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:40.763 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:40.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:40.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:40.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:41.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:41.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:41.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:41.171 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:40 smithi154 ceph-mon[102314]: pgmap v900: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 683 op/s; 29231/199254 objects degraded (14.670%) 2024-09-13T14:57:41.172 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:40 smithi154 ceph-mon[107038]: pgmap v900: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 683 op/s; 29231/199254 objects degraded (14.670%) 2024-09-13T14:57:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.177 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:41.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:41.184 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:40 smithi172 ceph-mon[100696]: pgmap v900: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 683 op/s; 29231/199254 objects degraded (14.670%) 2024-09-13T14:57:41.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:41.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:41.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:41.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:41.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:41.565 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:41.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:41.693 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:41.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:41.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:41.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:41.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:41.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:42.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:42.074 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:57:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:57:41] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-09-13T14:57:42.078 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:42.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:42.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:42.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:57:42.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.339 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:42.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:42.409 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:42.410 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:42.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:42.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:42.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:42.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:42.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:42.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:42.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:42.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:42.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:43.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:43.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:43.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:43.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:43.201 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[102314]: pgmap v901: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 27763/192213 objects degraded (14.444%); 0 B/s, 0 objects/s recovering 2024-09-13T14:57:43.201 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 27763/192213 objects degraded (14.444%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:43.201 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:43.201 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:43.201 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[107038]: pgmap v901: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 27763/192213 objects degraded (14.444%); 0 B/s, 0 objects/s recovering 2024-09-13T14:57:43.201 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 27763/192213 objects degraded (14.444%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:43.201 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:43.202 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:43.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T14:57:43.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:43.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:43.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:43.461 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:43 smithi172 ceph-mon[100696]: pgmap v901: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 27763/192213 objects degraded (14.444%); 0 B/s, 0 objects/s recovering 2024-09-13T14:57:43.461 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:43 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 27763/192213 objects degraded (14.444%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:43.461 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:57:43.461 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:43.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:43.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:43.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:43.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:43.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:43.758 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5] out_osds: [7] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-13T14:57:43.758 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:57:43.758 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-13T14:57:43.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-13T14:57:43.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:43.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:43.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:43.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:44.020 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:44.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:44.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:44.146 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:44.149 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:44.154 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:44 smithi154 ceph-mon[102314]: pgmap v902: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 27478/187797 objects degraded (14.632%); 0 B/s, 0 objects/s recovering 2024-09-13T14:57:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:44 smithi154 ceph-mon[107038]: pgmap v902: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 27478/187797 objects degraded (14.632%); 0 B/s, 0 objects/s recovering 2024-09-13T14:57:44.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:44.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:44.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:44.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:44 smithi172 ceph-mon[100696]: pgmap v902: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 27478/187797 objects degraded (14.632%); 0 B/s, 0 objects/s recovering 2024-09-13T14:57:44.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:44.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:44.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:44.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:44.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:44.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:44.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:44.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:44.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:45.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:45.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:45.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.221 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:45.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:45.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:45.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:45.350 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:45.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:45.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:45.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:45.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:45.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:45.746 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:45.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:45.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:45.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:45.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:46.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:46.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:46.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:46.165 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:57:46.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.280 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:46.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:46.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:46.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:46 smithi154 ceph-mon[102314]: pgmap v903: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 25693/178080 objects degraded (14.428%) 2024-09-13T14:57:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:46 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1324314401' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:46 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:46 smithi154 ceph-mon[107038]: pgmap v903: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 25693/178080 objects degraded (14.428%) 2024-09-13T14:57:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:46 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1324314401' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:46 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:46 smithi172 ceph-mon[100696]: pgmap v903: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 25693/178080 objects degraded (14.428%) 2024-09-13T14:57:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:46 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1324314401' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:46 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:57:46.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:46.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:46.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:46.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:46.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:46.791 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:46.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:46.916 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:46.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:47.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:47.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:47.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:47.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:47.270 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:47 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:47.270 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:47 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:47.270 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:47 smithi154 ceph-mon[102314]: osdmap e284: 8 total, 7 up, 7 in 2024-09-13T14:57:47.271 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:47 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:47.271 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:47 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:47.271 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:47 smithi154 ceph-mon[107038]: osdmap e284: 8 total, 7 up, 7 in 2024-09-13T14:57:47.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:47.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:47.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:47.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:47 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:57:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:47 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:57:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:47 smithi172 ceph-mon[100696]: osdmap e284: 8 total, 7 up, 7 in 2024-09-13T14:57:47.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:47.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:47.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:47.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:47.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:47.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:47.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:47.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:47.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:48.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.120 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:48.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:48.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:48.244 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:48.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:48.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:48 smithi154 ceph-mon[102314]: pgmap v905: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 25693/178080 objects degraded (14.428%) 2024-09-13T14:57:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:48 smithi154 ceph-mon[107038]: pgmap v905: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 25693/178080 objects degraded (14.428%) 2024-09-13T14:57:48.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:48.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:48 smithi172 ceph-mon[100696]: pgmap v905: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 25693/178080 objects degraded (14.428%) 2024-09-13T14:57:48.515 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:48.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:48.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:48.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:48.777 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:48.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:48.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:48.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:49.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:49.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:49.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:57:49.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:49.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:49.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:49.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:49.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:49.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:49.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:49 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 25154/171300 objects degraded (14.684%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:49.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:49 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 25154/171300 objects degraded (14.684%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:49 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 25154/171300 objects degraded (14.684%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:49.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:49.592 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:49.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:49.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:49.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:49.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:49.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:49.975 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:49.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:50.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:50.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:50.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:50.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:50.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:50.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:50.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:50 smithi172 ceph-mon[100696]: pgmap v906: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 931 op/s; 25154/171300 objects degraded (14.684%) 2024-09-13T14:57:50.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:50.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:50.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:50.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:50.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:50 smithi154 ceph-mon[102314]: pgmap v906: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 931 op/s; 25154/171300 objects degraded (14.684%) 2024-09-13T14:57:50.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:50 smithi154 ceph-mon[107038]: pgmap v906: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 931 op/s; 25154/171300 objects degraded (14.684%) 2024-09-13T14:57:50.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:50.800 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:50.902 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:50.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:50.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:51.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:51.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:51.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:51.216 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:51.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:51.596 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:51.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:51.725 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:51.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:51.856 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:51.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:51.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:51.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:52.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:52.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:57:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:57:51] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-09-13T14:57:52.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:52.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:52.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:52.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:52 smithi172 ceph-mon[100696]: pgmap v907: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 23589/163911 objects degraded (14.391%) 2024-09-13T14:57:52.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:52.550 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:52.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:52 smithi154 ceph-mon[102314]: pgmap v907: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 23589/163911 objects degraded (14.391%) 2024-09-13T14:57:52.620 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:52 smithi154 ceph-mon[107038]: pgmap v907: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 23589/163911 objects degraded (14.391%) 2024-09-13T14:57:52.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:52.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:52.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:52.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:52.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:52.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:52.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:53.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:53.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:53.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:53.220 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:53.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:53.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:53.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:53.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:53.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:53.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:53.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:53.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:53.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:53.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:53.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:54.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:54.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:54.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:54.179 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:54.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:54.304 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:54.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:54.430 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:54 smithi172 ceph-mon[100696]: pgmap v908: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 23307/159519 objects degraded (14.611%) 2024-09-13T14:57:54.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:54.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:54.581 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:54 smithi154 ceph-mon[102314]: pgmap v908: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 23307/159519 objects degraded (14.611%) 2024-09-13T14:57:54.581 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:54 smithi154 ceph-mon[107038]: pgmap v908: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 23307/159519 objects degraded (14.611%) 2024-09-13T14:57:54.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:54.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:54.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.828 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:54.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:54.864 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:57:54.864 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:57:54.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:54.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:54.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:55.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:55.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:55.113 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:57:55.113 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:57:55.148 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:57:55.149 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:57:55.149 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:54 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T14:57:54.986+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e284 definitely_dead 0 2024-09-13T14:57:55.152 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:57:55.186 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:57:55.187 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:57:55.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:55.216 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:57:55.216 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:57:55.231 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:55.251 INFO:teuthology.orchestra.run.smithi154.stderr:marked in osd.7. 2024-09-13T14:57:55.257 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:57:55.257 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:57:55.294 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:57:55.294 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:57:55.312 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:57:55.312 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:57:55.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:55.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:55.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:55.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 23307/159519 objects degraded (14.611%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:55.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2052485985' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T14:57:55.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T14:57:55.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2272929594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:55.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 23307/159519 objects degraded (14.611%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:55.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2052485985' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T14:57:55.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T14:57:55.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2272929594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:55.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:55.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 23307/159519 objects degraded (14.611%), 14 pgs degraded (PG_DEGRADED) 2024-09-13T14:57:55.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2052485985' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T14:57:55.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:55 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T14:57:55.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2272929594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:57:55.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:55.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:55.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:55.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:55.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.846 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:55.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:55.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:55.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:55.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:56.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:56.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:56.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T14:57:56.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:56.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:56.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:56.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:56.437 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:56 smithi154 ceph-mon[102314]: pgmap v909: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 949 op/s; 21527/149592 objects degraded (14.390%) 2024-09-13T14:57:56.437 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:56 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T14:57:56.438 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:56 smithi154 ceph-mon[102314]: osdmap e285: 8 total, 7 up, 8 in 2024-09-13T14:57:56.438 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:56 smithi154 ceph-mon[107038]: pgmap v909: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 949 op/s; 21527/149592 objects degraded (14.390%) 2024-09-13T14:57:56.438 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:56 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T14:57:56.438 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:56 smithi154 ceph-mon[107038]: osdmap e285: 8 total, 7 up, 8 in 2024-09-13T14:57:56.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:56.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:56.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:56.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:56.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:56.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:56 smithi172 ceph-mon[100696]: pgmap v909: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 949 op/s; 21527/149592 objects degraded (14.390%) 2024-09-13T14:57:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:56 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T14:57:56.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:56 smithi172 ceph-mon[100696]: osdmap e285: 8 total, 7 up, 8 in 2024-09-13T14:57:56.771 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-13T14:57:56.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:56.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:56.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:56.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:56.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.064 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:57.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:57.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:57.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:57.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:57:57.222 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:57:57.293 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:57.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:57.321 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:57:57.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:57.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:57.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:57 smithi154 ceph-mon[102314]: osdmap e286: 8 total, 7 up, 8 in 2024-09-13T14:57:57.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:57 smithi154 ceph-mon[102314]: osdmap e287: 8 total, 7 up, 8 in 2024-09-13T14:57:57.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:57 smithi154 ceph-mon[107038]: osdmap e286: 8 total, 7 up, 8 in 2024-09-13T14:57:57.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:57 smithi154 ceph-mon[107038]: osdmap e287: 8 total, 7 up, 8 in 2024-09-13T14:57:57.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:57.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:57.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:57:57.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:57:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:57 smithi172 ceph-mon[100696]: osdmap e286: 8 total, 7 up, 8 in 2024-09-13T14:57:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:57 smithi172 ceph-mon[100696]: osdmap e287: 8 total, 7 up, 8 in 2024-09-13T14:57:57.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:57.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:57.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:57.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:57:58.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:57:58.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:58.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:58.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:58.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:58.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:58.423 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[102314]: pgmap v912: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 21527/149592 objects degraded (14.390%) 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[102314]: 7.8 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10793 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[102314]: 7.c starting backfill to osd.7 from (0'0,0'0] MAX to 284'10583 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[102314]: 7.5 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10539 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[102314]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10524 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[102314]: 7.f starting backfill to osd.7 from (0'0,0'0] MAX to 284'10601 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[107038]: pgmap v912: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 21527/149592 objects degraded (14.390%) 2024-09-13T14:57:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[107038]: 7.8 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10793 2024-09-13T14:57:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[107038]: 7.c starting backfill to osd.7 from (0'0,0'0] MAX to 284'10583 2024-09-13T14:57:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[107038]: 7.5 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10539 2024-09-13T14:57:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[107038]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10524 2024-09-13T14:57:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[107038]: 7.f starting backfill to osd.7 from (0'0,0'0] MAX to 284'10601 2024-09-13T14:57:58.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:58.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.557 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:58.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:58.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:58.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:58 smithi172 ceph-mon[100696]: pgmap v912: 89 pgs: 18 active+undersized, 14 active+undersized+degraded, 57 active+clean; 3.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 21527/149592 objects degraded (14.390%) 2024-09-13T14:57:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:58 smithi172 ceph-mon[100696]: 7.8 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10793 2024-09-13T14:57:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:58 smithi172 ceph-mon[100696]: 7.c starting backfill to osd.7 from (0'0,0'0] MAX to 284'10583 2024-09-13T14:57:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:58 smithi172 ceph-mon[100696]: 7.5 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10539 2024-09-13T14:57:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:58 smithi172 ceph-mon[100696]: 7.6 starting backfill to osd.7 from (0'0,0'0] MAX to 284'10524 2024-09-13T14:57:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:58 smithi172 ceph-mon[100696]: 7.f starting backfill to osd.7 from (0'0,0'0] MAX to 284'10601 2024-09-13T14:57:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:57:58.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:58.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:57:58.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:58.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:58.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:59.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:57:59.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:57:59.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:59.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:57:59.231 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:59.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:59.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:59.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:57:59.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:57:59 smithi154 ceph-mon[102314]: osdmap e288: 8 total, 7 up, 8 in 2024-09-13T14:57:59.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:57:59 smithi154 ceph-mon[107038]: osdmap e288: 8 total, 7 up, 8 in 2024-09-13T14:57:59.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:57:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:57:59.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:57:59.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:57:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:57:59 smithi172 ceph-mon[100696]: osdmap e288: 8 total, 7 up, 8 in 2024-09-13T14:57:59.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:57:59.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:57:59.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:57:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:57:59.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:57:59.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:00.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:00.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:00.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:00.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:00.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.338 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:00.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:00.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:00.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:00.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:00.580 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:00 smithi154 ceph-mon[102314]: pgmap v915: 89 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 remapped+peering, 16 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 354 op/s; 17724/145332 objects degraded (12.196%); 3/145332 objects misplaced (0.002%); 7.1 MiB/s, 2 keys/s, 118 objects/s recovering 2024-09-13T14:58:00.580 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:00 smithi154 ceph-mon[102314]: osdmap e289: 8 total, 7 up, 8 in 2024-09-13T14:58:00.580 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 17724/145332 objects degraded (12.196%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:00.581 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:00 smithi154 ceph-mon[107038]: pgmap v915: 89 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 remapped+peering, 16 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 354 op/s; 17724/145332 objects degraded (12.196%); 3/145332 objects misplaced (0.002%); 7.1 MiB/s, 2 keys/s, 118 objects/s recovering 2024-09-13T14:58:00.581 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:00 smithi154 ceph-mon[107038]: osdmap e289: 8 total, 7 up, 8 in 2024-09-13T14:58:00.581 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 17724/145332 objects degraded (12.196%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:00.597 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:00.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:00.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:00 smithi172 ceph-mon[100696]: pgmap v915: 89 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 remapped+peering, 16 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 354 op/s; 17724/145332 objects degraded (12.196%); 3/145332 objects misplaced (0.002%); 7.1 MiB/s, 2 keys/s, 118 objects/s recovering 2024-09-13T14:58:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:00 smithi172 ceph-mon[100696]: osdmap e289: 8 total, 7 up, 8 in 2024-09-13T14:58:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 17724/145332 objects degraded (12.196%), 11 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:00.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:00.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:00.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:00.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:01.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:01.078 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:58:01.078 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:99412 6.0 6:0e5ca8c0:::benchmark_data_smithi154_175640_object13636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:27.920512+0000", 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 392.936137514, 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89906116800000002, 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.079 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 99412 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920512+0000", 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920512+0000", 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920514+0000", 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.542e-06 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920521+0000", 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4250000000000001e-06 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920538+0000", 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6626999999999999e-05 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920545+0000", 2024-09-13T14:58:01.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0849999999999999e-06 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920601+0000", 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5792999999999999e-05 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.920608+0000", 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7130000000000005e-06 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.811980+0000", 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89137181600000004 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.812104+0000", 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000123647 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.812208+0000", 2024-09-13T14:58:01.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010425 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.815905+0000", 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036968489999999999 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.815918+0000", 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2677000000000001e-05 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819459+0000", 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035411840000000002 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819524+0000", 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5659000000000003e-05 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819573+0000", 2024-09-13T14:58:01.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8900999999999998e-05 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:99418 6.0 6:0fb8d46d:::benchmark_data_smithi154_175640_object13642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:27.948072+0000", 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 392.908577562, 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.88284572800000005, 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.088 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 99418 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948072+0000", 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948072+0000", 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948075+0000", 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6749999999999998e-06 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948085+0000", 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0324e-05 2024-09-13T14:58:01.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948108+0000", 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3227e-05 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948119+0000", 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1058e-05 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948206+0000", 2024-09-13T14:58:01.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6368999999999999e-05 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.948220+0000", 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3973e-05 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.815945+0000", 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86772523000000001 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.816019+0000", 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4066000000000005e-05 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.816098+0000", 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9357999999999995e-05 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819620+0000", 2024-09-13T14:58:01.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035217120000000002 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819657+0000", 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6918999999999997e-05 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830811+0000", 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011153573999999999 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830870+0000", 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9154999999999999e-05 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830918+0000", 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8087999999999998e-05 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:99429 6.0 6:046c2ea6:::benchmark_data_smithi154_175640_object13653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:51:27.979663+0000", 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 392.87698621099997, 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.85791414200000005, 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 99429 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.096 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979663+0000", 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979663+0000", 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979665+0000", 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6300000000000001e-06 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979672+0000", 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.029e-06 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979685+0000", 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2554e-05 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979692+0000", 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.34e-06 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979752+0000", 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9768000000000001e-05 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:27.979760+0000", 2024-09-13T14:58:01.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5150000000000006e-06 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819683+0000", 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83992291600000002 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819812+0000", 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012889100000000001 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.819945+0000", 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013302400000000001 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.830928+0000", 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0109833 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837394+0000", 2024-09-13T14:58:01.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064660960000000002 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837453+0000", 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8161000000000001e-05 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837528+0000", 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5260999999999997e-05 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:51:28.837578+0000", 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9657000000000002e-05 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:119325 6.9 6:90d7e752:::benchmark_data_smithi154_175640_object33549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:52:49.420653+0000", 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 311.43599667500001, 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0220450670000001, 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.103 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 119325 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420653+0000", 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420653+0000", 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420655+0000", 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2369999999999999e-06 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420664+0000", 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5890000000000003e-06 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420689+0000", 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4791e-05 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420700+0000", 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1528e-05 2024-09-13T14:58:01.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420778+0000", 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7929999999999994e-05 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:49.420789+0000", 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1442e-05 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.430918+0000", 2024-09-13T14:58:01.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.01012883 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.431019+0000", 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101117 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.431099+0000", 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9592000000000006e-05 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.434734+0000", 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036347409999999999 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.434830+0000", 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5964000000000005e-05 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.442599+0000", 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077691039999999998 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.442652+0000", 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3643000000000001e-05 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:52:50.442698+0000", 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5559000000000002e-05 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45383.0:41022 7.7 e265/260 7:e3e17f15:::benchmark_data_smithi154_225924_object41021:head v 265'2634, mlcod=265'2631)", 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:55:27.347045+0000", 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 153.509604174, 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31500234199999999, 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347045+0000", 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347045+0000", 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347046+0000", 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0999999999999997e-07 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347080+0000", 2024-09-13T14:58:01.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3317e-05 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347091+0000", 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1097999999999999e-05 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347098+0000", 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9040000000000003e-06 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347171+0000", 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2949999999999998e-05 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.347189+0000", 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8142999999999999e-05 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.661938+0000", 2024-09-13T14:58:01.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31474970899999999 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.662048+0000", 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109311 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:41027 7.9 7:9fec0f17:::benchmark_data_smithi154_225924_object41026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:55:27.350399+0000", 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 153.506250706, 2024-09-13T14:58:01.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31545366200000002, 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 41027 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.350399+0000", 2024-09-13T14:58:01.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.350399+0000", 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.350401+0000", 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.902e-06 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.350427+0000", 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6173000000000002e-05 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.350440+0000", 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3113e-05 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.350448+0000", 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7819999999999994e-06 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.350736+0000", 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028859099999999999 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351579+0000", 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00084207599999999998 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351716+0000", 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137589 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.664596+0000", 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31287973499999999 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.665591+0000", 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000994965 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.665605+0000", 2024-09-13T14:58:01.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3903999999999999e-05 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.665799+0000", 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019427499999999999 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.665853+0000", 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3557000000000002e-05 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:41032 7.9 7:9d56d548:::benchmark_data_smithi154_225924_object41031:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:55:27.351321+0000", 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 153.505328711, 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31569127499999999, 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 41032 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.122 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351321+0000", 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351321+0000", 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351321+0000", 2024-09-13T14:58:01.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2799999999999996e-07 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351341+0000", 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9360000000000001e-05 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351348+0000", 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9330000000000004e-06 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.351353+0000", 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3730000000000001e-06 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.354930+0000", 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035774100000000001 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.661940+0000", 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30700939799999999 2024-09-13T14:58:01.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.662106+0000", 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016578799999999999 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.665865+0000", 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037596249999999999 2024-09-13T14:58:01.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.666878+0000", 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001012813 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.666921+0000", 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3129e-05 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.666974+0000", 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3139000000000001e-05 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:55:27.667012+0000", 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7778999999999999e-05 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:150537 7.5 7:a670c78a:::benchmark_data_smithi154_225924_object40273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:31.310476+0000", 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.546173565, 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.026179865, 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 150537 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.129 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310476+0000", 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310476+0000", 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310477+0000", 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2219999999999999e-06 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310483+0000", 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3410000000000002e-06 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310495+0000", 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2473999999999999e-05 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310502+0000", 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1760000000000004e-06 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310572+0000", 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9937999999999993e-05 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:31.310584+0000", 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1426999999999999e-05 2024-09-13T14:58:01.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:32.302990+0000", 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99240618300000005 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:32.306312+0000", 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033218879999999998 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1", 2024-09-13T14:58:01.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:32.306589+0000", 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027776299999999999 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:32.321782+0000", 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015192054 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:32.336507+0000", 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014725771 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:32.336558+0000", 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0915999999999998e-05 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:32.336656+0000", 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7712000000000002e-05 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.135 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172067 7.f 7:f8fc3a7a:::benchmark_data_smithi154_225924_object61803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:55.250041+0000", 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.6066081580000002, 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2235757180000002, 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172067 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250041+0000", 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250041+0000", 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250042+0000", 2024-09-13T14:58:01.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4799999999999999e-07 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250045+0000", 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6359999999999998e-06 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250051+0000", 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1670000000000004e-06 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250055+0000", 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7770000000000002e-06 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250107+0000", 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.181e-05 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250214+0000", 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010710200000000001 2024-09-13T14:58:01.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4", 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.250799+0000", 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058477699999999997 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.266389+0000", 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015589878999999999 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.266400+0000", 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1119e-05 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.266789+0000", 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.000388652 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.266842+0000", 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3690000000000003e-05 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.464064+0000", 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1972213250000001 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.464176+0000", 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112089 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473537+0000", 2024-09-13T14:58:01.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093610840000000004 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473617+0000", 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0162999999999998e-05 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172071 7.6 7:6c0d6374:::benchmark_data_smithi154_225924_object61807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:55.258208+0000", 2024-09-13T14:58:01.143 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5984413379999998, 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2227888070000001, 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172071 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.258208+0000", 2024-09-13T14:58:01.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.258208+0000", 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.258209+0000", 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9699999999999994e-07 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.258214+0000", 2024-09-13T14:58:01.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5290000000000002e-06 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.258223+0000", 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3570000000000004e-06 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.258230+0000", 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5529999999999998e-06 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.272734+0000", 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014504222000000001 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.272744+0000", 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7289999999999995e-06 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269701+0000", 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99695777600000002 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269711+0000", 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5030000000000003e-06 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.454994+0000", 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1852827050000001 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.455102+0000", 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108317 2024-09-13T14:58:01.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5,7", 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.455213+0000", 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111241 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460854+0000", 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056407489999999996 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460916+0000", 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2000000000000003e-05 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.471485+0000", 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010568706000000001 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.480843+0000", 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093584870000000004 2024-09-13T14:58:01.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.480893+0000", 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9756999999999998e-05 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.480997+0000", 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010417900000000001 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.152 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172072 7.6 7:6619a025:::benchmark_data_smithi154_225924_object61808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:55.263918+0000", 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5927314900000002, 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2172222440000002, 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172072 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.263918+0000", 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.263918+0000", 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.263920+0000", 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0470000000000001e-06 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.263924+0000", 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2590000000000002e-06 2024-09-13T14:58:01.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.263935+0000", 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0282000000000001e-05 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.263944+0000", 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4469999999999995e-06 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.272768+0000", 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088240750000000007 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.272775+0000", 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0940000000000002e-06 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269735+0000", 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.996959822 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269742+0000", 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3170000000000004e-06 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460952+0000", 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.191209696 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.461069+0000", 2024-09-13T14:58:01.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011675699999999999 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5,7", 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.461193+0000", 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012375400000000001 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.471535+0000", 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010342555999999999 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.471590+0000", 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4888e-05 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.471635+0000", 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5272999999999998e-05 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481004+0000", 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093687809999999996 2024-09-13T14:58:01.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481045+0000", 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0889000000000001e-05 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481140+0000", 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5307000000000001e-05 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172074 7.f 7:f82d7e5b:::benchmark_data_smithi154_225924_object61810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:55.266994+0000", 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5896559139999997, 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2124111759999998, 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172074 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.162 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.266994+0000", 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.266994+0000", 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.266996+0000", 2024-09-13T14:58:01.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9360000000000002e-06 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.267000+0000", 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9009999999999996e-06 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.267009+0000", 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9520000000000006e-06 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.267022+0000", 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2728000000000001e-05 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.273302+0000", 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062796830000000003 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.273328+0000", 2024-09-13T14:58:01.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6407999999999999e-05 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.266891+0000", 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99356282299999998 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.266920+0000", 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8736000000000001e-05 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.464415+0000", 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.197494719 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.464516+0000", 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101888 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4,7", 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.464642+0000", 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012560100000000001 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473217+0000", 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085745270000000002 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473652+0000", 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043576999999999998 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473772+0000", 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120066 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.479222+0000", 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054493320000000003 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.479284+0000", 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1991999999999999e-05 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.479405+0000", 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012111400000000001 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172078 7.6 7:6cd1f181:::benchmark_data_smithi154_225924_object61814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:55.275055+0000", 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5815942920000001, 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.206358056, 2024-09-13T14:58:01.170 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172078 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.275055+0000", 2024-09-13T14:58:01.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.275055+0000", 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.275057+0000", 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3459999999999999e-06 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.275061+0000", 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6229999999999996e-06 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.275072+0000", 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0327e-05 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.275080+0000", 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0499999999999992e-06 2024-09-13T14:58:01.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.276395+0000", 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013150150000000001 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.276405+0000", 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0345e-05 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269766+0000", 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99336114200000003 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269774+0000", 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8089999999999993e-06 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.465903+0000", 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.196128879 2024-09-13T14:58:01.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.466015+0000", 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111748 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5,7", 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.466751+0000", 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073684200000000003 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.480792+0000", 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014040258999999999 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.480831+0000", 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9555999999999999e-05 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481147+0000", 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031616599999999999 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481263+0000", 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011516299999999999 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481318+0000", 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5909000000000003e-05 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481413+0000", 2024-09-13T14:58:01.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4877000000000007e-05 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172091 7.6 7:6fe8a60d:::benchmark_data_smithi154_225924_object61827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:55.291158+0000", 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5654913950000005, 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1904866439999999, 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172091 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291158+0000", 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291158+0000", 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291160+0000", 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5710000000000001e-06 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291168+0000", 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9810000000000003e-06 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291186+0000", 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8037e-05 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291193+0000", 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0060000000000003e-06 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291267+0000", 2024-09-13T14:58:01.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4363999999999995e-05 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.291279+0000", 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2183e-05 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269798+0000", 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97851850200000001 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.269806+0000", 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0169999999999996e-06 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.471674+0000", 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2018684509999999 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.471811+0000", 2024-09-13T14:58:01.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013625099999999999 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5,7", 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.471951+0000", 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014034200000000001 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481153+0000", 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092016210000000001 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481187+0000", 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4847e-05 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481229+0000", 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1498999999999997e-05 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481452+0000", 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022324000000000001 2024-09-13T14:58:01.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481536+0000", 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4023000000000005e-05 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.481645+0000", 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010870900000000001 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172099 7.f 7:f08b91d3:::benchmark_data_smithi154_225924_object61835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:55.316436+0000", 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5402139100000003, 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1631539580000001, 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172099 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.188 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316436+0000", 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316436+0000", 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316437+0000", 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.579e-06 2024-09-13T14:58:01.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316444+0000", 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7660000000000001e-06 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316460+0000", 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6120000000000002e-05 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316469+0000", 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8999999999999995e-06 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316541+0000", 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2311999999999995e-05 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:55.316553+0000", 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1956e-05 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.267103+0000", 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.95055009800000001 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.267115+0000", 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1583000000000001e-05 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.468570+0000", 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.201454958 2024-09-13T14:58:01.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.468697+0000", 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012671899999999999 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,4,7", 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.468823+0000", 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000126456 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473690+0000", 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048666220000000001 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473728+0000", 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7805000000000001e-05 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473778+0000", 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0238999999999998e-05 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.479443+0000", 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056650620000000002 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.479493+0000", 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9651000000000003e-05 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.479590+0000", 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7132000000000004e-05 2024-09-13T14:58:01.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172062 7.5 7:af9f25b6:::benchmark_data_smithi154_225924_object61798:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.252173+0000", 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.6044764440000003, 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.193477138, 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172062 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252173+0000", 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252173+0000", 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252175+0000", 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4050000000000001e-06 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252180+0000", 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5960000000000003e-06 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252196+0000", 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5891000000000001e-05 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252207+0000", 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1037e-05 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252363+0000", 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015554699999999999 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252370+0000", 2024-09-13T14:58:01.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3810000000000001e-06 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.266342+0000", 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013972076 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.266352+0000", 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5279999999999992e-06 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.440459+0000", 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1741073339999999 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.440610+0000", 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015117400000000001 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.440746+0000", 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013563 2024-09-13T14:58:01.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.445478+0000", 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047320130000000002 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.445487+0000", 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9129999999999992e-06 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.445513+0000", 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6129999999999999e-05 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.445558+0000", 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4852999999999999e-05 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.445650+0000", 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2629999999999999e-05 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172079 7.5 7:aea7042d:::benchmark_data_smithi154_225924_object61815:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.252231+0000", 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.604419043, 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2078454999999999, 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.204 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172079 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252231+0000", 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252231+0000", 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252231+0000", 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2200000000000004e-07 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252234+0000", 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.745e-06 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252241+0000", 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3329999999999999e-06 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252247+0000", 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9610000000000002e-06 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252470+0000", 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000222981 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252475+0000", 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2800000000000003e-06 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268323+0000", 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015847211 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268332+0000", 2024-09-13T14:58:01.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2620000000000004e-06 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.445796+0000", 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1774641260000001 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.445933+0000", 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013733800000000001 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:01.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.446008+0000", 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4598999999999995e-05 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.451487+0000", 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054792199999999999 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.451552+0000", 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5026999999999999e-05 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.459919+0000", 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083668219999999995 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.459972+0000", 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2979999999999998e-05 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460076+0000", 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010409299999999999 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.211 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172089 7.5 7:a875a346:::benchmark_data_smithi154_225924_object61825:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.252256+0000", 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.6043940210000001, 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2080063649999999, 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.212 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172089 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252256+0000", 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252256+0000", 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252256+0000", 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4900000000000001e-07 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252259+0000", 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7089999999999998e-06 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252265+0000", 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4169999999999997e-06 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252270+0000", 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0690000000000001e-06 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252498+0000", 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000228199 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252503+0000", 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6619999999999996e-06 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268355+0000", 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015852384000000001 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268362+0000", 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8730000000000001e-06 2024-09-13T14:58:01.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.451585+0000", 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1832230850000001 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.451678+0000", 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2981999999999998e-05 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:01.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.451786+0000", 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107354 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.459903+0000", 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081167400000000008 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460108+0000", 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020514700000000001 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460119+0000", 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1017e-05 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460166+0000", 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7441999999999997e-05 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460262+0000", 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5835999999999995e-05 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.219 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172090 7.5 7:aaa59eee:::benchmark_data_smithi154_225924_object61826:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.252277+0000", 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.6043721370000004, 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2217012890000001, 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.220 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172090 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252277+0000", 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252277+0000", 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252278+0000", 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4400000000000001e-07 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252280+0000", 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1880000000000001e-06 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252286+0000", 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0209999999999996e-06 2024-09-13T14:58:01.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252292+0000", 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.48e-06 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252525+0000", 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00023309700000000001 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252530+0000", 2024-09-13T14:58:01.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8879999999999996e-06 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268385+0000", 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015855391999999999 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268393+0000", 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9950000000000005e-06 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.455956+0000", 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1875631200000001 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.456088+0000", 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013202799999999999 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.456207+0000", 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118767 2024-09-13T14:58:01.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460309+0000", 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041019910000000001 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.468641+0000", 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083323149999999999 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473845+0000", 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052034730000000001 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473911+0000", 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5903999999999998e-05 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.473979+0000", 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8186000000000001e-05 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.227 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172095 7.5 7:a48429d9:::benchmark_data_smithi154_225924_object61831:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.252298+0000", 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.6043513550000004, 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2218510659999999, 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:58:01.228 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172095 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252298+0000", 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252298+0000", 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252299+0000", 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8799999999999998e-07 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252301+0000", 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0619999999999999e-06 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252307+0000", 2024-09-13T14:58:01.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.263e-06 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252312+0000", 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9139999999999999e-06 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252552+0000", 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000239979 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T14:58:01.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.252556+0000", 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4190000000000004e-06 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268416+0000", 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015859560000000002 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.268422+0000", 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3260000000000001e-06 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460357+0000", 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.191935119 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460440+0000", 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3042999999999998e-05 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,7", 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.460526+0000", 2024-09-13T14:58:01.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5461000000000001e-05 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.468686+0000", 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081603790000000006 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.468721+0000", 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5167999999999999e-05 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:01.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.474072+0000", 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053510579999999997 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.474099+0000", 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6495e-05 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.474149+0000", 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0432000000000001e-05 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.235 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.236 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.236 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:01.236 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:01.236 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:58:01.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:01.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:01.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:01.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:01.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:01.567 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:01.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:01.696 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:01.772 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-13T14:58:01.772 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:01.772 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-13T14:58:01.772 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:58:01.772 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:58:01.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-13T14:58:01.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:01.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:01.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:01.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:01.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:02.079 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:02.096 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:02.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:02.175 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:58:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:58:01] "GET /metrics HTTP/1.1" 200 36518 "" "Prometheus/2.51.0" 2024-09-13T14:58:02.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:02.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:02.335 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:58:02.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:02.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:02.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:02 smithi154 ceph-mon[102314]: pgmap v917: 89 pgs: 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 4 remapped+peering, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 571 op/s; 16854/141276 objects degraded (11.930%); 2900/141276 objects misplaced (2.053%); 6.0 MiB/s, 2 keys/s, 105 objects/s recovering 2024-09-13T14:58:02.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2941277652' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:02.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:02 smithi154 ceph-mon[107038]: pgmap v917: 89 pgs: 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 4 remapped+peering, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 571 op/s; 16854/141276 objects degraded (11.930%); 2900/141276 objects misplaced (2.053%); 6.0 MiB/s, 2 keys/s, 105 objects/s recovering 2024-09-13T14:58:02.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2941277652' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:02 smithi172 ceph-mon[100696]: pgmap v917: 89 pgs: 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfill_wait, 4 remapped+peering, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 571 op/s; 16854/141276 objects degraded (11.930%); 2900/141276 objects misplaced (2.053%); 6.0 MiB/s, 2 keys/s, 105 objects/s recovering 2024-09-13T14:58:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2941277652' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:02.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:02.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:02.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:02.693 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:02.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:02.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:02.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:02.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:02.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:03.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:03.102 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:03.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:03.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:03.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:03.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:03.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:03.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:03.532 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:03 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:03.532 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2941277652' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:03.532 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:03 smithi172 ceph-mon[100696]: osdmap e290: 8 total, 7 up, 8 in 2024-09-13T14:58:03.544 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:03.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:03 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:03.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2941277652' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:03.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:03 smithi154 ceph-mon[102314]: osdmap e290: 8 total, 7 up, 8 in 2024-09-13T14:58:03.586 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:03 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:03.586 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2941277652' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:03.586 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:03 smithi154 ceph-mon[107038]: osdmap e290: 8 total, 7 up, 8 in 2024-09-13T14:58:03.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:58:03.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:03.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:03.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:03.779 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:03.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:03.916 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:03.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.008 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:58:04.008 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:58:04.008 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:58:04.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:04.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:04.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:04.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:04 smithi154 ceph-mon[102314]: pgmap v919: 89 pgs: 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 19617/137487 objects degraded (14.268%); 11168/137487 objects misplaced (8.123%); 5.0 MiB/s, 1 keys/s, 87 objects/s recovering 2024-09-13T14:58:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:04 smithi154 ceph-mon[107038]: pgmap v919: 89 pgs: 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 19617/137487 objects degraded (14.268%); 11168/137487 objects misplaced (8.123%); 5.0 MiB/s, 1 keys/s, 87 objects/s recovering 2024-09-13T14:58:04.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.553 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:04.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:04 smithi172 ceph-mon[100696]: pgmap v919: 89 pgs: 4 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait, 1 remapped+peering, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 532 op/s; 19617/137487 objects degraded (14.268%); 11168/137487 objects misplaced (8.123%); 5.0 MiB/s, 1 keys/s, 87 objects/s recovering 2024-09-13T14:58:04.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.692 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:04.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:04.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:04.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:04.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:04.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:04.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:05.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:05.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:05.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:05.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:05.388 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:05.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19617/137487 objects degraded (14.268%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:05.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19617/137487 objects degraded (14.268%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:05.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:05.522 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:05.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:05.657 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19617/137487 objects degraded (14.268%), 9 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:05.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:05.789 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:05.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:05.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:05.947 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:05.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:06.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:06.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:06.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:06.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:58:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:06.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:06.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:06.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:06.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:06 smithi154 ceph-mon[102314]: pgmap v920: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 60 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 16520/129069 objects degraded (12.799%); 10902/129069 objects misplaced (8.447%); 20 MiB/s, 1 keys/s, 322 objects/s recovering 2024-09-13T14:58:06.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:06 smithi154 ceph-mon[107038]: pgmap v920: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 60 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 16520/129069 objects degraded (12.799%); 10902/129069 objects misplaced (8.447%); 20 MiB/s, 1 keys/s, 322 objects/s recovering 2024-09-13T14:58:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:06.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:06.564 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:58:06.565 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:06.596 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:58:06.597 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:06.597 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:58:06.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:06.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:06.628 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:58:06.629 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:06.658 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:58:06.658 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:06.691 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:58:06.691 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:06.729 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:58:06.729 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:06.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:06 smithi172 ceph-mon[100696]: pgmap v920: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 60 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 16520/129069 objects degraded (12.799%); 10902/129069 objects misplaced (8.447%); 20 MiB/s, 1 keys/s, 322 objects/s recovering 2024-09-13T14:58:06.756 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:06.764 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:58:06.764 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:06.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:06.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:06.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:07.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:07.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:07.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:07.177 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:07.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:07.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:07.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:07.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:07.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:07.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:07.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:07.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:07.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4032480263' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:07.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4032480263' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4032480263' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:07.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:07.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:07.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:58:07.918 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:58:07.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:07.948 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:07.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:07.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:08.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:08.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:08.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.196 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:08.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:08.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.320 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:08.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:08.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:08.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:08.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:08.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:08.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:08.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:08 smithi154 ceph-mon[102314]: pgmap v921: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 60 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 16520/129069 objects degraded (12.799%); 10902/129069 objects misplaced (8.447%); 13 MiB/s, 215 objects/s recovering 2024-09-13T14:58:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:08 smithi154 ceph-mon[102314]: osdmap e291: 8 total, 7 up, 8 in 2024-09-13T14:58:08.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:08 smithi154 ceph-mon[107038]: pgmap v921: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 60 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 16520/129069 objects degraded (12.799%); 10902/129069 objects misplaced (8.447%); 13 MiB/s, 215 objects/s recovering 2024-09-13T14:58:08.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:08 smithi154 ceph-mon[107038]: osdmap e291: 8 total, 7 up, 8 in 2024-09-13T14:58:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:08 smithi172 ceph-mon[100696]: pgmap v921: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 60 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 677 op/s; 16520/129069 objects degraded (12.799%); 10902/129069 objects misplaced (8.447%); 13 MiB/s, 215 objects/s recovering 2024-09-13T14:58:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:08 smithi172 ceph-mon[100696]: osdmap e291: 8 total, 7 up, 8 in 2024-09-13T14:58:08.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:08.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:08.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:08.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:08.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:09.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:09.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:09.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:09.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:09.296 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:09.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:09.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:09.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:09.516 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:09.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:09.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:09.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:09.709 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:09 smithi154 ceph-mon[107038]: osdmap e292: 8 total, 7 up, 8 in 2024-09-13T14:58:09.709 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:09 smithi154 ceph-mon[102314]: osdmap e292: 8 total, 7 up, 8 in 2024-09-13T14:58:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:09 smithi172 ceph-mon[100696]: osdmap e292: 8 total, 7 up, 8 in 2024-09-13T14:58:09.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:09.802 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:09.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:09.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:09.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:09.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:10.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:10.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:10.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:10.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:10.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:10.445 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:10.445 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:10.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:10.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:10.528 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:10 smithi154 ceph-mon[102314]: pgmap v924: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 15571/123051 objects degraded (12.654%); 10095/123051 objects misplaced (8.204%); 22 MiB/s, 348 objects/s recovering 2024-09-13T14:58:10.528 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 15571/123051 objects degraded (12.654%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:10.528 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:10 smithi154 ceph-mon[107038]: pgmap v924: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 15571/123051 objects degraded (12.654%); 10095/123051 objects misplaced (8.204%); 22 MiB/s, 348 objects/s recovering 2024-09-13T14:58:10.528 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 15571/123051 objects degraded (12.654%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:10.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:10.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:10.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:10.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:10 smithi172 ceph-mon[100696]: pgmap v924: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 15571/123051 objects degraded (12.654%); 10095/123051 objects misplaced (8.204%); 22 MiB/s, 348 objects/s recovering 2024-09-13T14:58:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 15571/123051 objects degraded (12.654%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:10.745 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:10.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:10.872 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-13T14:58:10.872 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:10.872 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T14:58:10.873 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T14:58:10.873 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T14:58:10.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T14:58:10.877 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:10.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:10.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:11.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:11.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:11.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:11.204 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:11.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:11.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:11.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:11.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:11.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:11.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:11.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:11.722 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:11.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:11.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:11.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:11.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:12.046 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:58:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:58:11] "GET /metrics HTTP/1.1" 200 36518 "" "Prometheus/2.51.0" 2024-09-13T14:58:12.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:12.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.230 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:12.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:12.260 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:12.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:58:12.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:12.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:12.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:12.511 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:12 smithi154 ceph-mon[102314]: pgmap v925: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 14391/117369 objects degraded (12.261%); 9854/117369 objects misplaced (8.396%); 18 MiB/s, 295 objects/s recovering 2024-09-13T14:58:12.511 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:12 smithi154 ceph-mon[107038]: pgmap v925: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 14391/117369 objects degraded (12.261%); 9854/117369 objects misplaced (8.396%); 18 MiB/s, 295 objects/s recovering 2024-09-13T14:58:12.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:12.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:12.655 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:12.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:12 smithi172 ceph-mon[100696]: pgmap v925: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 14391/117369 objects degraded (12.261%); 9854/117369 objects misplaced (8.396%); 18 MiB/s, 295 objects/s recovering 2024-09-13T14:58:12.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:12.784 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:12.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:12.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:12.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:13.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:13.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:13.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:13.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:13.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.346 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:13.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:13.417 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:58:13.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:13.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:13.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:13.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:13.632 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2679489374' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:13.633 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:13 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:13.633 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:13.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2679489374' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:13 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:13.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2679489374' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:13 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:13.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:13.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:13.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:13.890 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:13.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:13.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:14.019 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:14.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:14.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:14.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:14.294 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:14.372 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:58:14.372 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T14:58:14.372 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:58:14.396 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.432 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:14.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:14 smithi172 ceph-mon[100696]: pgmap v926: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 13199/113754 objects degraded (11.603%); 9141/113754 objects misplaced (8.036%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:58:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:14 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:14 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:14 smithi172 ceph-mon[100696]: osdmap e293: 8 total, 7 up, 8 in 2024-09-13T14:58:14.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:14.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:14.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.697 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:14.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[102314]: pgmap v926: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 13199/113754 objects degraded (11.603%); 9141/113754 objects misplaced (8.036%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:58:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[102314]: osdmap e293: 8 total, 7 up, 8 in 2024-09-13T14:58:14.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[107038]: pgmap v926: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 13199/113754 objects degraded (11.603%); 9141/113754 objects misplaced (8.036%); 13 MiB/s, 213 objects/s recovering 2024-09-13T14:58:14.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:14.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:14.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:14 smithi154 ceph-mon[107038]: osdmap e293: 8 total, 7 up, 8 in 2024-09-13T14:58:14.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:14.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:14.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:14.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:14.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:15.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:15.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:15.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:15.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:15.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:15.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:15.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:15.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:15.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:15.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:15.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 13199/113754 objects degraded (11.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:15.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 13199/113754 objects degraded (11.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 13199/113754 objects degraded (11.603%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:15.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.778 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:15.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:15.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:15.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:15.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:15.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:16.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:16.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:16.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:16.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:16.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:16.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:16.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:16.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:16.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:16.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:16.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:16.633 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:16 smithi154 ceph-mon[102314]: pgmap v928: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11853/104973 objects degraded (11.291%); 8867/104973 objects misplaced (8.447%); 13 MiB/s, 209 objects/s recovering 2024-09-13T14:58:16.633 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:16 smithi154 ceph-mon[107038]: pgmap v928: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11853/104973 objects degraded (11.291%); 8867/104973 objects misplaced (8.447%); 13 MiB/s, 209 objects/s recovering 2024-09-13T14:58:16.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:16.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:16 smithi172 ceph-mon[100696]: pgmap v928: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 857 op/s; 11853/104973 objects degraded (11.291%); 8867/104973 objects misplaced (8.447%); 13 MiB/s, 209 objects/s recovering 2024-09-13T14:58:16.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:16.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:16.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:16.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:16.995 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:58:16.996 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:17.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:17.023 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:17.032 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:58:17.032 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:17.036 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:58:17.078 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:58:17.078 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:17.114 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:58:17.115 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:17.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:17.152 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:58:17.153 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:17.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:17.181 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:58:17.181 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:17.212 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:58:17.213 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:17.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:17.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:17.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:17.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:17.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:17.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:17.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:17.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:17.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:17 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2684168676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:17.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:17 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2684168676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:17 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2684168676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:17.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:17.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:17.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:17.979 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:17.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:18.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:18.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:18.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:18.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:18.348 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:18.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:58:18.378 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:18.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:18.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:18.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:18.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:18.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:18 smithi154 ceph-mon[102314]: pgmap v929: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 760 op/s; 11853/104973 objects degraded (11.291%); 8867/104973 objects misplaced (8.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T14:58:18.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:18 smithi154 ceph-mon[107038]: pgmap v929: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 760 op/s; 11853/104973 objects degraded (11.291%); 8867/104973 objects misplaced (8.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T14:58:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:18 smithi172 ceph-mon[100696]: pgmap v929: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 760 op/s; 11853/104973 objects degraded (11.291%); 8867/104973 objects misplaced (8.447%); 12 MiB/s, 185 objects/s recovering 2024-09-13T14:58:18.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:18.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:18.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:18.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:18.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:18.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:19.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:19.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:19.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:19.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:19.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.308 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:19.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:19.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:19.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:19.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:19.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:19.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:19.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:19.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:58:19.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:58:19.893 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:19.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:19.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:20.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:20.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:20.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:20.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:20.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:20.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:20.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:20.342 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-13T14:58:20.342 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:20.343 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T14:58:20.343 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-13T14:58:20.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 0 2024-09-13T14:58:20.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:20.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:20.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:20.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:20.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:20.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:20 smithi154 ceph-mon[102314]: pgmap v930: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 11196/98463 objects degraded (11.371%); 8050/98463 objects misplaced (8.176%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T14:58:20.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 11196/98463 objects degraded (11.371%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:20 smithi154 ceph-mon[107038]: pgmap v930: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 11196/98463 objects degraded (11.371%); 8050/98463 objects misplaced (8.176%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T14:58:20.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 11196/98463 objects degraded (11.371%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:20.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:20 smithi172 ceph-mon[100696]: pgmap v930: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 11196/98463 objects degraded (11.371%); 8050/98463 objects misplaced (8.176%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T14:58:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 11196/98463 objects degraded (11.371%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:20.821 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:58:20.821 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:58:20.821 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T14:58:20.821 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T14:58:20.821 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:58:20.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:20.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:20.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:20.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:21.014 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:21.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:21.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:21.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:21.291 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:21.393 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:21.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:21.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:21.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:21.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:21.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:21.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:21.860 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:21.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:21.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:21.986 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:21.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:22.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:22.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:58:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:58:21] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-09-13T14:58:22.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:22.274 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:22.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:22.418 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:22.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.546 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:22.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.671 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:22.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:22 smithi154 ceph-mon[102314]: pgmap v931: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 9962/92439 objects degraded (10.777%); 7820/92439 objects misplaced (8.460%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T14:58:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:22 smithi154 ceph-mon[107038]: pgmap v931: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 9962/92439 objects degraded (10.777%); 7820/92439 objects misplaced (8.460%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T14:58:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:22 smithi172 ceph-mon[100696]: pgmap v931: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 9962/92439 objects degraded (10.777%); 7820/92439 objects misplaced (8.460%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T14:58:22.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.808 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:22.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:22.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:22.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:22.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:23.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:23.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:23.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:23.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:23.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:23.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:23.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:23.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:23.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:23.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:23.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:23.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:23.873 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:23.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:23.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:24.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:24.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:24.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:24.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:24.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:24.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:24.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:24.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:24.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:24.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:24 smithi172 ceph-mon[100696]: pgmap v932: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 9412/88650 objects degraded (10.617%); 7057/88650 objects misplaced (7.961%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:58:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:24 smithi172 ceph-mon[100696]: osdmap e294: 8 total, 7 up, 8 in 2024-09-13T14:58:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:24.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:24.796 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:24 smithi154 ceph-mon[102314]: pgmap v932: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 9412/88650 objects degraded (10.617%); 7057/88650 objects misplaced (7.961%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:58:24.796 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:24 smithi154 ceph-mon[102314]: osdmap e294: 8 total, 7 up, 8 in 2024-09-13T14:58:24.798 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:24 smithi154 ceph-mon[107038]: pgmap v932: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 9412/88650 objects degraded (10.617%); 7057/88650 objects misplaced (7.961%); 6.5 MiB/s, 103 objects/s recovering 2024-09-13T14:58:24.798 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:24 smithi154 ceph-mon[107038]: osdmap e294: 8 total, 7 up, 8 in 2024-09-13T14:58:24.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:58:24.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:24.900 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:24.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:25.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:25.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:25.114 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:25.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:25.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:25.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:25.352 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:25.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.485 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:25.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:25.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:25.613 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:25.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:25.691 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:25 smithi154 ceph-mon[107038]: osdmap e295: 8 total, 7 up, 8 in 2024-09-13T14:58:25.691 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 9412/88650 objects degraded (10.617%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:25.691 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:25 smithi154 ceph-mon[102314]: osdmap e295: 8 total, 7 up, 8 in 2024-09-13T14:58:25.691 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 9412/88650 objects degraded (10.617%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:25.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:25.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:25.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:25 smithi172 ceph-mon[100696]: osdmap e295: 8 total, 7 up, 8 in 2024-09-13T14:58:25.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 9412/88650 objects degraded (10.617%), 8 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:25.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:25.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:25.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:25.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:26.011 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:26.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:26.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:26.228 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:26.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:26.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:26.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:26.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:26.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:26.517 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:58:26.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:26.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:26.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:26.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:26.763 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:26 smithi154 ceph-mon[102314]: pgmap v935: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 7633/81435 objects degraded (9.373%); 5106/81435 objects misplaced (6.270%); 13 MiB/s, 208 objects/s recovering 2024-09-13T14:58:26.763 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2771862406' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:26.763 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:26 smithi154 ceph-mon[107038]: pgmap v935: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 7633/81435 objects degraded (9.373%); 5106/81435 objects misplaced (6.270%); 13 MiB/s, 208 objects/s recovering 2024-09-13T14:58:26.763 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2771862406' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:26.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:26.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:26.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:26.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:26.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:26 smithi172 ceph-mon[100696]: pgmap v935: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 7633/81435 objects degraded (9.373%); 5106/81435 objects misplaced (6.270%); 13 MiB/s, 208 objects/s recovering 2024-09-13T14:58:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2771862406' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:58:27.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:27.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:27.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:27.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:27.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:27.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:27.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:27.492 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:27.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:27.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:27.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:27 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:27.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2771862406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:27.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:27 smithi154 ceph-mon[102314]: osdmap e296: 8 total, 7 up, 8 in 2024-09-13T14:58:27.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:27 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:27.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2771862406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:27.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:27 smithi154 ceph-mon[107038]: osdmap e296: 8 total, 7 up, 8 in 2024-09-13T14:58:27.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:27.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:27.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:27.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:27.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:27.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:27 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2771862406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:58:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:27 smithi172 ceph-mon[100696]: osdmap e296: 8 total, 7 up, 8 in 2024-09-13T14:58:28.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:28.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:28.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.305 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:28.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:28.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:28.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:28.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:28.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:28.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:28.817 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:28 smithi154 ceph-mon[102314]: pgmap v937: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 7633/80742 objects degraded (9.454%); 5106/80742 objects misplaced (6.324%); 12 MiB/s, 190 objects/s recovering 2024-09-13T14:58:28.817 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:28.817 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:28 smithi154 ceph-mon[107038]: pgmap v937: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 7633/80742 objects degraded (9.454%); 5106/80742 objects misplaced (6.324%); 12 MiB/s, 190 objects/s recovering 2024-09-13T14:58:28.818 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:28.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:28.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:28.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:28 smithi172 ceph-mon[100696]: pgmap v937: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 remapped+peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 649 op/s; 7633/80742 objects degraded (9.454%); 5106/80742 objects misplaced (6.324%); 12 MiB/s, 190 objects/s recovering 2024-09-13T14:58:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:28.996 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:28.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:29.127 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:29.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:29.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:29.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:29.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:29.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:58:29.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:29.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:29.798 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:29.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:29.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:29.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:30.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:30.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:30.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:30.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:30.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:30.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:30.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:30.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:30.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:30.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:30.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:30.668 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:30.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:30 smithi154 ceph-mon[107038]: pgmap v938: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 7633/74967 objects degraded (10.182%); 6034/74967 objects misplaced (8.049%); 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:58:30.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7633/74967 objects degraded (10.182%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:30.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:30.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:30 smithi154 ceph-mon[102314]: pgmap v938: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 7633/74967 objects degraded (10.182%); 6034/74967 objects misplaced (8.049%); 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:58:30.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7633/74967 objects degraded (10.182%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:30.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:30.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:30.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:30.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:30.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:30.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:30.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:30 smithi172 ceph-mon[100696]: pgmap v938: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.5 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 7633/74967 objects degraded (10.182%); 6034/74967 objects misplaced (8.049%); 6.6 MiB/s, 106 objects/s recovering 2024-09-13T14:58:30.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7633/74967 objects degraded (10.182%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T14:58:30.994 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:58:31.011 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:31.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:31.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:31.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:31.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:31.382 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86673 6.4 6:24d0d169:::benchmark_data_smithi154_175640_object897:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.507975+0000", 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86940412199999, 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.107262861999999, 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.383 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86673 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507975+0000", 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507975+0000", 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507976+0000", 2024-09-13T14:58:31.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.502e-06 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507981+0000", 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0769999999999998e-06 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.507998+0000", 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6201999999999998e-05 2024-09-13T14:58:31.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508005+0000", 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.379e-06 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508217+0000", 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021226200000000001 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508230+0000", 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2646999999999999e-05 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.032888+0000", 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.524658221999999 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.032900+0000", 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1522000000000001e-05 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793624+0000", 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.76072437800000003 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793635+0000", 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1229999999999999e-05 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.513259+0000", 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7196241109999999 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.515815+0000", 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025560069999999999 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.612791+0000", 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.096975483000000001 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.615238+0000", 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00244684 2024-09-13T14:58:31.391 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86674 6.4 6:26d83999:::benchmark_data_smithi154_175640_object898:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86936198499995, 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.073532165000003, 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.392 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86674 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508017+0000", 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4999999999999998e-07 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508020+0000", 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1670000000000002e-06 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508026+0000", 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0850000000000002e-06 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508030+0000", 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5229999999999997e-06 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508255+0000", 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022519099999999999 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508264+0000", 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3420000000000008e-06 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036648+0000", 2024-09-13T14:58:31.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528384581000001 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036660+0000", 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1253999999999999e-05 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793661+0000", 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75700164999999997 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793669+0000", 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6190000000000002e-06 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.524233+0000", 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7305637449999995 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.528990+0000", 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047570019999999998 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.536276+0000", 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072860319999999996 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.567951+0000", 2024-09-13T14:58:31.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031675805000000001 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.569224+0000", 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012729099999999999 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.573311+0000", 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040869019999999999 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.575675+0000", 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023637990000000002 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.581549+0000", 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058741080000000003 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86702 6.4 6:2bd62bb7:::benchmark_data_smithi154_175640_object926:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86929220399998, 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.084438992000003, 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.403 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86702 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:58:31.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508087+0000", 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.89e-07 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508090+0000", 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3240000000000001e-06 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508097+0000", 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3170000000000004e-06 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508103+0000", 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.314e-06 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508291+0000", 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018816199999999999 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508300+0000", 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2199999999999992e-06 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036687+0000", 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528387514999999 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036695+0000", 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3590000000000008e-06 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793701+0000", 2024-09-13T14:58:31.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75700578900000004 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793708+0000", 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3210000000000001e-06 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.552107+0000", 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7583998209999994 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.556007+0000", 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038999799999999999 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.561750+0000", 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057424920000000001 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.572199+0000", 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010448756999999999 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.583202+0000", 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011003811 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.584404+0000", 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012016279999999999 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.586113+0000", 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001708839 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.592526+0000", 2024-09-13T14:58:31.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064129540000000002 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86716 6.4 6:26c9dec4:::benchmark_data_smithi154_175640_object940:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86926908299995, 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.150778488999997, 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.413 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86716 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508110+0000", 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.22e-07 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508112+0000", 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6869999999999999e-06 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508117+0000", 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2780000000000001e-06 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508121+0000", 2024-09-13T14:58:31.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.155e-06 2024-09-13T14:58:31.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508328+0000", 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000206162 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508336+0000", 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2819999999999996e-06 2024-09-13T14:58:31.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036721+0000", 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528384908 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036742+0000", 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1185000000000001e-05 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793733+0000", 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75699062800000005 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793738+0000", 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8939999999999998e-06 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.615597+0000", 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8218581789999995 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.619094+0000", 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003497432 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.624815+0000", 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057204880000000001 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.647410+0000", 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022595578000000002 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.648555+0000", 2024-09-13T14:58:31.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001144897 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.649148+0000", 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00059299699999999995 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.650609+0000", 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014611839999999999 2024-09-13T14:58:31.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.658888+0000", 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082792330000000004 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86721 6.4 6:2594c329:::benchmark_data_smithi154_175640_object945:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.425 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508126+0000", 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86925259600002, 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.183112059999999, 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.426 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86721 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508126+0000", 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508126+0000", 2024-09-13T14:58:31.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508127+0000", 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7300000000000002e-07 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508128+0000", 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4479999999999999e-06 2024-09-13T14:58:31.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508135+0000", 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5230000000000001e-06 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508139+0000", 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2980000000000002e-06 2024-09-13T14:58:31.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508364+0000", 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022497700000000001 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508372+0000", 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2500000000000006e-06 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036772+0000", 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528400153 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036782+0000", 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7280000000000007e-06 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793773+0000", 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75699076899999995 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793780+0000", 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8539999999999999e-06 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.629719+0000", 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8359395529999993 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.633166+0000", 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034471340000000001 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.640075+0000", 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069084089999999999 2024-09-13T14:58:31.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.659963+0000", 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01988848 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.661225+0000", 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012620699999999999 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.679879+0000", 2024-09-13T14:58:31.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018653508999999999 2024-09-13T14:58:31.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.681788+0000", 2024-09-13T14:58:31.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019092429999999999 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.691238+0000", 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009450389 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86742 6.4 6:271a548d:::benchmark_data_smithi154_175640_object966:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86923523099995, 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.218343629000003, 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.437 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86742 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508144+0000", 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7000000000000001e-07 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508146+0000", 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6649999999999999e-06 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508151+0000", 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4600000000000002e-06 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508155+0000", 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1250000000000003e-06 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508399+0000", 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000243964 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508407+0000", 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9079999999999995e-06 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036823+0000", 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528415599999999 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036831+0000", 2024-09-13T14:58:31.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2940000000000006e-06 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793809+0000", 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75697761200000002 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793816+0000", 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8510000000000001e-06 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.661864+0000", 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8680481600000007 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.666377+0000", 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045136229999999996 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.672872+0000", 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064947770000000002 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.713116+0000", 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.040243565000000002 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.714917+0000", 2024-09-13T14:58:31.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001801265 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.715728+0000", 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081117899999999998 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.717401+0000", 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001672782 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.726487+0000", 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090865289999999994 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86748 6.4 6:218d5a01:::benchmark_data_smithi154_175640_object972:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:58:31.447 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.869219026, 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.256619432000001, 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86748 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508160+0000", 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8200000000000001e-07 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508162+0000", 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6789999999999999e-06 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508167+0000", 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0690000000000001e-06 2024-09-13T14:58:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508171+0000", 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0080000000000004e-06 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508433+0000", 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026166600000000003 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508440+0000", 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.52e-06 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036906+0000", 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528465809 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036914+0000", 2024-09-13T14:58:31.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7260000000000002e-06 2024-09-13T14:58:31.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793841+0000", 2024-09-13T14:58:31.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75692729299999995 2024-09-13T14:58:31.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793847+0000", 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4189999999999999e-06 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.691732+0000", 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8978850040000008 2024-09-13T14:58:31.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.696147+0000", 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044142169999999998 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.702810+0000", 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066635089999999998 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.727854+0000", 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025043894000000001 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.752826+0000", 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024971942 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.753744+0000", 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00091785 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.755632+0000", 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018883229999999999 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.764779+0000", 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009147222 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86764 6.4 6:2ddc6785:::benchmark_data_smithi154_175640_object988:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86920318600005, 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.304434260000001, 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.458 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86764 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:58:31.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508176+0000", 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9299999999999999e-07 2024-09-13T14:58:31.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508178+0000", 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6339999999999999e-06 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508183+0000", 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.797e-06 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508187+0000", 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1169999999999997e-06 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508464+0000", 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000277481 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508472+0000", 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1929999999999993e-06 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036953+0000", 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528480492 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.036963+0000", 2024-09-13T14:58:31.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0277e-05 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793881+0000", 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.756918337 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793888+0000", 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7190000000000004e-06 2024-09-13T14:58:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.729082+0000", 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9351940640000009 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.737635+0000", 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085531549999999998 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.744938+0000", 2024-09-13T14:58:31.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007302411 2024-09-13T14:58:31.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.766104+0000", 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021166570999999999 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.767670+0000", 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015657780000000001 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.800097+0000", 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032427072000000001 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.803677+0000", 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035797239999999998 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.812610+0000", 2024-09-13T14:58:31.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008933145 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86766 6.4 6:20e2c325:::benchmark_data_smithi154_175640_object990:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86908945499999, 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.350745707000002, 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.469 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86766 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508290+0000", 2024-09-13T14:58:31.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9400000000000001e-07 2024-09-13T14:58:31.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508292+0000", 2024-09-13T14:58:31.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.215e-06 2024-09-13T14:58:31.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508299+0000", 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5699999999999998e-06 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508304+0000", 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9840000000000001e-06 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508500+0000", 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000195916 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508508+0000", 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.191e-06 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037024+0000", 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528515762000001 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037031+0000", 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6879999999999999e-06 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793912+0000", 2024-09-13T14:58:31.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75688088399999998 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793919+0000", 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6259999999999997e-06 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.771094+0000", 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9771756180000004 2024-09-13T14:58:31.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.776563+0000", 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054689760000000004 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.785532+0000", 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089686279999999993 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.796845+0000", 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011312971 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.798420+0000", 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015745500000000001 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.849219+0000", 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.050798994 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.850702+0000", 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001483053 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.859035+0000", 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083336869999999993 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86785 6.4 6:2ff89860:::benchmark_data_smithi154_175640_object1009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86901935599997, 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.401965842000003, 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.482 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86785 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:58:31.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508360+0000", 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7500000000000001e-07 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508362+0000", 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.323e-06 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508369+0000", 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3759999999999996e-06 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508374+0000", 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508532+0000", 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000158066 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508541+0000", 2024-09-13T14:58:31.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9740000000000008e-06 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037073+0000", 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528531924999999 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037081+0000", 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3380000000000005e-06 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793943+0000", 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75686249800000005 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793950+0000", 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8959999999999997e-06 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.813095+0000", 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.019144845 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.817234+0000", 2024-09-13T14:58:31.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041391170000000003 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.822708+0000", 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054735089999999997 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.860019+0000", 2024-09-13T14:58:31.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037311534 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.860598+0000", 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000578781 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.900474+0000", 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039876138999999998 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.902193+0000", 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001718844 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.910326+0000", 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081324269999999994 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.492 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86817 6.4 6:29f76746:::benchmark_data_smithi154_175640_object1041:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.508444+0000", 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.86893456400003, 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.427115067999999, 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86817 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508444+0000", 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508444+0000", 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508445+0000", 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0999999999999998e-07 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508448+0000", 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.416e-06 2024-09-13T14:58:31.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508458+0000", 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4469999999999995e-06 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508465+0000", 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3610000000000003e-06 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508567+0000", 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010213900000000001 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.508576+0000", 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3329999999999997e-06 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037114+0000", 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.528537827000001 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037121+0000", 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0129999999999996e-06 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793974+0000", 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75685360199999996 2024-09-13T14:58:31.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.793981+0000", 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5799999999999997e-06 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.829986+0000", 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.036004924 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.836767+0000", 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067812799999999998 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.842365+0000", 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055977359999999999 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.860883+0000", 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018518458000000002 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.862019+0000", 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011355339999999999 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.920345+0000", 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.058326714000000002 2024-09-13T14:58:31.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.924394+0000", 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004048856 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.935560+0000", 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011165237999999999 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.502 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86824 6.4 6:26c7fd74:::benchmark_data_smithi154_175640_object1048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.849029+0000", 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.52834983100001, 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.125902050000001, 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.503 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86824 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849029+0000", 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849029+0000", 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849031+0000", 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.649e-06 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849035+0000", 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8190000000000002e-06 2024-09-13T14:58:31.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849045+0000", 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.047e-05 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849050+0000", 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1139999999999997e-06 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849111+0000", 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0603999999999998e-05 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.849127+0000", 2024-09-13T14:58:31.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6089000000000001e-05 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037169+0000", 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.188041861999999 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037178+0000", 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7080000000000003e-06 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794006+0000", 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75682800400000005 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794012+0000", 2024-09-13T14:58:31.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6549999999999998e-06 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.865282+0000", 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.071270213 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.870806+0000", 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055233510000000001 2024-09-13T14:58:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.876156+0000", 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053498440000000003 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.911402+0000", 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035246016999999998 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.912156+0000", 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00075456099999999995 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.957551+0000", 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.045395104999999998 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.960921+0000", 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00337002 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.974931+0000", 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014009964999999999 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86833 6.4 6:299146a5:::benchmark_data_smithi154_175640_object1057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.876366+0000", 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.50101326000004, 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.111824777000002, 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.514 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86833 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876366+0000", 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876366+0000", 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876367+0000", 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.375e-06 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876372+0000", 2024-09-13T14:58:31.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9640000000000002e-06 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876384+0000", 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2367e-05 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876391+0000", 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7819999999999996e-06 2024-09-13T14:58:31.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876457+0000", 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5437000000000004e-05 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.876468+0000", 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.152e-05 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037217+0000", 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.160749016 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037225+0000", 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0350000000000001e-06 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794036+0000", 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75681060600000005 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794042+0000", 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5740000000000001e-06 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.881409+0000", 2024-09-13T14:58:31.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.087366431 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.886690+0000", 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052807449999999999 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.893946+0000", 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072567079999999997 2024-09-13T14:58:31.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.913284+0000", 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019337656000000002 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.917600+0000", 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043160029999999997 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.977875+0000", 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.060274600999999997 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.980107+0000", 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002232394 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.988191+0000", 2024-09-13T14:58:31.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080835630000000002 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86857 6.4 6:28ca1b9c:::benchmark_data_smithi154_175640_object1081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:58.966732+0000", 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.41064663199995, 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 34.108482150999997, 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.524 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86857 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966732+0000", 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966732+0000", 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966734+0000", 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.897e-06 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966741+0000", 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1729999999999998e-06 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966759+0000", 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7082999999999999e-05 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966772+0000", 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2978e-05 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966839+0000", 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7847000000000003e-05 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:58.966852+0000", 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2519e-05 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037265+0000", 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 23.070413295000002 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037273+0000", 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8490000000000006e-06 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794068+0000", 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75679514800000003 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794075+0000", 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6409999999999996e-06 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.936087+0000", 2024-09-13T14:58:31.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.14201224 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.941272+0000", 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051847730000000002 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.947691+0000", 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064196419999999997 2024-09-13T14:58:31.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.976256+0000", 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028564672999999999 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.022507+0000", 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046251160999999999 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.064329+0000", 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041822152000000001 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.066135+0000", 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018055149999999999 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.075215+0000", 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090795649999999995 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86871 6.4 6:2f2f4832:::benchmark_data_smithi154_175640_object1095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:49:59.156758+0000", 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 512.22062056300001, 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 33.933005207999997, 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.535 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86871 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156758+0000", 2024-09-13T14:58:31.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156758+0000", 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156760+0000", 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4759999999999999e-06 2024-09-13T14:58:31.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156764+0000", 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3340000000000003e-06 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156776+0000", 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1487e-05 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156784+0000", 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0630000000000006e-06 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156838+0000", 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4119999999999997e-05 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:49:59.156846+0000", 2024-09-13T14:58:31.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6590000000000007e-06 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037312+0000", 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 22.880466437999999 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.037319+0000", 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3950000000000003e-06 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794099+0000", 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.75677971600000005 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:22.794103+0000", 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0219999999999998e-06 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.988616+0000", 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 10.194513259000001 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.993901+0000", 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052842210000000004 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:31.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.000598+0000", 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066976589999999999 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.023408+0000", 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.02280983 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.025486+0000", 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020782859999999999 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.078543+0000", 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.053056568999999998 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.080957+0000", 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002413583 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.089764+0000", 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088070909999999995 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86889 6.4 6:247588cb:::benchmark_data_smithi154_175640_object1113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.582787+0000", 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 478.794591561, 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.52048566299999999, 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.548 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86889 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582787+0000", 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582787+0000", 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582789+0000", 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.251e-06 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582794+0000", 2024-09-13T14:58:31.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0529999999999997e-06 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582808+0000", 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4511999999999999e-05 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.582817+0000", 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2819999999999996e-06 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.007485+0000", 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42466857899999999 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.011316+0000", 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00383096 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.016788+0000", 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054720619999999998 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.024771+0000", 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079829429999999993 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.025835+0000", 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010641649999999999 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.092991+0000", 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.067156034000000003 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.095426+0000", 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024348059999999999 2024-09-13T14:58:31.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.103273+0000", 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078470160000000001 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86896 6.4 6:21378335:::benchmark_data_smithi154_175640_object1120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:58:31.555 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.622457+0000", 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 478.75492180499998, 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.49349832399999999, 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86896 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.556 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622457+0000", 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622457+0000", 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622458+0000", 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0839999999999999e-06 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622463+0000", 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5619999999999997e-06 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622473+0000", 2024-09-13T14:58:31.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0098000000000001e-05 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.622480+0000", 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2049999999999996e-06 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.028350+0000", 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.40586974300000001 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.034726+0000", 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063761699999999996 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.040426+0000", 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056998259999999999 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.077031+0000", 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036605407 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.090146+0000", 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013115096999999999 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.104571+0000", 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014424655999999999 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.106974+0000", 2024-09-13T14:58:31.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024029089999999999 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.115956+0000", 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089815669999999993 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.563 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86923 6.4 6:244d3275:::benchmark_data_smithi154_175640_object1147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.721799+0000", 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 478.65557978700002, 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.40723228500000003, 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86923 2024-09-13T14:58:31.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721799+0000", 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721799+0000", 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721802+0000", 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4559999999999999e-06 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721812+0000", 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8789999999999998e-06 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721826+0000", 2024-09-13T14:58:31.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4837999999999999e-05 2024-09-13T14:58:31.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.721837+0000", 2024-09-13T14:58:31.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0467e-05 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.046967+0000", 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.325129742 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.051648+0000", 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046818500000000004 2024-09-13T14:58:31.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.058047+0000", 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063980570000000004 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.090506+0000", 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032459384000000001 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.091521+0000", 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010147190000000001 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.117303+0000", 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025781999999999999 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.119836+0000", 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025335409999999998 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.129032+0000", 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091953520000000004 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:86953 6.4 6:2cb54e5d:::benchmark_data_smithi154_175640_object1177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:32.805319+0000", 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 478.572059841, 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42385360999999999, 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.572 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 86953 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805319+0000", 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805319+0000", 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805321+0000", 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.516e-06 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805327+0000", 2024-09-13T14:58:31.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1800000000000001e-06 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805343+0000", 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6021000000000001e-05 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:32.805351+0000", 2024-09-13T14:58:31.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0569999999999993e-06 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.129532+0000", 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.32418106200000002 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.134486+0000", 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049543770000000003 2024-09-13T14:58:31.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.140485+0000", 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059984080000000002 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.168220+0000", 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027735471000000001 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.217857+0000", 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049637037000000002 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.219707+0000", 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001849982 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.221214+0000", 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001507017 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.229173+0000", 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079584819999999994 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.579 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.44843.0:87070 6.4 6:25334854:::benchmark_data_smithi154_175640_object1294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:50:33.147617+0000", 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 478.22976157800002, 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.41807496500000002, 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:58:31.580 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.44843", 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 87070 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147617+0000", 2024-09-13T14:58:31.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147617+0000", 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147619+0000", 2024-09-13T14:58:31.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6249999999999999e-06 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147626+0000", 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4359999999999999e-06 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:58:31.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147637+0000", 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1781e-05 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.147645+0000", 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9160000000000001e-06 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.428804+0000", 2024-09-13T14:58:31.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.28115832400000002 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.433717+0000", 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049131590000000003 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.441378+0000", 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0076612609999999999 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.476105+0000", 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034727190999999998 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.502269+0000", 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026163842 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:31.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.554708+0000", 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052439432000000001 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.557604+0000", 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002895342 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:50:33.565692+0000", 2024-09-13T14:58:31.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080886559999999996 2024-09-13T14:58:31.588 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.588 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.588 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.588 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:58:31.588 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:58:31.588 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:58:31.588 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-13T14:58:31.589 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:31 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2831489670' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-13T14:58:31.589 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:31 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2831489670' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-13T14:58:31.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:31.669 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:31.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:31.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:31.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:31.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:31.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:31.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:31 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2831489670' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-13T14:58:32.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:32.043 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:32.137 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:58:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:58:31] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-13T14:58:32.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:32.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:32.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:32.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:32.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:32.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:32.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:32.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:32.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:32.663 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:32.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:32 smithi172 ceph-mon[100696]: pgmap v939: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 5930/68922 objects degraded (8.604%); 5785/68922 objects misplaced (8.394%); 10 MiB/s, 162 objects/s recovering 2024-09-13T14:58:32.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2831489670' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-13T14:58:32.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:32 smithi172 ceph-mon[100696]: osdmap e297: 8 total, 7 up, 8 in 2024-09-13T14:58:32.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:32.790 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:32.854 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:32 smithi154 ceph-mon[102314]: pgmap v939: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 5930/68922 objects degraded (8.604%); 5785/68922 objects misplaced (8.394%); 10 MiB/s, 162 objects/s recovering 2024-09-13T14:58:32.854 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2831489670' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-13T14:58:32.854 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:32 smithi154 ceph-mon[102314]: osdmap e297: 8 total, 7 up, 8 in 2024-09-13T14:58:32.854 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:32 smithi154 ceph-mon[107038]: pgmap v939: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 5930/68922 objects degraded (8.604%); 5785/68922 objects misplaced (8.394%); 10 MiB/s, 162 objects/s recovering 2024-09-13T14:58:32.854 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2831489670' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-13T14:58:32.854 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:32 smithi154 ceph-mon[107038]: osdmap e297: 8 total, 7 up, 8 in 2024-09-13T14:58:32.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:32.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:32.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:33.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:33.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:33.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.164 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:33.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:33.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:33.287 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:33.324 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:33.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:33.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:33.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:33.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:33.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:33.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:33.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:33.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:33.859 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:33 smithi154 ceph-mon[102314]: osdmap e298: 8 total, 7 up, 8 in 2024-09-13T14:58:33.859 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:33 smithi154 ceph-mon[107038]: osdmap e298: 8 total, 7 up, 8 in 2024-09-13T14:58:33.882 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:33 smithi172 ceph-mon[100696]: osdmap e298: 8 total, 7 up, 8 in 2024-09-13T14:58:33.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:33.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:33.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:34.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:34.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:34.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:34.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:34.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:34.310 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:34.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:34.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:34.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:34.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:34.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:34.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:34 smithi154 ceph-mon[102314]: pgmap v942: 89 pgs: 1 active+remapped+backfilling, 2 peering, 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 5930/65982 objects degraded (8.987%); 5136/65982 objects misplaced (7.784%); 7.9 MiB/s, 124 objects/s recovering 2024-09-13T14:58:34.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:34 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:34.662 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:34 smithi154 ceph-mon[102314]: osdmap e299: 8 total, 7 up, 8 in 2024-09-13T14:58:34.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:34 smithi154 ceph-mon[107038]: pgmap v942: 89 pgs: 1 active+remapped+backfilling, 2 peering, 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 5930/65982 objects degraded (8.987%); 5136/65982 objects misplaced (7.784%); 7.9 MiB/s, 124 objects/s recovering 2024-09-13T14:58:34.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:34 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:34.662 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:34 smithi154 ceph-mon[107038]: osdmap e299: 8 total, 7 up, 8 in 2024-09-13T14:58:34.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:34 smithi172 ceph-mon[100696]: pgmap v942: 89 pgs: 1 active+remapped+backfilling, 2 peering, 1 active+undersized+degraded+remapped+backfilling, 6 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 5930/65982 objects degraded (8.987%); 5136/65982 objects misplaced (7.784%); 7.9 MiB/s, 124 objects/s recovering 2024-09-13T14:58:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:34 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T14:58:34.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:34 smithi172 ceph-mon[100696]: osdmap e299: 8 total, 7 up, 8 in 2024-09-13T14:58:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:34.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:34.811 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:58:34.811 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T14:58:34.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:34.855 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:58:34.855 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:34.856 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:58:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:34.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:34.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:34.888 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:58:34.888 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:34.926 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:58:34.927 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:34.961 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:58:34.961 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:34.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:34.994 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:58:34.994 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:35.025 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:58:35.026 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:35.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:35.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:35.171 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:35.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:35.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:35.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:35.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:35.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:35.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:35.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.728 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:35.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:35.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:35.857 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:35.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:35.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:35.964 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3297608697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:35.964 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 5930/65982 objects degraded (8.987%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:35.964 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3297608697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:35.964 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 5930/65982 objects degraded (8.987%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:35.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3297608697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:35.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 5930/65982 objects degraded (8.987%), 7 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:36.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:36.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:36.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:36.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:36.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:36.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:36.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:36.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:36.545 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:36.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.675 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:36.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:36.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:36.803 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:36.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:36.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:36.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:36.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:36 smithi154 ceph-mon[102314]: pgmap v944: 89 pgs: 1 active+remapped+backfilling, 2 peering, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 4962/61917 objects degraded (8.014%); 4869/61917 objects misplaced (7.864%); 12 MiB/s, 189 objects/s recovering 2024-09-13T14:58:36.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:36.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:36 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:36.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:36 smithi154 ceph-mon[107038]: pgmap v944: 89 pgs: 1 active+remapped+backfilling, 2 peering, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 4962/61917 objects degraded (8.014%); 4869/61917 objects misplaced (7.864%); 12 MiB/s, 189 objects/s recovering 2024-09-13T14:58:36.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:36.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:36 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:37.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:37.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:37.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:37.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:36 smithi172 ceph-mon[100696]: pgmap v944: 89 pgs: 1 active+remapped+backfilling, 2 peering, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 4962/61917 objects degraded (8.014%); 4869/61917 objects misplaced (7.864%); 12 MiB/s, 189 objects/s recovering 2024-09-13T14:58:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:36 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:37.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:37.339 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:37.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:37.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:37.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:37.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:37.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:37.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:37.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:37.870 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:37.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:37.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:38.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:38.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.139 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:38.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:38.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:38.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:38.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:38.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:38.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:38.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.547 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:38.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:38.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:38.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:38.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:38.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:38.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:38.947 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:38.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:39.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:39.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:39.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:38 smithi154 ceph-mon[102314]: pgmap v945: 89 pgs: 1 active+remapped+backfilling, 2 peering, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 449 op/s; 4962/60828 objects degraded (8.157%); 4869/60828 objects misplaced (8.005%); 5.8 MiB/s, 88 objects/s recovering 2024-09-13T14:58:39.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:38 smithi154 ceph-mon[107038]: pgmap v945: 89 pgs: 1 active+remapped+backfilling, 2 peering, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 449 op/s; 4962/60828 objects degraded (8.157%); 4869/60828 objects misplaced (8.005%); 5.8 MiB/s, 88 objects/s recovering 2024-09-13T14:58:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:38 smithi172 ceph-mon[100696]: pgmap v945: 89 pgs: 1 active+remapped+backfilling, 2 peering, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 59 active+clean; 1.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 449 op/s; 4962/60828 objects degraded (8.157%); 4869/60828 objects misplaced (8.005%); 5.8 MiB/s, 88 objects/s recovering 2024-09-13T14:58:39.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:39.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:39.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:39.482 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:39.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:39.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:39.611 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:39.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:39.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:39.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:39.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:39.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:39.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.022 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:40.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:40.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:40.154 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:39 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:58:40.154 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:39 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:58:40.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:58:40.191 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:58:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:39 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T14:58:40.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:58:40.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:58:40.339 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-13T14:58:40.340 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:40.340 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 3, 4, 5, 6, 7] 2024-09-13T14:58:40.340 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T14:58:40.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0 2024-09-13T14:58:40.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:40.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:40.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:40.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:40.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:40.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:40.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:40.720 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:40 smithi154 systemd[1]: Stopping Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:58:40.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:40.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T14:58:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:40.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:40.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:40.998 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:40.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:41.082 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:41.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:41.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:41.188 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:40 smithi154 ceph-mon[102314]: pgmap v946: 89 pgs: 1 active+remapped+backfilling, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 4962/56019 objects degraded (8.858%); 2930/56019 objects misplaced (5.230%); 14 MiB/s, 227 objects/s recovering 2024-09-13T14:58:41.188 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4962/56019 objects degraded (8.858%), 6 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:41.188 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:40 smithi154 ceph-mon[107038]: pgmap v946: 89 pgs: 1 active+remapped+backfilling, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 4962/56019 objects degraded (8.858%); 2930/56019 objects misplaced (5.230%); 14 MiB/s, 227 objects/s recovering 2024-09-13T14:58:41.188 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4962/56019 objects degraded (8.858%), 6 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:41.206 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:40 smithi172 ceph-mon[100696]: pgmap v946: 89 pgs: 1 active+remapped+backfilling, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 4962/56019 objects degraded (8.858%); 2930/56019 objects misplaced (5.230%); 14 MiB/s, 227 objects/s recovering 2024-09-13T14:58:41.207 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4962/56019 objects degraded (8.858%), 6 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:41.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:41.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:41.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:41.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:41.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:41.499 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:41.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:41.624 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:41.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.749 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:41.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:41.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.872 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:41.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:41.874 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[118392]: 2024-09-13T14:58:41.477+0000 7fc42f782640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T14:58:41.874 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[118392]: 2024-09-13T14:58:41.477+0000 7fc42f782640 -1 osd.0 300 *** Got signal Terminated *** 2024-09-13T14:58:41.874 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[118392]: 2024-09-13T14:58:41.477+0000 7fc42f782640 -1 osd.0 300 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T14:58:41.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:41.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:41.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:42.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:42.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:42.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-mon[102314]: osdmap e300: 8 total, 7 up, 8 in 2024-09-13T14:58:42.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-mon[102314]: osd.0 marked itself down and dead 2024-09-13T14:58:42.207 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-mon[107038]: osdmap e300: 8 total, 7 up, 8 in 2024-09-13T14:58:42.207 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-mon[107038]: osd.0 marked itself down and dead 2024-09-13T14:58:42.207 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:58:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:58:41] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-13T14:58:42.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:41 smithi172 ceph-mon[100696]: osdmap e300: 8 total, 7 up, 8 in 2024-09-13T14:58:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:41 smithi172 ceph-mon[100696]: osd.0 marked itself down and dead 2024-09-13T14:58:42.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:42.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:42.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:42.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:42.488 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:42.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:42.612 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:42.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:42.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:42.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:42.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:42.894 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:58:42.903 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:42 smithi172 ceph-mon[100696]: pgmap v948: 89 pgs: 1 active+remapped+backfilling, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1020 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 3881/48828 objects degraded (7.948%); 2930/48828 objects misplaced (6.001%); 12 MiB/s, 186 objects/s recovering 2024-09-13T14:58:42.903 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:42 smithi172 ceph-mon[100696]: Health check update: 2 osds down (OSD_DOWN) 2024-09-13T14:58:42.903 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:42 smithi172 ceph-mon[100696]: osdmap e301: 8 total, 6 up, 8 in 2024-09-13T14:58:42.903 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1853593228' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:42.903 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:42 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:42.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:42.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:42.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:43.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:43.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[102314]: pgmap v948: 89 pgs: 1 active+remapped+backfilling, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1020 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 3881/48828 objects degraded (7.948%); 2930/48828 objects misplaced (6.001%); 12 MiB/s, 186 objects/s recovering 2024-09-13T14:58:43.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[102314]: Health check update: 2 osds down (OSD_DOWN) 2024-09-13T14:58:43.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[102314]: osdmap e301: 8 total, 6 up, 8 in 2024-09-13T14:58:43.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1853593228' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:43.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:43.182 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:42 smithi154 podman[247709]: 2024-09-13 14:58:42.855374351 +0000 UTC m=+1.742539055 container died 870726eb983e56dcecaaad94b84d13c5b8f07556c2bcf309b0595ae22567fca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-13T14:58:43.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[107038]: pgmap v948: 89 pgs: 1 active+remapped+backfilling, 7 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 6 active+undersized+degraded, 61 active+clean; 1020 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 3881/48828 objects degraded (7.948%); 2930/48828 objects misplaced (6.001%); 12 MiB/s, 186 objects/s recovering 2024-09-13T14:58:43.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[107038]: Health check update: 2 osds down (OSD_DOWN) 2024-09-13T14:58:43.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[107038]: osdmap e301: 8 total, 6 up, 8 in 2024-09-13T14:58:43.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1853593228' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:43.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:42 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:43.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:43.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:43.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:43.470 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:43 smithi154 podman[247709]: 2024-09-13 14:58:43.417751533 +0000 UTC m=+2.304916228 container cleanup 870726eb983e56dcecaaad94b84d13c5b8f07556c2bcf309b0595ae22567fca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:58:43.470 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:43 smithi154 bash[247709]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0 2024-09-13T14:58:43.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:43.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:43.606 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:43.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:43.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:43.807 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:43.808 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:43.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:43.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:43.908 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:43 smithi154 podman[247988]: 2024-09-13 14:58:43.610762661 +0000 UTC m=+1.078508257 container remove 870726eb983e56dcecaaad94b84d13c5b8f07556c2bcf309b0595ae22567fca7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:58:43.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:43.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:43.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:44.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:44.123 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:44.196 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:43 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:44.196 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:43 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:44.196 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:43 smithi172 ceph-mon[100696]: osdmap e302: 8 total, 6 up, 8 in 2024-09-13T14:58:44.196 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:44.196 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:43 smithi172 ceph-mon[100696]: osdmap e303: 8 total, 6 up, 8 in 2024-09-13T14:58:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[102314]: osdmap e302: 8 total, 6 up, 8 in 2024-09-13T14:58:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[102314]: osdmap e303: 8 total, 6 up, 8 in 2024-09-13T14:58:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[107038]: osdmap e302: 8 total, 6 up, 8 in 2024-09-13T14:58:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:43 smithi154 ceph-mon[107038]: osdmap e303: 8 total, 6 up, 8 in 2024-09-13T14:58:44.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.248 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:44.372 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:44.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:44.475 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:44.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:44.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:44.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T14:58:44.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:44.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:44.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:44.842 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:44.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:44.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:44.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:45.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:45.163 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:44 smithi154 ceph-mon[102314]: pgmap v951: 89 pgs: 1 stale+active+remapped+backfilling, 1 stale+active+remapped+backfill_wait, 1 stale+active+undersized, 1 stale+active+undersized+degraded, 14 stale+active+clean, 7 active+clean+remapped, 12 active+undersized, 5 active+undersized+degraded, 47 active+clean; 1020 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 3881/48828 objects degraded (7.948%); 2930/48828 objects misplaced (6.001%); 12 MiB/s, 196 objects/s recovering 2024-09-13T14:58:45.163 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:44 smithi154 ceph-mon[107038]: pgmap v951: 89 pgs: 1 stale+active+remapped+backfilling, 1 stale+active+remapped+backfill_wait, 1 stale+active+undersized, 1 stale+active+undersized+degraded, 14 stale+active+clean, 7 active+clean+remapped, 12 active+undersized, 5 active+undersized+degraded, 47 active+clean; 1020 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 3881/48828 objects degraded (7.948%); 2930/48828 objects misplaced (6.001%); 12 MiB/s, 196 objects/s recovering 2024-09-13T14:58:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:44 smithi172 ceph-mon[100696]: pgmap v951: 89 pgs: 1 stale+active+remapped+backfilling, 1 stale+active+remapped+backfill_wait, 1 stale+active+undersized, 1 stale+active+undersized+degraded, 14 stale+active+clean, 7 active+clean+remapped, 12 active+undersized, 5 active+undersized+degraded, 47 active+clean; 1020 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 3881/48828 objects degraded (7.948%); 2930/48828 objects misplaced (6.001%); 12 MiB/s, 196 objects/s recovering 2024-09-13T14:58:45.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:45.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:45.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:45.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:45.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:45.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:45.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:45.780 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:45.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:45.903 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:45.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:46.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:46.029 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:46.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:46.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:46.166 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:45 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3881/48828 objects degraded (7.948%), 6 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:46.166 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:45 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3881/48828 objects degraded (7.948%), 6 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:46.177 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:45 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3881/48828 objects degraded (7.948%), 6 pgs degraded, 19 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:46.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:46.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:46.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:46.388 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:46.388 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:46.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:46.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.526 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:46.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:46.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:46.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:46.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:46.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:46.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:46.895 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:46.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:46.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:47.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:47.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:58:47.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:47.143 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:47.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T14:58:47.213 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:46 smithi172 ceph-mon[100696]: pgmap v953: 89 pgs: 2 active+undersized+remapped, 1 stale+active+undersized, 3 stale+active+clean, 7 active+clean+remapped, 1 undersized+peered, 26 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 37 active+clean; 908 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 423 op/s; 6237/43443 objects degraded (14.357%); 4264/43443 objects misplaced (9.815%); 12 MiB/s, 184 objects/s recovering 2024-09-13T14:58:47.213 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:46 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:58:47.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:46 smithi154 ceph-mon[102314]: pgmap v953: 89 pgs: 2 active+undersized+remapped, 1 stale+active+undersized, 3 stale+active+clean, 7 active+clean+remapped, 1 undersized+peered, 26 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 37 active+clean; 908 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 423 op/s; 6237/43443 objects degraded (14.357%); 4264/43443 objects misplaced (9.815%); 12 MiB/s, 184 objects/s recovering 2024-09-13T14:58:47.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:46 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:58:47.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:46 smithi154 ceph-mon[107038]: pgmap v953: 89 pgs: 2 active+undersized+remapped, 1 stale+active+undersized, 3 stale+active+clean, 7 active+clean+remapped, 1 undersized+peered, 26 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 37 active+clean; 908 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 423 op/s; 6237/43443 objects degraded (14.357%); 4264/43443 objects misplaced (9.815%); 12 MiB/s, 184 objects/s recovering 2024-09-13T14:58:47.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:46 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-13T14:58:47.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:47.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:47.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.391 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:47.391 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.477 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:58:47.477 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T14:58:47.477 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:47.478 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:58:47.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.507 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:58:47.508 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:47.513 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.521 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:47.538 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:58:47.538 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:47.559 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:58:47.560 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:47.577 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:58:47.577 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:47.593 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:58:47.593 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:47.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:47.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:47.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:47.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:47.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:47.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:47.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:47.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:48.008 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:48.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:48.132 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:48.170 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.170 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.170 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:48.170 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:48.170 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.170 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/548976659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:48.170 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.171 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.171 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:48.171 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:48.171 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.171 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:47 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/548976659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:48.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:47 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:47 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:47 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T14:58:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:47 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T14:58:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:47 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:47 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/548976659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.257 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:48.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:48.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:48.379 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:48.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:48.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:48.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:48.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:48.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:48.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:48.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:48.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:48.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:48.944 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:48 smithi154 podman[248812]: 2024-09-13 14:58:48.502643652 +0000 UTC m=+0.950388211 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:58:48.945 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:48 smithi154 podman[248812]: 2024-09-13 14:58:48.600021784 +0000 UTC m=+1.047766332 container create ab8670863393f2c008e5da9d21808290e6fb47e3550e6257c824c838e99aee28 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T14:58:48.945 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:48 smithi154 podman[248812]: 2024-09-13 14:58:48.844335603 +0000 UTC m=+1.292080152 container init ab8670863393f2c008e5da9d21808290e6fb47e3550e6257c824c838e99aee28 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:58:48.945 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:48 smithi154 podman[248812]: 2024-09-13 14:58:48.847884957 +0000 UTC m=+1.295629513 container start ab8670863393f2c008e5da9d21808290e6fb47e3550e6257c824c838e99aee28 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:58:48.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:49.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:49.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.130 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.156 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:48 smithi172 ceph-mon[100696]: pgmap v954: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 886 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 357 op/s; 8024/42396 objects degraded (18.926%); 4264/42396 objects misplaced (10.058%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T14:58:49.156 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:48 smithi172 ceph-mon[100696]: pgmap v955: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 886 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 389 op/s; 8024/42396 objects degraded (18.926%); 4264/42396 objects misplaced (10.058%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T14:58:49.157 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:48 smithi172 ceph-mon[100696]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-13T14:58:49.176 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:58:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:58:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:58:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:58:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86743 6.e 6:7360c465:::benchmark_data_smithi154_175640_object967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:56.640282+0000", 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 532.53253377800002, 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89457589000000004, 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86743 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640282+0000", 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640282+0000", 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640280+0000", 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640288+0000", 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6480000000000003e-06 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640305+0000", 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6552e-05 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640311+0000", 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8510000000000001e-06 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640368+0000", 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6291e-05 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.640375+0000", 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8040000000000002e-06 2024-09-13T14:58:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.520758+0000", 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.88038205700000005 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.520867+0000", 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010984000000000001 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,5", 2024-09-13T14:58:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.520948+0000", 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0227000000000003e-05 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529430+0000", 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084824119999999999 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529509+0000", 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9183000000000004e-05 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534788+0000", 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052784499999999996 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534831+0000", 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3852e-05 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534858+0000", 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6291999999999998e-05 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86745 6.e 6:7d92d165:::benchmark_data_smithi154_175640_object969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:56.661715+0000", 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 532.51110022800003, 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.87325762500000004, 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86745 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661715+0000", 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661715+0000", 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661713+0000", 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661724+0000", 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0847e-05 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661745+0000", 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0888000000000001e-05 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661755+0000", 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5030000000000003e-06 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661810+0000", 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5704000000000001e-05 2024-09-13T14:58:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:56.661820+0000", 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4339999999999998e-06 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.523654+0000", 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86183458800000001 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.523785+0000", 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013034099999999999 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,5", 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.524120+0000", 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033531499999999998 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529542+0000", 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054218490000000003 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.529572+0000", 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0568000000000003e-05 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534919+0000", 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053466360000000001 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534955+0000", 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5871000000000003e-05 2024-09-13T14:58:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.534973+0000", 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8207999999999999e-05 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86780 6.f 6:f38bec78:::benchmark_data_smithi154_175640_object1004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.535104+0000", 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 531.637711169, 2024-09-13T14:58:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.269125445, 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86780 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535104+0000", 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535104+0000", 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535103+0000", 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535110+0000", 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7750000000000004e-06 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535127+0000", 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7268000000000001e-05 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.535137+0000", 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3610000000000007e-06 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547418+0000", 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012281337999999999 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547433+0000", 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4694e-05 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.781430+0000", 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2339976349999999 2024-09-13T14:58:49.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.782221+0000", 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079034699999999995 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.790514+0000", 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082936369999999995 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.790589+0000", 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4864999999999996e-05 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.790690+0000", 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101266 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800544+0000", 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098536419999999993 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800588+0000", 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4236000000000001e-05 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800624+0000", 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5914000000000002e-05 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.800689+0000", 2024-09-13T14:58:49.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4478000000000002e-05 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804230+0000", 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003541317 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86786 6.f 6:f6949069:::benchmark_data_smithi154_175640_object1010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.542654+0000", 2024-09-13T14:58:49.199 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 531.63016120500004, 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.265779684, 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86786 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542654+0000", 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542654+0000", 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542653+0000", 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542662+0000", 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0550000000000005e-06 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542682+0000", 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0769999999999999e-05 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.542691+0000", 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8799999999999997e-06 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547451+0000", 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047592829999999996 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.547455+0000", 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3610000000000003e-06 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.784425+0000", 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2369699439999999 2024-09-13T14:58:49.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.785202+0000", 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00077720700000000003 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.798177+0000", 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012974741999999999 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.798247+0000", 2024-09-13T14:58:49.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0505000000000003e-05 2024-09-13T14:58:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:58:49.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.798343+0000", 2024-09-13T14:58:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5259000000000004e-05 2024-09-13T14:58:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804963+0000", 2024-09-13T14:58:49.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066199359999999999 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804974+0000", 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1936999999999999e-05 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.805006+0000", 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2030999999999999e-05 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.805045+0000", 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8080999999999997e-05 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.808434+0000", 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033895520000000001 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86796 6.f 6:f0712b29:::benchmark_data_smithi154_175640_object1020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:58:49.208 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.589684+0000", 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 531.58313211500001, 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.262880338, 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86796 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589684+0000", 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589684+0000", 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589682+0000", 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589689+0000", 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6080000000000001e-06 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589699+0000", 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.066e-05 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589705+0000", 2024-09-13T14:58:49.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7490000000000004e-06 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589749+0000", 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4161e-05 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.589757+0000", 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9030000000000004e-06 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804277+0000", 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.214520332 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.804917+0000", 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00063914099999999997 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.820581+0000", 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015664843000000001 2024-09-13T14:58:49.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.820661+0000", 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0003e-05 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.820759+0000", 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7700000000000003e-05 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835612+0000", 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014852868999999999 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848754+0000", 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013141664000000001 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848813+0000", 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8989000000000001e-05 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848868+0000", 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852564+0000", 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036962660000000001 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.217 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86800 6.f 6:f5a03e3c:::benchmark_data_smithi154_175640_object1024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.668107+0000", 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 531.50470842599998, 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.188158123, 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.218 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86800 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668107+0000", 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668107+0000", 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668106+0000", 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668111+0000", 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3040000000000004e-06 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668121+0000", 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9229999999999997e-06 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668127+0000", 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9379999999999997e-06 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668170+0000", 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3164000000000002e-05 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.668185+0000", 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4419e-05 2024-09-13T14:58:49.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.808500+0000", 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1403150879999999 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.809148+0000", 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064803699999999996 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835647+0000", 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026499338000000001 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835732+0000", 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.509e-05 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.835828+0000", 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6115999999999999e-05 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.848685+0000", 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012856572 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852637+0000", 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039521770000000003 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852744+0000", 2024-09-13T14:58:49.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106524 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.852809+0000", 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4993999999999995e-05 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856265+0000", 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034566649999999998 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:86801 6.f 6:fed2b0bf:::benchmark_data_smithi154_175640_object1025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:49:57.746781+0000", 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 531.42603453300001, 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.112958592, 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 86801 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.226 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746781+0000", 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746781+0000", 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746780+0000", 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746787+0000", 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3760000000000002e-06 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746801+0000", 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4178999999999999e-05 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746807+0000", 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0020000000000003e-06 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746869+0000", 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1486000000000001e-05 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:57.746882+0000", 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3108999999999999e-05 2024-09-13T14:58:49.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.812349+0000", 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0654668549999999 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.813209+0000", 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086008199999999997 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.845548+0000", 2024-09-13T14:58:49.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032339097999999997 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.845678+0000", 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129651 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,6", 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.845862+0000", 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000184444 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856280+0000", 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010418487000000001 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856339+0000", 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8145e-05 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856384+0000", 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5139999999999998e-05 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.856422+0000", 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7910000000000001e-05 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:49:58.859740+0000", 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033180459999999998 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.233 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110738 6.e 6:780e6055:::benchmark_data_smithi154_175640_object24962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.142686+0000", 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 392.030129628, 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.041401764, 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110738 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142686+0000", 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142686+0000", 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142685+0000", 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142690+0000", 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000003e-06 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142700+0000", 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0345e-05 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142707+0000", 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9650000000000002e-06 2024-09-13T14:58:49.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142751+0000", 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4088999999999997e-05 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142759+0000", 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.294e-06 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.172515+0000", 2024-09-13T14:58:49.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.029756407 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.172632+0000", 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117419 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.173099+0000", 2024-09-13T14:58:49.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046696500000000001 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179720+0000", 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066204250000000001 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179777+0000", 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7538999999999999e-05 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.183999+0000", 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042218719999999998 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.184046+0000", 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7150000000000001e-05 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.184088+0000", 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1390000000000002e-05 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.240 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110739 6.e 6:78427618:::benchmark_data_smithi154_175640_object24963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 392.02608438099998, 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0540792329999999, 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.241 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110739 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146734+0000", 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8319999999999999e-06 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146743+0000", 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4460000000000001e-06 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146749+0000", 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3790000000000003e-06 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146788+0000", 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9172999999999999e-05 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146795+0000", 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3289999999999999e-06 2024-09-13T14:58:49.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.175941+0000", 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0291466950000001 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.176007+0000", 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5673999999999995e-05 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.176071+0000", 2024-09-13T14:58:49.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4245999999999994e-05 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.181605+0000", 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055336969999999997 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.194960+0000", 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013354428 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200751+0000", 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057913399999999999 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200782+0000", 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1278000000000001e-05 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200811+0000", 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8384000000000001e-05 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110766 6.e 6:7c2939fb:::benchmark_data_smithi154_175640_object24990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.287081+0000", 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 391.88573497800002, 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92337370100000005, 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110766 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287081+0000", 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287081+0000", 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287078+0000", 2024-09-13T14:58:49.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287089+0000", 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0464e-05 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287110+0000", 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1403000000000002e-05 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287119+0000", 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2359999999999993e-06 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287185+0000", 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5074999999999996e-05 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287193+0000", 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7730000000000005e-06 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.177878+0000", 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89068479700000003 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.177987+0000", 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108955 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.178102+0000", 2024-09-13T14:58:49.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114571 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.186553+0000", 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084510200000000001 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200852+0000", 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014299335999999999 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210402+0000", 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095498179999999998 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210431+0000", 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9479999999999999e-05 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210454+0000", 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3074e-05 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.254 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110767 6.e 6:7ea46229:::benchmark_data_smithi154_175640_object24991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.297777+0000", 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 391.87503895999998, 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91276756699999995, 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110767 2024-09-13T14:58:49.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297777+0000", 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297777+0000", 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297774+0000", 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297785+0000", 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0249e-05 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297802+0000", 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7652999999999999e-05 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297810+0000", 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9200000000000004e-06 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297866+0000", 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5358000000000001e-05 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297874+0000", 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3610000000000001e-06 2024-09-13T14:58:49.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179816+0000", 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.88194239600000002 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179923+0000", 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106557 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.180036+0000", 2024-09-13T14:58:49.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113286 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.191600+0000", 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011563982 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200934+0000", 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093336400000000007 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210492+0000", 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009557879 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210522+0000", 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0074e-05 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210544+0000", 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2561e-05 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40265 7.1 7:8a6771d1:::benchmark_data_smithi154_225924_object40264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.707202+0000", 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 202.46561409899999, 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.19042437100000001, 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40265 2024-09-13T14:58:49.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707202+0000", 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707202+0000", 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707201+0000", 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707233+0000", 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2341000000000001e-05 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707243+0000", 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2119999999999992e-06 2024-09-13T14:58:49.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707248+0000", 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7559999999999996e-06 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708726+0000", 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014780259999999999 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708933+0000", 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000206381 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.709077+0000", 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014402600000000001 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712231+0000", 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031540700000000001 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713576+0000", 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013451660000000001 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897560+0000", 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18398365 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897600+0000", 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0713999999999997e-05 2024-09-13T14:58:49.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897626+0000", 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5602999999999999e-05 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40266 7.1 7:810b8261:::benchmark_data_smithi154_225924_object40265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 202.46556045400001, 2024-09-13T14:58:49.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.190473265, 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40266 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707255+0000", 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707290+0000", 2024-09-13T14:58:49.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5580999999999997e-05 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707301+0000", 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0767e-05 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.707304+0000", 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5310000000000002e-06 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710646+0000", 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033420049999999999 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710809+0000", 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162168 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710955+0000", 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014668400000000001 2024-09-13T14:58:49.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713638+0000", 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026823720000000001 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.714885+0000", 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001247623 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897664+0000", 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18277848399999999 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897707+0000", 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3087999999999998e-05 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897728+0000", 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1569999999999998e-05 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40271 7.1 7:8fdb6a0e:::benchmark_data_smithi154_225924_object40270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.708865+0000", 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 202.463951071, 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.188944634, 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40271 2024-09-13T14:58:49.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708865+0000", 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708865+0000", 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708863+0000", 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708962+0000", 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8227999999999995e-05 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708974+0000", 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2170999999999999e-05 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.708981+0000", 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3599999999999998e-06 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712315+0000", 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033341109999999998 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712442+0000", 2024-09-13T14:58:49.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012720099999999999 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.712587+0000", 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014424199999999999 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.714890+0000", 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002303027 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.714932+0000", 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2055000000000003e-05 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897762+0000", 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18283049900000001 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897790+0000", 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7555e-05 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.897809+0000", 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9446999999999999e-05 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40273 7.1 7:8f2b799d:::benchmark_data_smithi154_225924_object40272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.710113+0000", 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 202.462702939, 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18851839400000001, 2024-09-13T14:58:49.281 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40273 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710113+0000", 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710113+0000", 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710111+0000", 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710163+0000", 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1301000000000002e-05 2024-09-13T14:58:49.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710177+0000", 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3839000000000001e-05 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.710184+0000", 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2880000000000004e-06 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713700+0000", 2024-09-13T14:58:49.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035157890000000001 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713804+0000", 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104338 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.713931+0000", 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012671700000000001 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.715112+0000", 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011810830000000001 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.715121+0000", 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8300000000000002e-06 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898553+0000", 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18343242800000001 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898597+0000", 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4369999999999997e-05 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898631+0000", 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3698000000000002e-05 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:40289 7.1 7:8e157fd9:::benchmark_data_smithi154_225924_object40288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:55:26.719678+0000", 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 202.45313730999999, 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.179173417, 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 40289 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719678+0000", 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719678+0000", 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719677+0000", 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719727+0000", 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0504999999999998e-05 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719738+0000", 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0623e-05 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719743+0000", 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1170000000000003e-06 2024-09-13T14:58:49.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719778+0000", 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5201999999999999e-05 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719881+0000", 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102693 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.719978+0000", 2024-09-13T14:58:49.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7448000000000006e-05 2024-09-13T14:58:49.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.293 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:48 smithi154 ceph-mon[102314]: pgmap v954: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 886 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 357 op/s; 8024/42396 objects degraded (18.926%); 4264/42396 objects misplaced (10.058%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T14:58:49.293 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:48 smithi154 ceph-mon[102314]: pgmap v955: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 886 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 389 op/s; 8024/42396 objects degraded (18.926%); 4264/42396 objects misplaced (10.058%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T14:58:49.293 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:48 smithi154 ceph-mon[102314]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-13T14:58:49.293 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:48 smithi154 ceph-mon[107038]: pgmap v954: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 886 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 357 op/s; 8024/42396 objects degraded (18.926%); 4264/42396 objects misplaced (10.058%); 8.2 MiB/s, 130 objects/s recovering 2024-09-13T14:58:49.294 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:48 smithi154 ceph-mon[107038]: pgmap v955: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 886 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 389 op/s; 8024/42396 objects degraded (18.926%); 4264/42396 objects misplaced (10.058%); 8.9 MiB/s, 142 objects/s recovering 2024-09-13T14:58:49.294 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:48 smithi154 ceph-mon[107038]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.721474+0000", 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014957880000000001 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.722455+0000", 2024-09-13T14:58:49.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00098106600000000006 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898774+0000", 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.17631929199999999 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898811+0000", 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6491e-05 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:55:26.898852+0000", 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0799e-05 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150521 7.8 7:11a94199:::benchmark_data_smithi154_225924_object40257:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.290904+0000", 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.881912076000006, 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.023286141, 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:58:49.296 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150521 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290904+0000", 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290904+0000", 2024-09-13T14:58:49.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290902+0000", 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290908+0000", 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9279999999999998e-06 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290919+0000", 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.132e-05 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290925+0000", 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2220000000000001e-06 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290978+0000", 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3409000000000003e-05 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290986+0000", 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3659999999999992e-06 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310964+0000", 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0199773940000001 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314190+0000", 2024-09-13T14:58:49.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032259120000000001 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150526 7.8 7:16bba6af:::benchmark_data_smithi154_225924_object40262:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.290939+0000", 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 77.881877000000003, 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0235627060000001, 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.301 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150526 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290939+0000", 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290939+0000", 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290938+0000", 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290940+0000", 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0020000000000001e-06 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290945+0000", 2024-09-13T14:58:49.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7840000000000003e-06 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290949+0000", 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.518e-06 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.291003+0000", 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4715e-05 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:58:49.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.291009+0000", 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1359999999999996e-06 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314426+0000", 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0234177849999999 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314501+0000", 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5047000000000002e-05 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:178062 7.1 7:8f0097a6:::benchmark_data_smithi154_225924_object67798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:04.145703+0000", 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.027112692000003, 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.18851287, 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 178062 2024-09-13T14:58:49.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145703+0000", 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145703+0000", 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145701+0000", 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145709+0000", 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5700000000000004e-06 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145724+0000", 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5545000000000001e-05 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145732+0000", 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1869999999999997e-06 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145798+0000", 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5766999999999996e-05 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.145920+0000", 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121407 2024-09-13T14:58:49.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,6", 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.146033+0000", 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113346 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.150630+0000", 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045968529999999997 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.155058+0000", 2024-09-13T14:58:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044284700000000003 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.334107+0000", 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.179048243 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.334171+0000", 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4227999999999996e-05 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.334216+0000", 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5046999999999997e-05 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:178085 7.3 e290/281 7:c63a889a:::benchmark_data_smithi154_225924_object67821:head v 290'11180, mlcod=290'11178)", 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:04.172806+0000", 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.000010148999998, 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.14275435, 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:58:49.312 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172806+0000", 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172806+0000", 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172805+0000", 2024-09-13T14:58:49.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172812+0000", 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.182e-06 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172825+0000", 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2843999999999999e-05 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172832+0000", 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3649999999999998e-06 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172877+0000", 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5203000000000001e-05 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.172898+0000", 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0398999999999999e-05 2024-09-13T14:58:49.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.315527+0000", 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.14262886299999999 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:04.315560+0000", 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3348e-05 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:58:49.316 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:58:49.317 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:58:49.317 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:49 smithi154 podman[248812]: 2024-09-13 14:58:49.044991912 +0000 UTC m=+1.492736505 container attach ab8670863393f2c008e5da9d21808290e6fb47e3550e6257c824c838e99aee28 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T14:58:49.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:49.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:49.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:49.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:58:49.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:58:49.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:49.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:49.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:49.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:49.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:49.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:49.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:50.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:50.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:50.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:50.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:49 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8024/42396 objects degraded (18.926%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:50.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T14:58:50.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:50.331 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:49 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8024/42396 objects degraded (18.926%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:50.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:49 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8024/42396 objects degraded (18.926%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.357 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:50.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:50.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.531 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:50.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.606 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:50.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:50.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:50.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:50.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:50.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:50.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:50.970 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:58:50 smithi154 podman[248812]: 2024-09-13 14:58:50.843753159 +0000 UTC m=+3.291497709 container died ab8670863393f2c008e5da9d21808290e6fb47e3550e6257c824c838e99aee28 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-13T14:58:50.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:50.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:50.981 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:58:51.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:51.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:51.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:51.228 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:51.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:50 smithi172 ceph-mon[100696]: pgmap v956: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 334 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:58:51.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1021996905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:51.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:50 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:51.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:51.353 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:51.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:50 smithi154 ceph-mon[102314]: pgmap v956: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 334 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:58:51.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1021996905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:51.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:50 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:51.365 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:50 smithi154 ceph-mon[107038]: pgmap v956: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 334 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%); 7.5 MiB/s, 120 objects/s recovering 2024-09-13T14:58:51.365 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1021996905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:51.366 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:50 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:58:51.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:51.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:51.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:58:51.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:51.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:51.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:51.850 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:51.926 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:58:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:58:51] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-13T14:58:51.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:51.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:51.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:52.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:52.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.175 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:58:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:58:52.176 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T14:58:52.176 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:58:52.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:51 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:51 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:51 smithi154 ceph-mon[102314]: osdmap e304: 8 total, 6 up, 8 in 2024-09-13T14:58:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:51 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:51 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:51 smithi154 ceph-mon[107038]: osdmap e304: 8 total, 6 up, 8 in 2024-09-13T14:58:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:52.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:51 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:58:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:51 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:58:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:51 smithi172 ceph-mon[100696]: osdmap e304: 8 total, 6 up, 8 in 2024-09-13T14:58:52.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T14:58:52.289 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:52.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:52.328 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T14:58:52.329 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T14:58:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.345 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:52.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:52.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:52.468 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:52.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:52.591 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:52.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:52.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:52.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:52.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:52.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:52.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:52.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:52.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T14:58:53.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:53.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:53.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:53.213 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:53.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:52 smithi154 ceph-mon[102314]: pgmap v958: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 289 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:58:53.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:52 smithi154 ceph-mon[107038]: pgmap v958: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 289 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:58:53.229 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:53.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:52 smithi172 ceph-mon[100696]: pgmap v958: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 289 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%); 6.5 MiB/s, 104 objects/s recovering 2024-09-13T14:58:53.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:53.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:53.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:53.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:53.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:53.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:53.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:53.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:53.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:58:53.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:53.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:53.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:53.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:53.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:53.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:54.053 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:58:54.054 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T14:58:54.054 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:58:54.054 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:58:54.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:54.056 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:58:54.056 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:58:54.072 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:58:54.072 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:58:54.077 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:54.088 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:58:54.089 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:58:54.110 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:58:54.110 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:58:54.131 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:58:54.131 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:58:54.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:54.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:54.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:54.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:54.372 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:54.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:54.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:54.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:54.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:54.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:54.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:54.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:58:54.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:58:54.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:54.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:54.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:54.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:54.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:54.959 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:54.960 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:55.040 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[102314]: pgmap v959: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 46 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:55.040 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4086471999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:55.040 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[102314]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-13T14:58:55.040 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 8024/42300 objects degraded (18.969%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:55.041 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[107038]: pgmap v959: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 46 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:55.041 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4086471999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:55.041 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[107038]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-13T14:58:55.041 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 8024/42300 objects degraded (18.969%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:55.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:55.070 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:55.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:55.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.330 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:55 smithi172 ceph-mon[100696]: pgmap v959: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 46 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:55.330 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4086471999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:58:55.331 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:55 smithi172 ceph-mon[100696]: Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-09-13T14:58:55.331 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 8024/42300 objects degraded (18.969%), 16 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-09-13T14:58:55.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:55.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:55.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:55.567 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:55.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T14:58:55.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:55.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:55.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:55.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:55.865 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:55.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:55.940 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:55.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:56.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:56.065 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:56.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:56.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:56.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:56.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:56.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:56.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:56.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:56.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:56.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:56.683 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:56.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:56.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:58:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:56.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:56.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:56.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:56.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:57.018 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:57.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:57.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:57.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:57.181 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:57.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:57.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:57.313 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:57 smithi172 ceph-mon[100696]: pgmap v960: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:57.325 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:57 smithi154 ceph-mon[102314]: pgmap v960: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:57.325 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:57 smithi154 ceph-mon[107038]: pgmap v960: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:57.330 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 5, 7] out_osds: [] dead_osds: [2, 0] live_osds: [1, 3, 4, 5, 6, 7] 2024-09-13T14:58:57.330 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:58:57.330 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 6, 5, 7] 2024-09-13T14:58:57.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd out 6 2024-09-13T14:58:57.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:57.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:57.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:57.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:57.567 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:58:57.568 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:58:57.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:57.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.677 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:57.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:57.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:57.800 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:57.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:57.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:57.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:58.012 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:58:58.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.048 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:58.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:58.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:58:58.171 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:58:58.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/471303004' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:58.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/471303004' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:58.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:58.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:58.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:58.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:58.305 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/471303004' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:58.305 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:58:58.305 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:58:58.305 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:58:58.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T14:58:58.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:58.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:58.543 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:58.560 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:58:58.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:58.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:58.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:58.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:58.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:58.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:58.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:59.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:59.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:59.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.164 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:59.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:59.198 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:58:59.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:58:59.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:58:59.300 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[102314]: pgmap v961: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:59.300 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:59.300 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:59.300 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[102314]: osdmap e305: 8 total, 6 up, 8 in 2024-09-13T14:58:59.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[107038]: pgmap v961: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:59.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:59.300 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:59.301 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:58:59 smithi154 ceph-mon[107038]: osdmap e305: 8 total, 6 up, 8 in 2024-09-13T14:58:59.371 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:59 smithi172 ceph-mon[100696]: pgmap v961: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 3 op/s; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:58:59.371 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:59 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:58:59.371 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:59 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:58:59.371 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:58:59 smithi172 ceph-mon[100696]: osdmap e305: 8 total, 6 up, 8 in 2024-09-13T14:58:59.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:58:59.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:58:59.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:59.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:58:59.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:58:59.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:58:59.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:58:59.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:58:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:58:59.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:58:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:58:59.908 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:58:59.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:00.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:00.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:00.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:00.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:00.207 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:59:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:59:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T14:59:00.208 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:59:00.208 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/6216442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:00.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/6216442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:00.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/6216442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:00.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:00.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:00.341 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T14:59:00.341 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:00.341 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:59:00.342 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:59:00.342 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:59:00.342 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:59:00.342 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:59:00.342 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:59:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:59:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:00.344 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:00.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:00.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:00.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:00.526 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:00.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:00.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:00.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:00.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:00.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:00.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T14:59:00.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:00.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:01.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.075 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:01.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:01.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:59:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:01.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:01.196 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:01 smithi172 ceph-mon[100696]: pgmap v963: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:01.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:01.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:01.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:01.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:01 smithi154 ceph-mon[107038]: pgmap v963: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:01.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:01 smithi154 ceph-mon[102314]: pgmap v963: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:01.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:59:01.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:59:01.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.574 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:01.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:01.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:01.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.766 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:01.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:01.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:01.890 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:01.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:01.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:02.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:02.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:02.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:02.137 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:02.155 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:02 smithi154 ceph-mon[102314]: pgmap v964: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:02.155 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:59:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:59:01] "GET /metrics HTTP/1.1" 200 36606 "" "Prometheus/2.51.0" 2024-09-13T14:59:02.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:02 smithi154 ceph-mon[107038]: pgmap v964: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:02.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:02.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:02 smithi172 ceph-mon[100696]: pgmap v964: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:02.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:02.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:02.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:02.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:02.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:02.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.632 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:02.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:02.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:02.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:02.808 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:59:02.808 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T14:59:02.809 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T14:59:02.809 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T14:59:02.809 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:59:02.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:02.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:02.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:02.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.005 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:03.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:03.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:03.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:03.205 INFO:teuthology.orchestra.run.smithi154.stderr:marked out osd.6. 2024-09-13T14:59:03.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:03.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:03.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:03.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:03.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1698975646' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:59:03.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:03 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:59:03.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:03 smithi154 ceph-mon[107038]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T14:59:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:03 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T14:59:03.142+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e305 definitely_dead 0 2024-09-13T14:59:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1698975646' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:59:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:03 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:59:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:03 smithi154 ceph-mon[102314]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T14:59:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1698975646' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:59:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:03 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-13T14:59:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:03 smithi172 ceph-mon[100696]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-13T14:59:03.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:03.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:03.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T14:59:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.576 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:03.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:03.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:03.699 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:03.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:03.771 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:03.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:03.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:03.924 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:03.946 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:03.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:04.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:04.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:04.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.197 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:04.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:04.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:04.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:04.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:04.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T14:59:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[107038]: osdmap e306: 8 total, 6 up, 7 in 2024-09-13T14:59:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[107038]: pgmap v966: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[107038]: osdmap e307: 8 total, 6 up, 7 in 2024-09-13T14:59:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T14:59:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[102314]: osdmap e306: 8 total, 6 up, 7 in 2024-09-13T14:59:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[102314]: pgmap v966: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:04 smithi154 ceph-mon[102314]: osdmap e307: 8 total, 6 up, 7 in 2024-09-13T14:59:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:04 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-13T14:59:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:04 smithi172 ceph-mon[100696]: osdmap e306: 8 total, 6 up, 7 in 2024-09-13T14:59:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:04 smithi172 ceph-mon[100696]: pgmap v966: 89 pgs: 2 active+undersized+remapped, 7 active+clean+remapped, 2 undersized+peered, 28 active+undersized, 3 undersized+degraded+peered, 13 active+undersized+degraded, 34 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 8024/42300 objects degraded (18.969%); 4218/42300 objects misplaced (9.972%) 2024-09-13T14:59:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:04 smithi172 ceph-mon[100696]: osdmap e307: 8 total, 6 up, 7 in 2024-09-13T14:59:04.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:04.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:04.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:04.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:04.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:04.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:04.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:04.949 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:04.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:05.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.075 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:05.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:05.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:05.199 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:05.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:05.322 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:05.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:05.437 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:05.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:05.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:05.488 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T14:59:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T14:59:05.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150525 7.c 7:3a1defd9:::benchmark_data_smithi154_225924_object40261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.271468+0000", 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.190260428000002, 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.038483329, 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.489 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150525 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271468+0000", 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271468+0000", 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271467+0000", 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271474+0000", 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.374e-06 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271489+0000", 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4942e-05 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271497+0000", 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4499999999999998e-06 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271547+0000", 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9669000000000002e-05 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271709+0000", 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016256800000000001 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271846+0000", 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013647600000000001 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.276339+0000", 2024-09-13T14:59:05.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044934969999999999 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.278783+0000", 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024441889999999998 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290520+0000", 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011737081 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290532+0000", 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1182999999999999e-05 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.309898+0000", 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.019366231 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.309952+0000", 2024-09-13T14:59:05.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4027999999999998e-05 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150538 7.c 7:32d10010:::benchmark_data_smithi154_225924_object40274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.311705+0000", 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.150024199000001, 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0082904450000001, 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.496 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150538 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311705+0000", 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311705+0000", 2024-09-13T14:59:05.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311703+0000", 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311710+0000", 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7000000000000002e-06 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311721+0000", 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1728e-05 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311727+0000", 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4070000000000001e-06 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311778+0000", 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1570999999999998e-05 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311789+0000", 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0328e-05 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310250+0000", 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99846129900000002 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310383+0000", 2024-09-13T14:59:05.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013281299999999999 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310500+0000", 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117446 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.318391+0000", 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078911080000000008 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319857+0000", 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014651569999999999 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319909+0000", 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2748999999999999e-05 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319995+0000", 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5818999999999999e-05 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198815 7.d 7:bb1741ae:::benchmark_data_smithi154_225924_object88551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.424755+0000", 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.036973426000003, 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080673513000000002, 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198815 2024-09-13T14:59:05.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424758+0000", 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8930000000000001e-06 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424765+0000", 2024-09-13T14:59:05.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8210000000000004e-06 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424769+0000", 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7369999999999997e-06 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428701+0000", 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039317220000000003 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428826+0000", 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012472699999999999 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428948+0000", 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012246300000000001 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.432869+0000", 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003920626 2024-09-13T14:59:05.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.466811+0000", 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033941885999999998 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505293+0000", 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038482159000000002 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505389+0000", 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6051000000000006e-05 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505429+0000", 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.509 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198852 7.7 7:e4156631:::benchmark_data_smithi154_225924_object88588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.510600+0000", 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.951129295000001, 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.098632827000000006, 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198852 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510600+0000", 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510600+0000", 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510599+0000", 2024-09-13T14:59:05.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510604+0000", 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839999999999997e-06 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510615+0000", 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1566000000000001e-05 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510622+0000", 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510665+0000", 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2363000000000001e-05 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510782+0000", 2024-09-13T14:59:05.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117868 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510914+0000", 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013160799999999999 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.515165+0000", 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042514379999999997 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.518048+0000", 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002882201 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609141+0000", 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.091092993999999997 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609191+0000", 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0241999999999998e-05 2024-09-13T14:59:05.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609232+0000", 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1523999999999997e-05 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198899 7.7 7:e8acc7c2:::benchmark_data_smithi154_225924_object88635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.610904+0000", 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.850825037999996, 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.078514312000000003, 2024-09-13T14:59:05.516 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198899 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610904+0000", 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610904+0000", 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610897+0000", 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999928 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610911+0000", 2024-09-13T14:59:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4121000000000001e-05 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610925+0000", 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4239e-05 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610932+0000", 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2400000000000001e-06 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610979+0000", 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7040999999999999e-05 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.611109+0000", 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013000300000000001 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.611226+0000", 2024-09-13T14:59:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011720800000000001 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.615610+0000", 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043831440000000003 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.617396+0000", 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017859589999999999 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689331+0000", 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.071935456999999994 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689380+0000", 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8606999999999997e-05 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689418+0000", 2024-09-13T14:59:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8600999999999999e-05 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198901 7.c 7:30e20b28:::benchmark_data_smithi154_225924_object88637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.614143+0000", 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.847586098000001, 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.088790589000000003, 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198901 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614143+0000", 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614143+0000", 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614141+0000", 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614147+0000", 2024-09-13T14:59:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9830000000000001e-06 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614159+0000", 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.168e-05 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614166+0000", 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6159999999999998e-06 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614211+0000", 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5210000000000003e-05 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614339+0000", 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012781500000000001 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:59:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614459+0000", 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120282 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.618850+0000", 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043914069999999999 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.619032+0000", 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018138900000000001 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702829+0000", 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083797778000000003 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702886+0000", 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6388999999999998e-05 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702933+0000", 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7550999999999999e-05 2024-09-13T14:59:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198956 7.c 7:37128aaa:::benchmark_data_smithi154_225924_object88692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.743414+0000", 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.718314403999997, 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.105403712, 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198956 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743414+0000", 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743414+0000", 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743413+0000", 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743419+0000", 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1510000000000001e-06 2024-09-13T14:59:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743432+0000", 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2309e-05 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743440+0000", 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5660000000000007e-06 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750304+0000", 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068643200000000001 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750419+0000", 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114232 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,7", 2024-09-13T14:59:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750550+0000", 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130949 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.754768+0000", 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042181919999999999 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.754801+0000", 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3374999999999998e-05 2024-09-13T14:59:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848517+0000", 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.093715883999999999 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848778+0000", 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026113500000000001 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848818+0000", 2024-09-13T14:59:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9990000000000002e-05 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198960 7.d 7:b59b0bff:::benchmark_data_smithi154_225924_object88696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.748323+0000", 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.713405938000001, 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084968948000000002, 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198960 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748323+0000", 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748323+0000", 2024-09-13T14:59:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748322+0000", 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748326+0000", 2024-09-13T14:59:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7859999999999996e-06 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748334+0000", 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9619999999999993e-06 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748339+0000", 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9570000000000001e-06 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748557+0000", 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000217216 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748697+0000", 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014085599999999999 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748819+0000", 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012206600000000001 2024-09-13T14:59:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758177+0000", 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093571000000000001 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.765093+0000", 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006916447 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833219+0000", 2024-09-13T14:59:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.068125854999999999 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833252+0000", 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2867999999999998e-05 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833292+0000", 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0201999999999999e-05 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198961 7.d 7:be6e4db7:::benchmark_data_smithi154_225924_object88697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.750495+0000", 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.711233923000002, 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082969548000000004, 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198961 2024-09-13T14:59:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750495+0000", 2024-09-13T14:59:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750495+0000", 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750494+0000", 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750500+0000", 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2380000000000002e-06 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750513+0000", 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2578999999999999e-05 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750520+0000", 2024-09-13T14:59:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9500000000000004e-06 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753665+0000", 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031451449999999998 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753794+0000", 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012921499999999999 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753911+0000", 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117296 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758467+0000", 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045562709999999998 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.765159+0000", 2024-09-13T14:59:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066919529999999996 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833387+0000", 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.068227963000000003 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833429+0000", 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1740999999999999e-05 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833465+0000", 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5429999999999998e-05 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198966 7.7 7:e3c5df65:::benchmark_data_smithi154_225924_object88702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.758920+0000", 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.702808552, 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.093099978999999999, 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T14:59:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198966 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758920+0000", 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T14:59:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758920+0000", 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758919+0000", 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758927+0000", 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2209999999999997e-06 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758939+0000", 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1528999999999999e-05 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758947+0000", 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6519999999999995e-06 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758999+0000", 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1884e-05 2024-09-13T14:59:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.759133+0000", 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000134346 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.759259+0000", 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000125463 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.762880+0000", 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036212050000000002 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.836898+0000", 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.074017828999999993 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.851938+0000", 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015040359 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.851994+0000", 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5305e-05 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T14:59:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.852020+0000", 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6689000000000001e-05 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T14:59:05.558 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T14:59:05.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:05.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:05.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:05.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:05.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:05.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:05.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:05.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:05.967 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:05.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:06.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.092 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:06.203 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:59:06.220 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:06.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T14:59:06.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:06.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:06.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:06.467 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:06.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[107038]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 303'12997 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[107038]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 303'12947 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[107038]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 303'13142 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[107038]: osdmap e308: 8 total, 6 up, 7 in 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[107038]: pgmap v969: 89 pgs: 17 remapped+peering, 3 peering, 2 active+undersized+remapped, 6 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 23 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4218/42300 objects misplaced (9.972%); 0 B/s, 0 objects/s recovering 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2311985437' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[102314]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 303'12997 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[102314]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 303'12947 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[102314]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 303'13142 2024-09-13T14:59:06.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[102314]: osdmap e308: 8 total, 6 up, 7 in 2024-09-13T14:59:06.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[102314]: pgmap v969: 89 pgs: 17 remapped+peering, 3 peering, 2 active+undersized+remapped, 6 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 23 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4218/42300 objects misplaced (9.972%); 0 B/s, 0 objects/s recovering 2024-09-13T14:59:06.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2311985437' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:06.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:06 smithi172 ceph-mon[100696]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 303'12997 2024-09-13T14:59:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:06 smithi172 ceph-mon[100696]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 303'12947 2024-09-13T14:59:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:06 smithi172 ceph-mon[100696]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 303'13142 2024-09-13T14:59:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:06 smithi172 ceph-mon[100696]: osdmap e308: 8 total, 6 up, 7 in 2024-09-13T14:59:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:06 smithi172 ceph-mon[100696]: pgmap v969: 89 pgs: 17 remapped+peering, 3 peering, 2 active+undersized+remapped, 6 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 23 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4218/42300 objects misplaced (9.972%); 0 B/s, 0 objects/s recovering 2024-09-13T14:59:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2311985437' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:06.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:06.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:06.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:06.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:06.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:06.973 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:06.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:07.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:07.097 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:07.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:07.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:07.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:07.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:07.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2311985437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[102314]: osdmap e309: 8 total, 6 up, 7 in 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2311985437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:07.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:07 smithi154 ceph-mon[107038]: osdmap e309: 8 total, 6 up, 7 in 2024-09-13T14:59:07.471 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:07 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:07 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2311985437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:07 smithi172 ceph-mon[100696]: osdmap e309: 8 total, 6 up, 7 in 2024-09-13T14:59:07.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.596 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:07.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:07.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:07.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:07.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:07.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:07.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:07.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:07.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:08.037 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T14:59:08.037 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:08.037 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:59:08.037 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:59:08.050 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:59:08.051 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:08.068 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:59:08.068 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:08.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.084 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:59:08.085 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:08.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.125 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:59:08.125 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:08.143 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:59:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:08.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.304 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:08.304 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:08.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:08.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:08.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:08.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:08 smithi154 ceph-mon[102314]: osdmap e310: 8 total, 6 up, 7 in 2024-09-13T14:59:08.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:08 smithi154 ceph-mon[102314]: pgmap v972: 89 pgs: 17 remapped+peering, 3 peering, 2 active+undersized+remapped, 6 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 23 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4218/42300 objects misplaced (9.972%); 0 B/s, 0 objects/s recovering 2024-09-13T14:59:08.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3601418016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:08.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:08 smithi154 ceph-mon[107038]: osdmap e310: 8 total, 6 up, 7 in 2024-09-13T14:59:08.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:08 smithi154 ceph-mon[107038]: pgmap v972: 89 pgs: 17 remapped+peering, 3 peering, 2 active+undersized+remapped, 6 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 23 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4218/42300 objects misplaced (9.972%); 0 B/s, 0 objects/s recovering 2024-09-13T14:59:08.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3601418016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:08 smithi172 ceph-mon[100696]: osdmap e310: 8 total, 6 up, 7 in 2024-09-13T14:59:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:08 smithi172 ceph-mon[100696]: pgmap v972: 89 pgs: 17 remapped+peering, 3 peering, 2 active+undersized+remapped, 6 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 23 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4218/42300 objects misplaced (9.972%); 0 B/s, 0 objects/s recovering 2024-09-13T14:59:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3601418016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:08.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:08.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:08.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.713 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:08.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:08.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:08.837 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:59:08.905 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:59:08.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:08.962 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:08.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:09.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:09.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:09.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T14:59:09.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:09.211 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:09.313 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:09.335 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:09.347 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:09.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:09.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:09.467 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:09 smithi172 ceph-mon[100696]: osdmap e311: 8 total, 6 up, 7 in 2024-09-13T14:59:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:09 smithi154 ceph-mon[102314]: osdmap e311: 8 total, 6 up, 7 in 2024-09-13T14:59:09.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:09 smithi154 ceph-mon[107038]: osdmap e311: 8 total, 6 up, 7 in 2024-09-13T14:59:09.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:59:09.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:09.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:09.676 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:09.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:09.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:09.835 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2, 0] live_osds: [1, 3, 4, 5, 6, 7] 2024-09-13T14:59:09.835 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:59:09.835 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.852364533921138 2024-09-13T14:59:09.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 3 0.852364533921138 2024-09-13T14:59:09.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:09.954 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:09.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:10.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:10.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:10.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:10.201 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:10.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.324 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:10.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:10.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:10.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:10 smithi172 ceph-mon[100696]: osdmap e312: 8 total, 6 up, 7 in 2024-09-13T14:59:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:10 smithi172 ceph-mon[100696]: pgmap v975: 89 pgs: 1 peering, 11 remapped+peering, 2 active+undersized+remapped, 13 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4224/42300 objects misplaced (9.986%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:59:10.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:10.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:10.630 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:10 smithi154 ceph-mon[102314]: osdmap e312: 8 total, 6 up, 7 in 2024-09-13T14:59:10.630 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:10 smithi154 ceph-mon[102314]: pgmap v975: 89 pgs: 1 peering, 11 remapped+peering, 2 active+undersized+remapped, 13 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4224/42300 objects misplaced (9.986%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:59:10.631 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:10 smithi154 ceph-mon[107038]: osdmap e312: 8 total, 6 up, 7 in 2024-09-13T14:59:10.631 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:10 smithi154 ceph-mon[107038]: pgmap v975: 89 pgs: 1 peering, 11 remapped+peering, 2 active+undersized+remapped, 13 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 4224/42300 objects misplaced (9.986%); 14 MiB/s, 218 objects/s recovering 2024-09-13T14:59:10.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:10.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:10.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:10.906 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:10.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:10.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:10.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:11.036 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:11.037 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:11.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:11.065 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:11.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:11.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:11.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:11.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:11.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:11.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:11.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:11.488 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:11 smithi154 ceph-mon[102314]: osdmap e313: 8 total, 6 up, 7 in 2024-09-13T14:59:11.488 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:11 smithi154 ceph-mon[107038]: osdmap e313: 8 total, 6 up, 7 in 2024-09-13T14:59:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:11 smithi172 ceph-mon[100696]: osdmap e313: 8 total, 6 up, 7 in 2024-09-13T14:59:11.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:11.561 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:11.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:11.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:11.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T14:59:11.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:11.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:11.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:11.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:11.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:11.939 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T14:59:12.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.062 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:12.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:12.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:12.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:12.217 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:59:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:59:11] "GET /metrics HTTP/1.1" 200 36606 "" "Prometheus/2.51.0" 2024-09-13T14:59:12.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:12.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.310 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:12.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:12.327 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:12 smithi172 ceph-mon[100696]: pgmap v977: 89 pgs: 1 peering, 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 25 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 39 MiB/s, 625 objects/s recovering 2024-09-13T14:59:12.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:12.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:12 smithi154 ceph-mon[102314]: pgmap v977: 89 pgs: 1 peering, 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 25 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 39 MiB/s, 625 objects/s recovering 2024-09-13T14:59:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:12 smithi154 ceph-mon[107038]: pgmap v977: 89 pgs: 1 peering, 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 25 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 39 MiB/s, 625 objects/s recovering 2024-09-13T14:59:12.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.559 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:12.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:12.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:12.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:12.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:12.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:12.822 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:12.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:12.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:12.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:13.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:13.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:13.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:13.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:13.249 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:59:13.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.301 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:13.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:13.375 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:13.375 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2571575754' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:13.376 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:13 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:13.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:13.423 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:13.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:13.547 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:13.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:13.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2571575754' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:13.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:13 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:13.594 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:13.594 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2571575754' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:13.594 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:13 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:13.618 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:13.618 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:13.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:13.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:13.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:13.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.794 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:13.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:13.814 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:13.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:13.916 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:13.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:14.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:14.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:14.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.164 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:14.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:14.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:14.287 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:14 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:14.287 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:14 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:14.287 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:14 smithi172 ceph-mon[100696]: osdmap e314: 8 total, 6 up, 7 in 2024-09-13T14:59:14.287 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:14 smithi172 ceph-mon[100696]: pgmap v979: 89 pgs: 1 peering, 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 25 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 32 MiB/s, 505 objects/s recovering 2024-09-13T14:59:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:14.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:14.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:14.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:14.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:14.533 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:14.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:14.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:14.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[102314]: osdmap e314: 8 total, 6 up, 7 in 2024-09-13T14:59:14.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[102314]: pgmap v979: 89 pgs: 1 peering, 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 25 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 32 MiB/s, 505 objects/s recovering 2024-09-13T14:59:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[107038]: osdmap e314: 8 total, 6 up, 7 in 2024-09-13T14:59:14.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:14 smithi154 ceph-mon[107038]: pgmap v979: 89 pgs: 1 peering, 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 25 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 32 MiB/s, 505 objects/s recovering 2024-09-13T14:59:14.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T14:59:14.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:14.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:14.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:14.934 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:14.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:15.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:15.057 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:15.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:15.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:15.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:15.304 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:15.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:15.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:15.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:15.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:15.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.675 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:15.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:15.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:15.798 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:15.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:15.922 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:15.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:16.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:16.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:16.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:16.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:16.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:16.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:16.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:16.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T14:59:16.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:16.416 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:16.474 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.3 to 0.852365 (da34) 2024-09-13T14:59:16.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:16.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:16.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:16.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:16.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:16 smithi154 ceph-mon[102314]: pgmap v980: 89 pgs: 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 26 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 26 MiB/s, 423 objects/s recovering 2024-09-13T14:59:16.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2102080216' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138} : dispatch 2024-09-13T14:59:16.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:16 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138} : dispatch 2024-09-13T14:59:16.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:16 smithi154 ceph-mon[107038]: pgmap v980: 89 pgs: 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 26 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 26 MiB/s, 423 objects/s recovering 2024-09-13T14:59:16.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2102080216' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138} : dispatch 2024-09-13T14:59:16.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:16 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138} : dispatch 2024-09-13T14:59:16.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:16 smithi172 ceph-mon[100696]: pgmap v980: 89 pgs: 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 26 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%); 26 MiB/s, 423 objects/s recovering 2024-09-13T14:59:16.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2102080216' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138} : dispatch 2024-09-13T14:59:16.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:16 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138} : dispatch 2024-09-13T14:59:16.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:16.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:16.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:16.912 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:16.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:17.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:17.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:17.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:17.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:17.190 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:17.212 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:17.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:17.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:17.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:17.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:17.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:17.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:17.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:17.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:17.714 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:17 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138}]': finished 2024-09-13T14:59:17.714 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:17 smithi154 ceph-mon[102314]: osdmap e315: 8 total, 6 up, 7 in 2024-09-13T14:59:17.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:17 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138}]': finished 2024-09-13T14:59:17.714 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:17 smithi154 ceph-mon[107038]: osdmap e315: 8 total, 6 up, 7 in 2024-09-13T14:59:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:17 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.852364533921138}]': finished 2024-09-13T14:59:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:17 smithi172 ceph-mon[100696]: osdmap e315: 8 total, 6 up, 7 in 2024-09-13T14:59:17.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.785 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:17.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:17.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:17.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:17.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:18.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:18.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:18.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:18.286 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.331 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:59:18.331 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T14:59:18.332 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:18.332 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:59:18.344 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:59:18.345 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:18.361 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:59:18.361 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:18.379 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:59:18.379 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:18.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:18.394 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:59:18.395 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:18.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.413 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:59:18.413 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:18.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:18.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:18.594 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:18 smithi154 ceph-mon[102314]: pgmap v982: 89 pgs: 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 26 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%) 2024-09-13T14:59:18.594 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:18 smithi154 ceph-mon[102314]: osdmap e316: 8 total, 6 up, 7 in 2024-09-13T14:59:18.594 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4200738436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:18.595 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:18 smithi154 ceph-mon[107038]: pgmap v982: 89 pgs: 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 26 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%) 2024-09-13T14:59:18.595 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:18 smithi154 ceph-mon[107038]: osdmap e316: 8 total, 6 up, 7 in 2024-09-13T14:59:18.595 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4200738436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:18.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:18.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:18 smithi172 ceph-mon[100696]: pgmap v982: 89 pgs: 4 active+undersized+remapped, 21 active+clean+remapped, 2 undersized+peered, 24 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 26 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 7778/42300 objects misplaced (18.388%) 2024-09-13T14:59:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:18 smithi172 ceph-mon[100696]: osdmap e316: 8 total, 6 up, 7 in 2024-09-13T14:59:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4200738436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:18.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:18.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:18.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:18.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:18.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:19.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:19.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:19.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:19.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:19.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:19.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:19.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:19.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:19.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:19.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:19.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:19.653 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:19.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:19 smithi154 ceph-mon[102314]: osdmap e317: 8 total, 6 up, 7 in 2024-09-13T14:59:19.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:19 smithi154 ceph-mon[107038]: osdmap e317: 8 total, 6 up, 7 in 2024-09-13T14:59:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:19 smithi172 ceph-mon[100696]: osdmap e317: 8 total, 6 up, 7 in 2024-09-13T14:59:19.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:19.776 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:59:19.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:59:19.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:19.900 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:19.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:20.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:20.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:20.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:20.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:20.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:20.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:20.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.399 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:20.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:20.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:20.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:20.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.648 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:20.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:20.713 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:20 smithi154 ceph-mon[102314]: pgmap v985: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:20.713 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:20.713 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:20 smithi154 ceph-mon[107038]: pgmap v985: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:20.713 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:20 smithi172 ceph-mon[100696]: pgmap v985: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:20.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:20.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:20.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:20.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:20.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:20.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:21.024 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:21.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:21.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:21.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:21.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:21.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:21.400 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:21.488 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T14:59:21.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:21.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:21.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:21.629 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/505522865' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:21.630 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/505522865' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.652 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:21.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:21.728 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:59:21.728 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T14:59:21.728 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T14:59:21.728 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:59:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/505522865' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T14:59:21.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.776 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:21.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:21.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:21.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:21.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:21.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:21.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:59:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:59:21] "GET /metrics HTTP/1.1" 200 36638 "" "Prometheus/2.51.0" 2024-09-13T14:59:22.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:22.022 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:22.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:22.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:22.152 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2, 0] live_osds: [1, 3, 4, 5, 6, 7] 2024-09-13T14:59:22.152 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T14:59:22.152 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-13T14:59:22.152 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-09-13T14:59:22.152 DEBUG:teuthology.orchestra.run.smithi154:osd.0> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T14:59:22.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl start ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0 2024-09-13T14:59:22.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:22.271 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:22.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:22.391 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:22 smithi154 systemd[1]: Starting Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:59:22.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:22.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T14:59:22.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:22.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:22.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.642 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:22.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[102314]: pgmap v986: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/505522865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[102314]: osdmap e318: 8 total, 6 up, 7 in 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[107038]: pgmap v986: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/505522865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:22 smithi154 ceph-mon[107038]: osdmap e318: 8 total, 6 up, 7 in 2024-09-13T14:59:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:22 smithi172 ceph-mon[100696]: pgmap v986: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:22 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:22 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/505522865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T14:59:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:22 smithi172 ceph-mon[100696]: osdmap e318: 8 total, 6 up, 7 in 2024-09-13T14:59:22.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:22.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:22.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:22.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:22.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:22.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:23.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:23.081 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:23.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:23.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:23.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:23.259 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:23.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:23.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:23.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:23.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:23.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:23.629 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:23.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:23.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:23.789 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:23.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:23.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:23.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:23.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:24.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:24.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:24.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:24.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:24.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:23 smithi154 podman[252617]: 2024-09-13 14:59:23.794528838 +0000 UTC m=+0.675474845 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:59:24.221 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:23 smithi154 podman[252617]: 2024-09-13 14:59:23.924360339 +0000 UTC m=+0.805306345 container create 6ad42062c5fc5a8f77e1d3af59bbc643c37ad02a0e6ffcba4d0d097574c6777a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T14:59:24.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:24.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:24.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:24.372 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:24.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.496 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:24.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:24.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:24.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:24.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:24.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:24.759 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:24 smithi172 ceph-mon[100696]: pgmap v988: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:24.797 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:24 smithi154 ceph-mon[102314]: pgmap v988: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:24.798 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:24 smithi154 ceph-mon[107038]: pgmap v988: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:24.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:24.868 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:24.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:24.992 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:24.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:25.047 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:24 smithi154 podman[252617]: 2024-09-13 14:59:24.907580891 +0000 UTC m=+1.788526898 container init 6ad42062c5fc5a8f77e1d3af59bbc643c37ad02a0e6ffcba4d0d097574c6777a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T14:59:25.047 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:24 smithi154 podman[252617]: 2024-09-13 14:59:24.911799201 +0000 UTC m=+1.792745208 container start 6ad42062c5fc5a8f77e1d3af59bbc643c37ad02a0e6ffcba4d0d097574c6777a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-13T14:59:25.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T14:59:25.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:25.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:25.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:25.238 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:25.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:25.354 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.355 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.355 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:59:25.355 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.355 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.355 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:59:25.355 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:59:25.361 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:25.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.485 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:25.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:25.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:25.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:25.632 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:25.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:25.720 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 podman[252617]: 2024-09-13 14:59:25.577054125 +0000 UTC m=+2.458000132 container attach 6ad42062c5fc5a8f77e1d3af59bbc643c37ad02a0e6ffcba4d0d097574c6777a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T14:59:25.720 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 bash[252617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 bash[252617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 bash[252617]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:59:25.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 bash[252617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 bash[252617]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:25.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 bash[252617]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:59:25.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:25 smithi154 bash[252617]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-09-13T14:59:25.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:25.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:25.856 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:25.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:25.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:25.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:26.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:26.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:26.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:26.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:26.312 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:59:26.312 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 bash[252617]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:59:26.312 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:59:26.312 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 bash[252617]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-09-13T14:59:26.313 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:59:26.313 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 bash[252617]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-13T14:59:26.313 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:59:26.313 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 bash[252617]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T14:59:26.313 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate[252693]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:59:26.313 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 bash[252617]: --> ceph-volume lvm activate successful for osd ID: 0 2024-09-13T14:59:26.313 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:26 smithi154 podman[252617]: 2024-09-13 14:59:26.310968523 +0000 UTC m=+3.191914532 container died 6ad42062c5fc5a8f77e1d3af59bbc643c37ad02a0e6ffcba4d0d097574c6777a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0) 2024-09-13T14:59:26.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.353 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:26.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:26.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:26.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:26.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:26.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:26.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:26 smithi154 ceph-mon[107038]: pgmap v989: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:26.627 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:26 smithi154 ceph-mon[102314]: pgmap v989: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:26.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:26.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:26.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:26.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:26.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:26.976 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:26.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:26.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:26 smithi172 ceph-mon[100696]: pgmap v989: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:27.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:27.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:27.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:27.226 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:27.303 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:27 smithi154 podman[253000]: 2024-09-13 14:59:27.035792069 +0000 UTC m=+0.846758407 container remove 6ad42062c5fc5a8f77e1d3af59bbc643c37ad02a0e6ffcba4d0d097574c6777a (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T14:59:27.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:27.351 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:27.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:27.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:27.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:27.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:27.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:27.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:27.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:27.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:27.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:27.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:27.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:28.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.103 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:28.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:28.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:28.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:28.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:28.351 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:28.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:28.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:28.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:28.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:28.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:28.707 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T14:59:28.707 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:59:28.707 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:28.708 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:59:28.710 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:59:28.710 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:28 smithi154 ceph-mon[102314]: pgmap v990: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:28.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:28 smithi154 podman[253159]: 2024-09-13 14:59:28.599559859 +0000 UTC m=+0.349702075 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:59:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:28 smithi154 ceph-mon[107038]: pgmap v990: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:28.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:28.729 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:59:28.729 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:28.745 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:59:28.745 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:28.762 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:59:28.763 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:28.780 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:59:28.780 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:28.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.856 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:28.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:28.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:28.982 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:28.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:28 smithi172 ceph-mon[100696]: pgmap v990: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:29.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:29.109 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:29.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:29.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:29.335 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:29 smithi154 podman[253159]: 2024-09-13 14:59:29.135512137 +0000 UTC m=+0.885654339 container create ab20bf2d0511297fe9a2d35b428088bf7b0849df4cf6d152a0f7dfc94c7f38f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-13T14:59:29.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:29.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:29.413 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:29.414 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:29.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:29.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:29.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:29.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:29.692 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:29 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/663352959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:29.692 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:29 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/663352959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:29.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:29.740 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:29.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:29.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:29.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:29.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T14:59:29.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:29 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/663352959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:29.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:29.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:30.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:30.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:30.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:30.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:30.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:30.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:30.383 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:59:30.470 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:30 smithi154 podman[253159]: 2024-09-13 14:59:30.218726354 +0000 UTC m=+1.968868551 container init ab20bf2d0511297fe9a2d35b428088bf7b0849df4cf6d152a0f7dfc94c7f38f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-13T14:59:30.471 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:30 smithi154 podman[253159]: 2024-09-13 14:59:30.223374185 +0000 UTC m=+1.973516381 container start ab20bf2d0511297fe9a2d35b428088bf7b0849df4cf6d152a0f7dfc94c7f38f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux ) 2024-09-13T14:59:30.471 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:30 smithi154 ceph-osd[253376]: -- 172.21.15.154:0/1885456522 <== mon.0 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b0712da40 con 0x555b07efa000 2024-09-13T14:59:30.471 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:30 smithi154 bash[253159]: ab20bf2d0511297fe9a2d35b428088bf7b0849df4cf6d152a0f7dfc94c7f38f5 2024-09-13T14:59:30.471 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:30 smithi154 systemd[1]: Started Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T14:59:30.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:30.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:30.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:30.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:30.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:30.743 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:30.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:30.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:30.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:30 smithi154 ceph-mon[107038]: pgmap v991: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:30.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:30 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2202543269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:30 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:30 smithi154 ceph-mon[102314]: pgmap v991: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:30.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:30 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2202543269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:30 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:30 smithi172 ceph-mon[100696]: pgmap v991: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:30 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2202543269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:30 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:30.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:30.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:59:31.059 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:59:31.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:31.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:31.165 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T14:59:31.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:31.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:31.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:31.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:31.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:31.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:31.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:31.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:31.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:31.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:31.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-mon[102314]: osdmap e319: 8 total, 6 up, 7 in 2024-09-13T14:59:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-mon[107038]: osdmap e319: 8 total, 6 up, 7 in 2024-09-13T14:59:31.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:31.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:31.835 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:31.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:31.874 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:31.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:31.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:31.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:31 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T14:59:31.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:31 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T14:59:31.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:31 smithi172 ceph-mon[100696]: osdmap e319: 8 total, 6 up, 7 in 2024-09-13T14:59:32.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:32.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:32.120 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[253281]: 2024-09-13T14:59:31.969+0000 7f74687a2640 -1 Falling back to public interface 2024-09-13T14:59:32.121 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:59:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:59:31] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-13T14:59:32.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:32.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:32.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:32.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:32.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:32.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:32.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:32.593 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:32.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:32.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:32.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.767 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:32.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:32.794 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:32 smithi154 ceph-mon[102314]: pgmap v993: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:32.794 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:32 smithi154 ceph-mon[107038]: pgmap v993: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:32.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:32.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:32.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:32 smithi172 ceph-mon[100696]: pgmap v993: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:32.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:33.020 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:33.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:33.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:33.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:33.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:33.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:33.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:33.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:33.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:33.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:33.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.658 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:33.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:33.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:33.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T14:59:33.785 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:33.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:33.913 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:33.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:34.011 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:34.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:34.021 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:34.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:34.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:34.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:34.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:34.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:34.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:34.427 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:34.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:34.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:34.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:34.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:34.775 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:34 smithi154 ceph-mon[102314]: pgmap v994: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:34.775 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:34 smithi154 ceph-mon[107038]: pgmap v994: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:34.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:34.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:34.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:34.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:34.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:34.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:34 smithi172 ceph-mon[100696]: pgmap v994: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:35.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:35.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:35.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:35.185 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:35.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:35.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:35.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:35.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:35.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:no valid command found; 10 closest matches: 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:0 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:1 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:2 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:abort 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:assert 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs debug_inject_read_zeros 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs files list 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs stats 2024-09-13T14:59:35.546 INFO:teuthology.orchestra.run.smithi154.stderr:bluestore bluefs device info [] 2024-09-13T14:59:35.547 INFO:teuthology.orchestra.run.smithi154.stderr:config diff 2024-09-13T14:59:35.547 INFO:teuthology.orchestra.run.smithi154.stderr:admin_socket: invalid command 2024-09-13T14:59:35.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:35.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:35.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:35.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:35.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:35.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:35.946 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:35.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:36.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:36.072 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T14:59:36.161 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-13T14:59:36.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:36.199 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:36.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:36.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:36.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:36.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:36.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:36.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:36.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:36.701 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:36 smithi154 ceph-mon[102314]: pgmap v995: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:36 smithi154 ceph-mon[107038]: pgmap v995: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:36.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:36.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:36.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:36.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:36.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:36.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:36 smithi172 ceph-mon[100696]: pgmap v995: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:37.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:37.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:37.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.211 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:37.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:37.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:37.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:37.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:37.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:37.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:37.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:37.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:37.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:37.816 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:37.816 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:37.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:37.842 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:37.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:37.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:37.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:38.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:38.096 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:38.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:38.226 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:38.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:38.349 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:38.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:38.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:38.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:38.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:38.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:38.733 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:38.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:38.861 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:38.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:38.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:38.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:38 smithi154 ceph-mon[102314]: pgmap v996: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:38.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:38 smithi154 ceph-mon[107038]: pgmap v996: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:38.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:38.988 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:38.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:38.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:38 smithi172 ceph-mon[100696]: pgmap v996: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:39.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T14:59:39.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:39.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:39.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.348 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:39.374 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:39.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:39.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:39.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:39.627 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:39.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.754 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:39.881 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:39.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:39.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:40.007 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:40.088 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:39 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[253281]: 2024-09-13T14:59:39.801+0000 7f74687a2640 -1 osd.0 301 log_to_monitors true 2024-09-13T14:59:40.109 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:40.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:40.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.259 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:40.361 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:40.384 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:40.449 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T14:59:40.449 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:40.449 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:59:40.449 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:40.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:59:40.450 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:59:40.456 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:59:40.457 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:40.473 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:59:40.473 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:40.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:40.490 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:59:40.490 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:40.506 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:59:40.506 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:40.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:40.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:40.720 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 14:59:40 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[253281]: 2024-09-13T14:59:40.565+0000 7f745c544640 -1 osd.0 320 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T14:59:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:40 smithi154 ceph-mon[102314]: pgmap v997: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:40 smithi154 ceph-mon[102314]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:59:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:40 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/16134687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:40 smithi154 ceph-mon[107038]: pgmap v997: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:40 smithi154 ceph-mon[107038]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:59:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:40 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/16134687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:40.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:40.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.884 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:40.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:40.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:40 smithi172 ceph-mon[100696]: pgmap v997: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:40 smithi172 ceph-mon[100696]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-13T14:59:40.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:40 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/16134687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.010 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:41.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:41.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T14:59:41.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:41.257 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:41.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:41.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:41.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:59:41.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:59:41.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:41.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:41.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:41.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:41 smithi154 ceph-mon[102314]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:59:41.658 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:41 smithi154 ceph-mon[102314]: osdmap e320: 8 total, 6 up, 7 in 2024-09-13T14:59:41.658 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:41 smithi154 ceph-mon[102314]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:59:41.658 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:41 smithi154 ceph-mon[107038]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:59:41.658 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:41 smithi154 ceph-mon[107038]: osdmap e320: 8 total, 6 up, 7 in 2024-09-13T14:59:41.658 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:41 smithi154 ceph-mon[107038]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:59:41.701 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:41.733 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:41.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:41.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:41.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:41.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:41.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:41.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:59:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:59:41] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-13T14:59:41.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:41 smithi172 ceph-mon[100696]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T14:59:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:41 smithi172 ceph-mon[100696]: osdmap e320: 8 total, 6 up, 7 in 2024-09-13T14:59:41.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:41 smithi172 ceph-mon[100696]: from='osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T14:59:42.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:42.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:42.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:42.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:42.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:42.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:42.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:42.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:42.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:42.536 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T14:59:42.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:42.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:42.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: pgmap v999: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: Health check update: 1 osds down (OSD_DOWN) 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424] boot 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: osdmap e321: 8 total, 7 up, 7 in 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1546867391' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1546867391' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[107038]: osdmap e322: 8 total, 7 up, 7 in 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: pgmap v999: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:42.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: Health check update: 1 osds down (OSD_DOWN) 2024-09-13T14:59:42.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424] boot 2024-09-13T14:59:42.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: osdmap e321: 8 total, 7 up, 7 in 2024-09-13T14:59:42.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:59:42.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1546867391' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:42.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:42.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1546867391' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:42.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:42 smithi154 ceph-mon[102314]: osdmap e322: 8 total, 7 up, 7 in 2024-09-13T14:59:42.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.747 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:42.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:42.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:42.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:42.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: pgmap v999: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: Health check update: 1 osds down (OSD_DOWN) 2024-09-13T14:59:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: osd.0 [v2:172.21.15.154:6802/3127717424,v1:172.21.15.154:6803/3127717424] boot 2024-09-13T14:59:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: osdmap e321: 8 total, 7 up, 7 in 2024-09-13T14:59:42.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-13T14:59:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1546867391' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T14:59:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T14:59:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1546867391' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T14:59:42.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:42 smithi172 ceph-mon[100696]: osdmap e322: 8 total, 7 up, 7 in 2024-09-13T14:59:42.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:42.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:43.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:43.118 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:43.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:43.243 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:43.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:43.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:43.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T14:59:43.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:43.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:43.591 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:43.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:43.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:43.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:43 smithi154 ceph-mon[102314]: 7.1 starting backfill to osd.0 from (0'0,0'0] MAX to 303'12997 2024-09-13T14:59:43.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:43.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:43 smithi154 ceph-mon[102314]: osdmap e323: 8 total, 7 up, 7 in 2024-09-13T14:59:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:43 smithi154 ceph-mon[107038]: 7.1 starting backfill to osd.0 from (0'0,0'0] MAX to 303'12997 2024-09-13T14:59:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:43.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:43 smithi154 ceph-mon[107038]: osdmap e323: 8 total, 7 up, 7 in 2024-09-13T14:59:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.742 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:43.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:43.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:43.874 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:43.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:43.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:43 smithi172 ceph-mon[100696]: 7.1 starting backfill to osd.0 from (0'0,0'0] MAX to 303'12997 2024-09-13T14:59:43.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:43.893 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:43 smithi172 ceph-mon[100696]: osdmap e323: 8 total, 7 up, 7 in 2024-09-13T14:59:43.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.004 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:44.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:44.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:44.146 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:44.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:44.294 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:44.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:44.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:44.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:44.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:44.600 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:44 smithi172 ceph-mon[100696]: pgmap v1002: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:44.600 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:44 smithi172 ceph-mon[100696]: Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2024-09-13T14:59:44.600 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:44 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:44.683 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:207449 7.2 7:49fd1986:::benchmark_data_smithi154_225924_object97185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:44.654760+0000", 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.026707847, 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026738187, 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:59:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 207449 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.654760+0000", 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.654760+0000", 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.654760+0000", 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3300000000000003e-07 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.654762+0000", 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0090000000000002e-06 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.654768+0000", 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0480000000000004e-06 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.654773+0000", 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9230000000000001e-06 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.666614+0000", 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01184109 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.669474+0000", 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028594620000000001 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1", 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.669602+0000", 2024-09-13T14:59:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012790999999999999 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.679980+0000", 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010378317999999999 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:207457 7.c 7:3c48775e:::benchmark_data_smithi154_225924_object97193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:44.670631+0000", 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.010836520000000001, 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010945208, 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 207457 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670631+0000", 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670631+0000", 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670633+0000", 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7370000000000001e-06 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670640+0000", 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3459999999999996e-06 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670654+0000", 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3675999999999999e-05 2024-09-13T14:59:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670662+0000", 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1939999999999998e-06 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670756+0000", 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3954000000000002e-05 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670885+0000", 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012945099999999999 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6,7", 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.670990+0000", 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105009 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.674447+0000", 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003456476 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.674607+0000", 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016019499999999999 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:207460 7.2 7:47cbeedb:::benchmark_data_smithi154_225924_object97196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:44.673288+0000", 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0081791220000000005, 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083581699999999998, 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 207460 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673288+0000", 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673288+0000", 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673290+0000", 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.254e-06 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673296+0000", 2024-09-13T14:59:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9970000000000004e-06 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673306+0000", 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0783e-05 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673313+0000", 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0380000000000002e-06 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673365+0000", 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1792000000000002e-05 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673485+0000", 2024-09-13T14:59:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119652 2024-09-13T14:59:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1", 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673602+0000", 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011687 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.679986+0000", 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063839839999999997 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45383.0:207458 7.8 e323/298 7:1cc7608e:::benchmark_data_smithi154_225924_object97194:head v 323'13001, mlcod=323'12999)", 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:44.673714+0000", 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0077531689999999999, 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080015390000000002, 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673714+0000", 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673714+0000", 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673715+0000", 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1260000000000001e-06 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673722+0000", 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0550000000000002e-06 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673735+0000", 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2312e-05 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.673742+0000", 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5379999999999997e-06 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.674795+0000", 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010528429999999999 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.674856+0000", 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0813999999999997e-05 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45383.0:207459 7.f e323/322 7:f94367c1:::benchmark_data_smithi154_225924_object97195:head v 323'12802, mlcod=323'12800)", 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:44.678264+0000", 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0032032990000000002, 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035010280000000002, 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678264+0000", 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678264+0000", 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678265+0000", 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1400000000000001e-06 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678273+0000", 2024-09-13T14:59:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.858e-06 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678291+0000", 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8080999999999998e-05 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678300+0000", 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.721e-06 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678356+0000", 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6511999999999997e-05 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:44.678378+0000", 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1713999999999999e-05 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 5 2024-09-13T14:59:44.709 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:59:44.738 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:44.778 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:44 smithi154 ceph-mon[102314]: pgmap v1002: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:44.778 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:44 smithi154 ceph-mon[102314]: Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2024-09-13T14:59:44.779 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:44 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:44.779 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:44 smithi154 ceph-mon[107038]: pgmap v1002: 89 pgs: 5 active+undersized+remapped, 25 active+clean+remapped, 2 undersized+peered, 21 active+undersized, 3 undersized+degraded+peered, 9 active+undersized+degraded, 24 active+clean; 884 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 6279/42300 objects degraded (14.844%); 8649/42300 objects misplaced (20.447%) 2024-09-13T14:59:44.779 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:44 smithi154 ceph-mon[107038]: Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2024-09-13T14:59:44.779 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:44 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6279/42300 objects degraded (14.844%), 12 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:44.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:44.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:44.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:44.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:45.006 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:45.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:45.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:45.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:45.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:45.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:45.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:45.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:45.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:45.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:45.673 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:45.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T14:59:45.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:45.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:45.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:45.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:45.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:46.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:46.087 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:46.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:46.215 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:46.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:46.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T14:59:46.280 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:46.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:46.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:46.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:46.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:46.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.645 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:46.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:46.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:46.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:46.814 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:46 smithi154 ceph-mon[102314]: pgmap v1004: 89 pgs: 1 active+remapped+backfilling, 21 active+clean+remapped, 1 undersized+peered, 15 active+undersized, 1 undersized+degraded+peered, 8 active+undersized+degraded, 42 active+clean; 836 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 157 op/s; 5023/39990 objects degraded (12.561%); 6206/39990 objects misplaced (15.519%); 6.6 MiB/s, 115 objects/s recovering 2024-09-13T14:59:46.815 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:46 smithi154 ceph-mon[107038]: pgmap v1004: 89 pgs: 1 active+remapped+backfilling, 21 active+clean+remapped, 1 undersized+peered, 15 active+undersized, 1 undersized+degraded+peered, 8 active+undersized+degraded, 42 active+clean; 836 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 157 op/s; 5023/39990 objects degraded (12.561%); 6206/39990 objects misplaced (15.519%); 6.6 MiB/s, 115 objects/s recovering 2024-09-13T14:59:46.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:46.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:46.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:46.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:46 smithi172 ceph-mon[100696]: pgmap v1004: 89 pgs: 1 active+remapped+backfilling, 21 active+clean+remapped, 1 undersized+peered, 15 active+undersized, 1 undersized+degraded+peered, 8 active+undersized+degraded, 42 active+clean; 836 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 157 op/s; 5023/39990 objects degraded (12.561%); 6206/39990 objects misplaced (15.519%); 6.6 MiB/s, 115 objects/s recovering 2024-09-13T14:59:47.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:47.056 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:47.094 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:209712 7.7 7:e25c3357:::benchmark_data_smithi154_225924_object99448:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.052774+0000", 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.034946632999999998, 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034977932000000003, 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:59:47.095 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 209712 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.052774+0000", 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.052774+0000", 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.052775+0000", 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.717e-06 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.052780+0000", 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8609999999999997e-06 2024-09-13T14:59:47.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.054000+0000", 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00121978 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.054010+0000", 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0044000000000001e-05 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:47.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.054111+0000", 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010084199999999999 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.054275+0000", 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016435199999999999 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.054373+0000", 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7703999999999998e-05 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.076326+0000", 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021952758999999999 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.45383.0:209712 7.7 e323/322)", 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.077923+0000", 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0097969100000000007, 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099016380000000008, 2024-09-13T14:59:47.100 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077923+0000", 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077923+0000", 2024-09-13T14:59:47.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077924+0000", 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5000000000000001e-07 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077929+0000", 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.78e-06 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077944+0000", 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5218999999999999e-05 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077951+0000", 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7769999999999997e-06 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.103 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:209731 7.9 7:9dc77935:::benchmark_data_smithi154_225924_object99467:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.077976+0000", 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0097437500000000007, 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098994770000000003, 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 209731 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077976+0000", 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077976+0000", 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077978+0000", 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.463e-06 2024-09-13T14:59:47.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077981+0000", 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2009999999999999e-06 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077990+0000", 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9500000000000007e-06 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.077995+0000", 2024-09-13T14:59:47.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2959999999999998e-06 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.080953+0000", 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029576070000000001 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081088+0000", 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013463700000000001 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4", 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081204+0000", 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116035 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45383.0:209733 7.5 e323/322 7:a5f92be1:::benchmark_data_smithi154_225924_object99469:head v 323'12937, mlcod=323'12935)", 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.081009+0000", 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0067108150000000002, 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069233560000000003, 2024-09-13T14:59:47.108 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081009+0000", 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081009+0000", 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081010+0000", 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6899999999999998e-07 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081016+0000", 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1700000000000002e-06 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081024+0000", 2024-09-13T14:59:47.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.055e-06 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081030+0000", 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7400000000000001e-06 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081090+0000", 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0628000000000003e-05 2024-09-13T14:59:47.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:47.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.081114+0000", 2024-09-13T14:59:47.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3306e-05 2024-09-13T14:59:47.112 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.112 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.112 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45383.0:209736 7.1 e323/321 7:856bff0d:::benchmark_data_smithi154_225924_object99472:head v 323'13192, mlcod=323'13189)", 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.087221+0000", 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.00049946099999999998, 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00076595199999999997, 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087221+0000", 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087221+0000", 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087221+0000", 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0699999999999996e-07 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087226+0000", 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3599999999999998e-06 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087238+0000", 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2071e-05 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087245+0000", 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2980000000000003e-06 2024-09-13T14:59:47.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087368+0000", 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012317200000000001 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087389+0000", 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0496000000000002e-05 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.116 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.45383.0:209731 7.9 e323/321)", 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.087267+0000", 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.00045343400000000001, 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00076246599999999995, 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087267+0000", 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087267+0000", 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087267+0000", 2024-09-13T14:59:47.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1e-07 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087270+0000", 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.711e-06 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087276+0000", 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6800000000000004e-06 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.087281+0000", 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5329999999999996e-06 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:209718 7.7 7:e138c8b4:::benchmark_data_smithi154_225924_object99454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.064195+0000", 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.023524977999999998, 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023873628000000001, 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.120 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 209718 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.064195+0000", 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.064195+0000", 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.064196+0000", 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9999999999999998e-07 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.064198+0000", 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2539999999999999e-06 2024-09-13T14:59:47.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.064205+0000", 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8619999999999997e-06 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.064210+0000", 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.955e-06 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.076354+0000", 2024-09-13T14:59:47.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012144245 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.076464+0000", 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110176 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.078364+0000", 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018993910000000001 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.124 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:209726 7.7 7:e25508a1:::benchmark_data_smithi154_225924_object99462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:47.071216+0000", 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.016503974000000001, 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016911498000000001, 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T14:59:47.125 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 209726 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.071216+0000", 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.071216+0000", 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T14:59:47.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.071217+0000", 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0999999999999997e-07 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.071221+0000", 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3680000000000004e-06 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.071231+0000", 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3889999999999994e-06 2024-09-13T14:59:47.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.071236+0000", 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7310000000000001e-06 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.085453+0000", 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014217855999999999 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.085571+0000", 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011804299999999999 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:47.085686+0000", 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114071 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T14:59:47.129 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 8 2024-09-13T14:59:47.130 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T14:59:47.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:47.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:47.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:47.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:47.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:47.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:47.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:47.576 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:47.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:47.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:47.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:47.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:47.841 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:47 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:47.842 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:47 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:47.928 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:47.951 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:47.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:47.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:47 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T14:59:48.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:48.075 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:48.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:48.201 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:48.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:48.327 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:48.429 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:48.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:48.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:48.575 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:48.647 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:48.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:48.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:48.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:48.824 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:48 smithi154 ceph-mon[107038]: pgmap v1005: 89 pgs: 1 active+remapped+backfilling, 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 58 active+clean; 798 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 228 op/s; 3210/38181 objects degraded (8.407%); 3621/38181 objects misplaced (9.484%); 5.4 MiB/s, 111 objects/s recovering 2024-09-13T14:59:48.824 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:48 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-09-13T14:59:48.824 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:48 smithi154 ceph-mon[107038]: osdmap e324: 8 total, 7 up, 7 in 2024-09-13T14:59:48.824 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:48 smithi154 ceph-mon[102314]: pgmap v1005: 89 pgs: 1 active+remapped+backfilling, 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 58 active+clean; 798 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 228 op/s; 3210/38181 objects degraded (8.407%); 3621/38181 objects misplaced (9.484%); 5.4 MiB/s, 111 objects/s recovering 2024-09-13T14:59:48.824 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:48 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-09-13T14:59:48.825 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:48 smithi154 ceph-mon[102314]: osdmap e324: 8 total, 7 up, 7 in 2024-09-13T14:59:48.837 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:48.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:48.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:48.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:48.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:48 smithi172 ceph-mon[100696]: pgmap v1005: 89 pgs: 1 active+remapped+backfilling, 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 58 active+clean; 798 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 228 op/s; 3210/38181 objects degraded (8.407%); 3621/38181 objects misplaced (9.484%); 5.4 MiB/s, 111 objects/s recovering 2024-09-13T14:59:48.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:48 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-09-13T14:59:48.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:48 smithi172 ceph-mon[100696]: osdmap e324: 8 total, 7 up, 7 in 2024-09-13T14:59:49.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:49.102 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:49.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:49.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:49.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:49.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:49.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:49.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:49.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:49.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:49 smithi172 ceph-mon[100696]: osdmap e325: 8 total, 7 up, 7 in 2024-09-13T14:59:49.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:49.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.788 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:49.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:49.850 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:49 smithi154 ceph-mon[107038]: osdmap e325: 8 total, 7 up, 7 in 2024-09-13T14:59:49.851 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:49 smithi154 ceph-mon[102314]: osdmap e325: 8 total, 7 up, 7 in 2024-09-13T14:59:49.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:49.912 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:49.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:50.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:50.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:50.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:50.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:50.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:50.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:50.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:50.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:50.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:50.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:50.614 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:50.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:50.731 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:50.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:50.877 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:50 smithi172 ceph-mon[100696]: pgmap v1008: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 754 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 347 op/s; 3210/36042 objects degraded (8.906%); 2808/36042 objects misplaced (7.791%); 8.6 MiB/s, 163 objects/s recovering 2024-09-13T14:59:50.877 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:50 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3210/36042 objects degraded (8.906%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:50.878 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:50 smithi154 ceph-mon[102314]: pgmap v1008: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 754 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 347 op/s; 3210/36042 objects degraded (8.906%); 2808/36042 objects misplaced (7.791%); 8.6 MiB/s, 163 objects/s recovering 2024-09-13T14:59:50.878 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:50 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3210/36042 objects degraded (8.906%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:50.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:50 smithi154 ceph-mon[107038]: pgmap v1008: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 754 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 347 op/s; 3210/36042 objects degraded (8.906%); 2808/36042 objects misplaced (7.791%); 8.6 MiB/s, 163 objects/s recovering 2024-09-13T14:59:50.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:50 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3210/36042 objects degraded (8.906%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:50.884 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:50.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:50.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:51.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:51.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:51.148 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:51.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:51.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:51.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:51.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:51.531 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:51.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:51.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:51.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:51.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:51.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:51.942 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:51.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:52.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:52.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:52.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:52.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:52.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 14:59:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:14:59:51] "GET /metrics HTTP/1.1" 200 36472 "" "Prometheus/2.51.0" 2024-09-13T14:59:52.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:52.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:52.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:52.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:52.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:52.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:52.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:52.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:52.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:52.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:52.863 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:52 smithi154 ceph-mon[102314]: pgmap v1009: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 537 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 2114/25644 objects degraded (8.244%); 2274/25644 objects misplaced (8.868%); 6.6 MiB/s, 124 objects/s recovering 2024-09-13T14:59:52.863 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:52 smithi154 ceph-mon[107038]: pgmap v1009: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 537 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 2114/25644 objects degraded (8.244%); 2274/25644 objects misplaced (8.868%); 6.6 MiB/s, 124 objects/s recovering 2024-09-13T14:59:52.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:52.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:52.951 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T14:59:52.951 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T14:59:52.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:52.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:52 smithi172 ceph-mon[100696]: pgmap v1009: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 537 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 2114/25644 objects degraded (8.244%); 2274/25644 objects misplaced (8.868%); 6.6 MiB/s, 124 objects/s recovering 2024-09-13T14:59:52.994 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T14:59:52.994 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T14:59:52.997 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:59:53.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:53.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:53.034 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T14:59:53.034 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T14:59:53.072 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T14:59:53.072 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T14:59:53.106 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T14:59:53.106 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T14:59:53.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:53.140 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T14:59:53.140 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T14:59:53.145 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:53.169 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T14:59:53.169 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T14:59:53.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:53.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:53.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:53.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:53.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:53.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:53.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:53.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:53.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2] live_osds: [1, 3, 4, 5, 6, 7, 0] 2024-09-13T14:59:53.586 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-13T14:59:53.587 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-09-13T14:59:53.587 DEBUG:teuthology.orchestra.run.smithi154:osd.2> sudo journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T14:59:53.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl start ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2 2024-09-13T14:59:53.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:53.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:53.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:53.788 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:53.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:53.903 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:53 smithi154 systemd[1]: Starting Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T14:59:53.903 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:53 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:53.903 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:53 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:53.904 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:53 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1877403733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:53.904 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:53 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:53.904 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:53 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:53.904 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:53 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1877403733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:53.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:53.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:53.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:53 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:53.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:53 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:53.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:53 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1877403733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T14:59:54.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:54.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:54.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:54.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:54.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:54.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:54.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:54.491 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:54.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:54.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T14:59:54.681 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T14:59:54.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:54.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:54.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:54.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:54.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:54.934 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:54 smithi154 ceph-mon[102314]: pgmap v1010: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 537 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 597 op/s; 2114/25644 objects degraded (8.244%); 2274/25644 objects misplaced (8.868%); 2.4 MiB/s, 51 objects/s recovering 2024-09-13T14:59:54.935 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:54 smithi154 ceph-mon[107038]: pgmap v1010: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 537 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 597 op/s; 2114/25644 objects degraded (8.244%); 2274/25644 objects misplaced (8.868%); 2.4 MiB/s, 51 objects/s recovering 2024-09-13T14:59:54.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:54 smithi172 ceph-mon[100696]: pgmap v1010: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 537 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 597 op/s; 2114/25644 objects degraded (8.244%); 2274/25644 objects misplaced (8.868%); 2.4 MiB/s, 51 objects/s recovering 2024-09-13T14:59:55.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:55.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:55.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:55.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:55.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:55.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:55.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:55.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:55.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:55.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:55.622 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T14:59:55.622 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T14:59:55.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:55.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:55.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:55.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:55.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:55.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2114/25644 objects degraded (8.244%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:55.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2114/25644 objects degraded (8.244%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:55.983 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:55.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:55.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2114/25644 objects degraded (8.244%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T14:59:56.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:56.108 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:56.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:56.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:56.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:56.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:56.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:56.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:56.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:56.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:56.648 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:56.680 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:56 smithi154 podman[256539]: 2024-09-13 14:59:56.31567101 +0000 UTC m=+0.270700989 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T14:59:56.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T14:59:56.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:56.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:56.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T14:59:56.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:56.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:56.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:56.942 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:56 smithi154 ceph-mon[102314]: pgmap v1011: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 339 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 1004/16152 objects degraded (6.216%); 1064/16152 objects misplaced (6.587%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:59:56.942 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:56 smithi154 ceph-mon[107038]: pgmap v1011: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 339 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 1004/16152 objects degraded (6.216%); 1064/16152 objects misplaced (6.587%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:59:56.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:56 smithi172 ceph-mon[100696]: pgmap v1011: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 339 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 1004/16152 objects degraded (6.216%); 1064/16152 objects misplaced (6.587%); 2.4 MiB/s, 38 objects/s recovering 2024-09-13T14:59:57.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:57.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:57.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:57.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.201 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:57.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:57.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:56 smithi154 podman[256539]: 2024-09-13 14:59:56.991759369 +0000 UTC m=+0.946789338 container create e8b98c8f2438a2cac7cfbca9e6ba8242b5a1ce4dfd47861e5607ce5a5144a7e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-13T14:59:57.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:57.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:57.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:57.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:57.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T14:59:57.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T14:59:57.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:57.743 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:57.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:57.869 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:57.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:57.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:57.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:58.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:58.124 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:58.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:58.250 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:58.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:58.377 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:58.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:58.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:58.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:58.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:58.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.789 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:58.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:58.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:58.914 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:58.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:58.942 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 podman[256539]: 2024-09-13 14:59:58.716221003 +0000 UTC m=+2.671250984 container init e8b98c8f2438a2cac7cfbca9e6ba8242b5a1ce4dfd47861e5607ce5a5144a7e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T14:59:58.942 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 podman[256539]: 2024-09-13 14:59:58.724181055 +0000 UTC m=+2.679211029 container start e8b98c8f2438a2cac7cfbca9e6ba8242b5a1ce4dfd47861e5607ce5a5144a7e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-13T14:59:58.942 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 podman[256539]: 2024-09-13 14:59:58.82453019 +0000 UTC m=+2.779560161 container attach e8b98c8f2438a2cac7cfbca9e6ba8242b5a1ce4dfd47861e5607ce5a5144a7e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T14:59:58.947 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:58.960 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T14:59:59.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T14:59:59.019 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T14:59:59.039 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T14:59:59.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:59.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:59.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[107038]: pgmap v1012: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 251 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 1004/11901 objects degraded (8.436%); 1064/11901 objects misplaced (8.940%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:59:59.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:59.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:59.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1837567579' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[102314]: pgmap v1012: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 251 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 1004/11901 objects degraded (8.436%); 1064/11901 objects misplaced (8.940%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:59:59.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:59.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:59.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1837567579' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.207 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.207 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 bash[256539]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.207 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.207 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 bash[256539]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.207 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:59:59.207 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 bash[256539]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:58 smithi154 bash[256539]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:59:59.208 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:59:59.209 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Removed 110261 objects 2024-09-13T14:59:59.209 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Clean up completed and total clean up time :202.3 2024-09-13T14:59:59.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:59.321 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T14:59:59.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T14:59:59.444 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate[256709]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:59:59.471 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 14:59:59 smithi154 bash[256539]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-13T14:59:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:58 smithi172 ceph-mon[100696]: pgmap v1012: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 251 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 1004/11901 objects degraded (8.436%); 1064/11901 objects misplaced (8.940%); 2.0 MiB/s, 31 objects/s recovering 2024-09-13T14:59:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T14:59:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T14:59:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1837567579' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T14:59:59.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T14:59:59.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T14:59:59.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T14:59:59.674 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-13T14:59:59.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-13T14:59:59.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T14:59:59.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T14:59:59.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T14:59:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T14:59:59.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T14:59:59.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T14:59:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T14:59:59.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:00.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:00.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:00.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:00.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:00.244 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:00.244 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:00.244 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[102314]: osdmap e326: 8 total, 7 up, 7 in 2024-09-13T15:00:00.244 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:00.245 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:00.245 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:00.245 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[107038]: osdmap e326: 8 total, 7 up, 7 in 2024-09-13T15:00:00.245 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 14:59:59 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:00.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:00.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:00.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:00.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:59 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:59 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:59 smithi172 ceph-mon[100696]: osdmap e326: 8 total, 7 up, 7 in 2024-09-13T15:00:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 14:59:59 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:00.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:00.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:00.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:00.684 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:00.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:00.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.810 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:00.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:00.875 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:00 smithi154 podman[256539]: 2024-09-13 15:00:00.565853682 +0000 UTC m=+4.520883656 container died e8b98c8f2438a2cac7cfbca9e6ba8242b5a1ce4dfd47861e5607ce5a5144a7e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:00:00.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:00.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:00.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:01.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:01.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:01.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:01.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:01.209 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pgmap v1014: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 197 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 1004/9312 objects degraded (10.782%); 493/9312 objects misplaced (5.294%) 2024-09-13T15:00:01.209 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized 2024-09-13T15:00:01.209 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-13T15:00:01.209 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: daemon osd.0 on smithi154 is in unknown state 2024-09-13T15:00:01.209 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T15:00:01.209 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: [WRN] OSD_DOWN: 1 osds down 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: osd.2 (root=default,host=smithi154) is down 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: [WRN] PG_DEGRADED: Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 3.4 is stuck undersized for 2m, current state active+undersized, last acting [1,5] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 3.5 is stuck undersized for 2m, current state active+undersized, last acting [5,3] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 3.1a is stuck undersized for 2m, current state active+undersized, last acting [4,1] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 4.4 is stuck undersized for 2m, current state active+undersized, last acting [5,3] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 4.d is stuck undersized for 2m, current state active+undersized, last acting [4,1] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 4.f is stuck undersized for 2m, current state active+undersized, last acting [3,4] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 4.1c is stuck undersized for 2m, current state active+undersized, last acting [1,3] 2024-09-13T15:00:01.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 7.2 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,1] 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 7.3 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,4] 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 7.a is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,5] 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[102314]: pg 7.e is active+undersized+degraded, acting [5,0] 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pgmap v1014: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 197 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 1004/9312 objects degraded (10.782%); 493/9312 objects misplaced (5.294%) 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: daemon osd.0 on smithi154 is in unknown state 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: [WRN] OSD_DOWN: 1 osds down 2024-09-13T15:00:01.211 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: osd.2 (root=default,host=smithi154) is down 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: [WRN] PG_DEGRADED: Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 3.4 is stuck undersized for 2m, current state active+undersized, last acting [1,5] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 3.5 is stuck undersized for 2m, current state active+undersized, last acting [5,3] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 3.1a is stuck undersized for 2m, current state active+undersized, last acting [4,1] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 4.4 is stuck undersized for 2m, current state active+undersized, last acting [5,3] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 4.d is stuck undersized for 2m, current state active+undersized, last acting [4,1] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 4.f is stuck undersized for 2m, current state active+undersized, last acting [3,4] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 4.1c is stuck undersized for 2m, current state active+undersized, last acting [1,3] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 7.2 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,1] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 7.3 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,4] 2024-09-13T15:00:01.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 7.a is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,5] 2024-09-13T15:00:01.213 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:00 smithi154 ceph-mon[107038]: pg 7.e is active+undersized+degraded, acting [5,0] 2024-09-13T15:00:01.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:01.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:01.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:01.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:01 smithi154 podman[257028]: 2024-09-13 15:00:01.207809716 +0000 UTC m=+1.960358966 container remove e8b98c8f2438a2cac7cfbca9e6ba8242b5a1ce4dfd47861e5607ce5a5144a7e7 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-13T15:00:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pgmap v1014: 89 pgs: 15 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 59 active+clean; 197 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 1004/9312 objects degraded (10.782%); 493/9312 objects misplaced (5.294%) 2024-09-13T15:00:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized 2024-09-13T15:00:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-13T15:00:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: daemon osd.0 on smithi154 is in unknown state 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: [WRN] OSD_DOWN: 1 osds down 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: osd.2 (root=default,host=smithi154) is down 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: [WRN] PG_DEGRADED: Degraded data redundancy: 1004/11901 objects degraded (8.436%), 6 pgs degraded, 10 pgs undersized 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 3.4 is stuck undersized for 2m, current state active+undersized, last acting [1,5] 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 3.5 is stuck undersized for 2m, current state active+undersized, last acting [5,3] 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 3.1a is stuck undersized for 2m, current state active+undersized, last acting [4,1] 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 4.4 is stuck undersized for 2m, current state active+undersized, last acting [5,3] 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 4.d is stuck undersized for 2m, current state active+undersized, last acting [4,1] 2024-09-13T15:00:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 4.f is stuck undersized for 2m, current state active+undersized, last acting [3,4] 2024-09-13T15:00:01.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 4.1c is stuck undersized for 2m, current state active+undersized, last acting [1,3] 2024-09-13T15:00:01.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 7.2 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,1] 2024-09-13T15:00:01.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 7.3 is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,4] 2024-09-13T15:00:01.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 7.a is stuck undersized for 2m, current state active+undersized+degraded, last acting [3,5] 2024-09-13T15:00:01.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:00 smithi172 ceph-mon[100696]: pg 7.e is active+undersized+degraded, acting [5,0] 2024-09-13T15:00:01.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:01.561 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:01.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:01.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:01.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:01.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:01.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:02.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:02.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:00:02.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:02.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:02.194 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:00:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:00:01] "GET /metrics HTTP/1.1" 200 36632 "" "Prometheus/2.51.0" 2024-09-13T15:00:02.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:02 smithi154 podman[257261]: 2024-09-13 15:00:01.968218252 +0000 UTC m=+0.346694956 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:00:02.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:02 smithi154 podman[257261]: 2024-09-13 15:00:02.105659507 +0000 UTC m=+0.484136204 container create 7bcbe6e813375b16d2cb5144dfbf05e1f770e37ee9bcab210bc8ea34b8e4672e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:00:02.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:02.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:02.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:02.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:02.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:02.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:02.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:02.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:02.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:02.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:02.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:02.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:02.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:02.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:03.002 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:02 smithi154 ceph-mon[102314]: pgmap v1015: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 5.3 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 3/120 objects degraded (2.500%); 20/120 objects misplaced (16.667%) 2024-09-13T15:00:03.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:02 smithi154 ceph-mon[107038]: pgmap v1015: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 5.3 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 3/120 objects degraded (2.500%); 20/120 objects misplaced (16.667%) 2024-09-13T15:00:03.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:03.087 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:03.189 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:03.215 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:03.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:03.317 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:03.318 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:03.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:03.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:03.467 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:03.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:03 smithi154 podman[257261]: 2024-09-13 15:00:03.10706107 +0000 UTC m=+1.485537770 container init 7bcbe6e813375b16d2cb5144dfbf05e1f770e37ee9bcab210bc8ea34b8e4672e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD) 2024-09-13T15:00:03.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:03 smithi154 podman[257261]: 2024-09-13 15:00:03.110941504 +0000 UTC m=+1.489418195 container start 7bcbe6e813375b16d2cb5144dfbf05e1f770e37ee9bcab210bc8ea34b8e4672e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-09-13T15:00:03.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:03 smithi154 ceph-osd[257434]: -- 172.21.15.154:0/1925481596 <== mon.2 v2:172.21.15.154:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c665185a40 con 0x55c665f54400 2024-09-13T15:00:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:02 smithi172 ceph-mon[100696]: pgmap v1015: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 5.3 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 3/120 objects degraded (2.500%); 20/120 objects misplaced (16.667%) 2024-09-13T15:00:03.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:03.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:03.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:00:03.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:03.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:03.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:03.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:03.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:03.970 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:03 smithi154 bash[257261]: 7bcbe6e813375b16d2cb5144dfbf05e1f770e37ee9bcab210bc8ea34b8e4672e 2024-09-13T15:00:03.970 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:03 smithi154 systemd[1]: Started Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T15:00:03.975 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:03.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:04.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:04.102 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:04.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:00:04.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:04.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:04.349 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:04.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:04.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:04.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:04.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:04.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:04.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:04.732 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:04.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:04.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:04.900 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[257425]: 2024-09-13T15:00:04.481+0000 7f6ff2f29640 -1 Falling back to public interface 2024-09-13T15:00:04.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:04.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:04.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:05.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:05.815 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:05 smithi154 ceph-mon[102314]: pgmap v1016: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 5.3 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 3/120 objects degraded (2.500%); 20/120 objects misplaced (16.667%) 2024-09-13T15:00:05.815 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3/120 objects degraded (2.500%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:05.816 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:05 smithi154 ceph-mon[107038]: pgmap v1016: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 5.3 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 3/120 objects degraded (2.500%); 20/120 objects misplaced (16.667%) 2024-09-13T15:00:05.816 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3/120 objects degraded (2.500%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:05.816 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:05.816 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:00:05.816 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:05.818 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:00:05.818 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:05.819 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:05 smithi172 ceph-mon[100696]: pgmap v1016: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 5.3 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 850 op/s; 3/120 objects degraded (2.500%); 20/120 objects misplaced (16.667%) 2024-09-13T15:00:05.819 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3/120 objects degraded (2.500%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:05.820 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:00:05.821 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:05.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:05.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:05.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:05.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:05.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:06.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:06.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:06.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:06.216 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:06.302 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/497399941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:06.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:06.320 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:00:06.320 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:00:06.320 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T15:00:06.321 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:00:06.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:06.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/497399941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:06.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/497399941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:06.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:06.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:06.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:06.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:06.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:06.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:06.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:06.838 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:06.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:06.859 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:06.882 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:06.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:00:06.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:07.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:07.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:07.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:07.144 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:07.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:07 smithi154 ceph-mon[102314]: pgmap v1017: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:07.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:07 smithi154 ceph-mon[107038]: pgmap v1017: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:07.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:07.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:07 smithi172 ceph-mon[100696]: pgmap v1017: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 535 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:07.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T15:00:07.280 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T15:00:07.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:07.374 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:07.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:07.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:07.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:07.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:07.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:07.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:07.794 INFO:teuthology.orchestra.run.smithi154.stderr:no valid command found; 10 closest matches: 2024-09-13T15:00:07.794 INFO:teuthology.orchestra.run.smithi154.stderr:0 2024-09-13T15:00:07.794 INFO:teuthology.orchestra.run.smithi154.stderr:1 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:2 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:abort 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:assert 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs debug_inject_read_zeros 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs files list 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs stats 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:bluestore bluefs device info [] 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:config diff 2024-09-13T15:00:07.795 INFO:teuthology.orchestra.run.smithi154.stderr:admin_socket: invalid command 2024-09-13T15:00:07.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:07.874 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:07.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:07.998 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:07.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:08.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.122 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:08.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:08.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:08.243 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:08.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:08.367 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:08.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:08.490 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T15:00:08.551 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-13T15:00:08.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:08.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:08.701 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:00:08.701 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:00:08.701 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:00:08.702 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:00:08.702 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:00:08.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:08.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:08.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:08.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:08.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:08.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:08.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:09.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:09.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:09.132 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:09 smithi154 ceph-mon[102314]: pgmap v1018: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 393 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:09.132 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:09 smithi154 ceph-mon[107038]: pgmap v1018: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 393 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:09.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:09.236 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:09.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:00:09.329 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:09 smithi172 ceph-mon[100696]: pgmap v1018: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 393 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:09.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:09.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:00:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:09.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:09.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:09.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:09.553 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:09.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:09.606 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:09.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:09.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:09.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:09.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:09.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:09.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:09.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:10.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:10.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:10.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:10.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:10.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:10.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:10.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:10.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:10.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:10.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:10.471 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:10.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:10.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:10.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:10.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:10.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:10.844 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:10.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:10.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:10.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:11.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:11.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:11.151 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:11 smithi154 ceph-mon[102314]: pgmap v1019: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 378 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:11.151 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:11 smithi154 ceph-mon[107038]: pgmap v1019: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 378 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:11.158 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:00:11.158 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:00:11.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110738 6.e 6:780e6055:::benchmark_data_smithi154_175640_object24962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.142686+0000", 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 474.01320489099999, 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.041401764, 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T15:00:11.159 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110738 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142686+0000", 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142686+0000", 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142685+0000", 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142690+0000", 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7790000000000003e-06 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142700+0000", 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0345e-05 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142707+0000", 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9650000000000002e-06 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142751+0000", 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4088999999999997e-05 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.142759+0000", 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.294e-06 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.172515+0000", 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.029756407 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.172632+0000", 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117419 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.173099+0000", 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046696500000000001 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179720+0000", 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066204250000000001 2024-09-13T15:00:11.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179777+0000", 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7538999999999999e-05 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.183999+0000", 2024-09-13T15:00:11.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042218719999999998 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.184046+0000", 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7150000000000001e-05 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.184088+0000", 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1390000000000002e-05 2024-09-13T15:00:11.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110739 6.e 6:78427618:::benchmark_data_smithi154_175640_object24963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.146731+0000", 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 474.00915964400002, 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0540792329999999, 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.167 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110739 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T15:00:11.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146731+0000", 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146734+0000", 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8319999999999999e-06 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146743+0000", 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4460000000000001e-06 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146749+0000", 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3790000000000003e-06 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146788+0000", 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9172999999999999e-05 2024-09-13T15:00:11.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.146795+0000", 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3289999999999999e-06 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.175941+0000", 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0291466950000001 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.176007+0000", 2024-09-13T15:00:11.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5673999999999995e-05 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.176071+0000", 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4245999999999994e-05 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.181605+0000", 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055336969999999997 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.194960+0000", 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013354428 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200751+0000", 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057913399999999999 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200782+0000", 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1278000000000001e-05 2024-09-13T15:00:11.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200811+0000", 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8384000000000001e-05 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110766 6.e 6:7c2939fb:::benchmark_data_smithi154_175640_object24990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.287081+0000", 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 473.86881024100001, 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92337370100000005, 2024-09-13T15:00:11.175 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110766 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287081+0000", 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287081+0000", 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287078+0000", 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287089+0000", 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0464e-05 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287110+0000", 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1403000000000002e-05 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287119+0000", 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2359999999999993e-06 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287185+0000", 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5074999999999996e-05 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.287193+0000", 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7730000000000005e-06 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.177878+0000", 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89068479700000003 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.177987+0000", 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108955 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.178102+0000", 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114571 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.186553+0000", 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084510200000000001 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200852+0000", 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014299335999999999 2024-09-13T15:00:11.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210402+0000", 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095498179999999998 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210431+0000", 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9479999999999999e-05 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210454+0000", 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3074e-05 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.44843.0:110767 6.e 6:7ea46229:::benchmark_data_smithi154_175640_object24991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:52:17.297777+0000", 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 473.85811422299997, 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91276756699999995, 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.183 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.44843", 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2942872735", 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110767 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297777+0000", 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297777+0000", 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297774+0000", 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297785+0000", 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0249e-05 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297802+0000", 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7652999999999999e-05 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297810+0000", 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9200000000000004e-06 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297866+0000", 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5358000000000001e-05 2024-09-13T15:00:11.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:17.297874+0000", 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3610000000000001e-06 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179816+0000", 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.88194239600000002 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.179923+0000", 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106557 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.180036+0000", 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000113286 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.191600+0000", 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011563982 2024-09-13T15:00:11.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.200934+0000", 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093336400000000007 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210492+0000", 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009557879 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210522+0000", 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0074e-05 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:52:18.210544+0000", 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2561e-05 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.190 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150521 7.8 7:11a94199:::benchmark_data_smithi154_225924_object40257:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.290904+0000", 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 159.86498733900001, 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.023286141, 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150521 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290904+0000", 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290904+0000", 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290902+0000", 2024-09-13T15:00:11.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290908+0000", 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9279999999999998e-06 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290919+0000", 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.132e-05 2024-09-13T15:00:11.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290925+0000", 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2220000000000001e-06 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290978+0000", 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3409000000000003e-05 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290986+0000", 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3659999999999992e-06 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310964+0000", 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0199773940000001 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314190+0000", 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032259120000000001 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.195 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150526 7.8 7:16bba6af:::benchmark_data_smithi154_225924_object40262:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.290939+0000", 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 159.86495226299999, 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0235627060000001, 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.196 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150526 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290939+0000", 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290939+0000", 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290938+0000", 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290940+0000", 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0020000000000001e-06 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290945+0000", 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7840000000000003e-06 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290949+0000", 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.518e-06 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.291003+0000", 2024-09-13T15:00:11.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4715e-05 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.291009+0000", 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1359999999999996e-06 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314426+0000", 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0234177849999999 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314501+0000", 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5047000000000002e-05 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210203 7.1 7:855d5ed3:::benchmark_data_smithi154_225924_object99939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.562662+0000", 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.593229014999999, 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0491108170000001, 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.201 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210203 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562662+0000", 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562662+0000", 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562661+0000", 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562666+0000", 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.023e-06 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562694+0000", 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8348e-05 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562702+0000", 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3300000000000001e-06 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.570904+0000", 2024-09-13T15:00:11.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082024420000000008 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.571040+0000", 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000135506 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1,3", 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.571167+0000", 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127704 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.574660+0000", 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003492878 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.576360+0000", 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017001539999999999 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.581800+0000", 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054399729999999999 2024-09-13T15:00:11.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.581814+0000", 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3392e-05 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.611724+0000", 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.029910165 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.611773+0000", 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8810999999999997e-05 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210228 7.1 7:8f77d107:::benchmark_data_smithi154_225924_object99964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.597289+0000", 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.558601441, 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0257111800000001, 2024-09-13T15:00:11.208 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210228 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597289+0000", 2024-09-13T15:00:11.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597289+0000", 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597288+0000", 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597296+0000", 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7209999999999994e-06 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597311+0000", 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5577e-05 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597319+0000", 2024-09-13T15:00:11.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0369999999999995e-06 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597361+0000", 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2259999999999999e-05 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597371+0000", 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3510000000000008e-06 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.612091+0000", 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0147197939999999 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.612177+0000", 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6237999999999997e-05 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.612273+0000", 2024-09-13T15:00:11.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5977999999999995e-05 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617796+0000", 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055232850000000002 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617836+0000", 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9480999999999997e-05 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.622948+0000", 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051121159999999999 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.622978+0000", 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0528999999999999e-05 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623001+0000", 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2422e-05 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.215 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210229 7.1 7:8b22e94d:::benchmark_data_smithi154_225924_object99965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.597333+0000", 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.558558311999999, 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.028199431, 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.216 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210229 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597333+0000", 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597333+0000", 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597332+0000", 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597335+0000", 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2430000000000001e-06 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597343+0000", 2024-09-13T15:00:11.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5750000000000003e-06 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597350+0000", 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8739999999999997e-06 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597390+0000", 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9935999999999998e-05 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597395+0000", 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0420000000000002e-06 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.615283+0000", 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0178883190000001 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.615370+0000", 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6768999999999995e-05 2024-09-13T15:00:11.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.615445+0000", 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5568000000000006e-05 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.619948+0000", 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045030319999999997 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.619965+0000", 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6731999999999998e-05 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625482+0000", 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055164360000000004 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625510+0000", 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8212999999999999e-05 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625532+0000", 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2183000000000001e-05 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210246 7.1 7:85e1b4c7:::benchmark_data_smithi154_225924_object99982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.613962+0000", 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.541928772999999, 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.011727381, 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.223 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210246 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613962+0000", 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613962+0000", 2024-09-13T15:00:11.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613961+0000", 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613967+0000", 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6039999999999998e-06 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613981+0000", 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3553000000000001e-05 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613987+0000", 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9050000000000002e-06 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.614024+0000", 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7338000000000001e-05 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.614031+0000", 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2400000000000004e-06 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617851+0000", 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.003820465 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617918+0000", 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6568000000000004e-05 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617981+0000", 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3125000000000003e-05 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623045+0000", 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050646149999999997 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625537+0000", 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024919159999999998 2024-09-13T15:00:11.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625618+0000", 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0315999999999995e-05 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625648+0000", 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0035e-05 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625690+0000", 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1974000000000002e-05 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210303 7.1 7:8502be6f:::benchmark_data_smithi154_225924_object100039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.669545+0000", 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.486346112, 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95965307700000002, 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210303 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669545+0000", 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669545+0000", 2024-09-13T15:00:11.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669543+0000", 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669551+0000", 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9289999999999998e-06 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669567+0000", 2024-09-13T15:00:11.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5787e-05 2024-09-13T15:00:11.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669576+0000", 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6289999999999999e-06 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669619+0000", 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3779000000000002e-05 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669627+0000", 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2660000000000004e-06 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.619977+0000", 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95034996500000002 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.620055+0000", 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7961999999999997e-05 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.620156+0000", 2024-09-13T15:00:11.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101539 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625574+0000", 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054177369999999997 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625698+0000", 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123786 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.629150+0000", 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003452685 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.629176+0000", 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5774000000000001e-05 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.629198+0000", 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1840000000000001e-05 2024-09-13T15:00:11.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210385 7.1 7:886f9cba:::benchmark_data_smithi154_225924_object100121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.754596+0000", 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.401295221000002, 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89118368199999998, 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.239 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210385 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754596+0000", 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754596+0000", 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754594+0000", 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754603+0000", 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8880000000000003e-06 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754621+0000", 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7768e-05 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754630+0000", 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.312e-06 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754681+0000", 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1175999999999999e-05 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754689+0000", 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3839999999999997e-06 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623069+0000", 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86837958500000001 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623165+0000", 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6273000000000005e-05 2024-09-13T15:00:11.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623278+0000", 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112574 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.631655+0000", 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083774060000000004 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.633250+0000", 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001594449 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645708+0000", 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012458545 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645740+0000", 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1436999999999997e-05 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645779+0000", 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9666000000000003e-05 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210391 7.1 7:89b8523a:::benchmark_data_smithi154_225924_object100127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.760314+0000", 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.395577066000001, 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.88564320399999996, 2024-09-13T15:00:11.246 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210391 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760314+0000", 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760314+0000", 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760312+0000", 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760320+0000", 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0709999999999995e-06 2024-09-13T15:00:11.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760336+0000", 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5668e-05 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760344+0000", 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.439e-06 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760412+0000", 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7853000000000002e-05 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760429+0000", 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7289999999999999e-05 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625724+0000", 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.865294118 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625795+0000", 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1068000000000005e-05 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625864+0000", 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9266999999999999e-05 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.633215+0000", 2024-09-13T15:00:11.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0073508699999999998 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645821+0000", 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012605893999999999 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645900+0000", 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299000000000001e-05 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645924+0000", 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4477999999999998e-05 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645957+0000", 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2577999999999999e-05 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:11.253 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:00:11.254 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:11 smithi172 ceph-mon[100696]: pgmap v1019: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 378 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:11.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:11.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.397 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:11.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:11.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:11.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:11.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:11.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:11.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:11.766 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:11.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:11.890 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:11.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:11.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:12.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:12.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:00:12.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:12.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:12.159 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:00:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:00:11] "GET /metrics HTTP/1.1" 200 36632 "" "Prometheus/2.51.0" 2024-09-13T15:00:12.221 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:12.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:12.260 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:12.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:12.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:12.461 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:12 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[257425]: 2024-09-13T15:00:12.276+0000 7f6ff2f29640 -1 osd.2 280 log_to_monitors true 2024-09-13T15:00:12.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:12.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:12.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:12.632 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:12.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:12.755 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:12.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:12.879 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:12.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:12.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:13.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:13.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.126 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:13.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:13.250 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:13.328 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:13 smithi172 ceph-mon[100696]: pgmap v1020: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:13.328 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:13 smithi172 ceph-mon[100696]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T15:00:13.328 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:13.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:13.373 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:13.470 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:00:13 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[257425]: 2024-09-13T15:00:13.078+0000 7f6fe6ccb640 -1 osd.2 327 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:00:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:13 smithi154 ceph-mon[102314]: pgmap v1020: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:13 smithi154 ceph-mon[102314]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T15:00:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:13 smithi154 ceph-mon[107038]: pgmap v1020: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 256 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:13 smithi154 ceph-mon[107038]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-13T15:00:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:13.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:13.496 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:13.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:00:13.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:13.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:13.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:13.743 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:13.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:13.848 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:00:13.848 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:00:13.848 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T15:00:13.848 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:00:13.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:13.945 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:13.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:13.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:13.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.037 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:00:14.050 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_2' removed 2024-09-13T15:00:14.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.110 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:14 smithi172 ceph-mon[100696]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T15:00:14.110 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:14 smithi172 ceph-mon[100696]: osdmap e327: 8 total, 7 up, 7 in 2024-09-13T15:00:14.110 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:14 smithi172 ceph-mon[100696]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T15:00:14.110 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1159518039' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:14.110 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/981293594' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:14.110 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:14 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.128 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:14.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:14.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:14.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:00:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[102314]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T15:00:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[102314]: osdmap e327: 8 total, 7 up, 7 in 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[102314]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1159518039' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/981293594' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[107038]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[107038]: osdmap e327: 8 total, 7 up, 7 in 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[107038]: from='osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi154", "root=default"]} : dispatch 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1159518039' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/981293594' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:14 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:00:14.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:14.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:14.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.630 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.662 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:14.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.754 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:14.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:14.855 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:14.877 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:14.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:14.926 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T15:00:14.927 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T15:00:14.927 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-13T15:00:14.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-13T15:00:14.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:14.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:15.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:15.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:15.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: pgmap v1022: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:15.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:15.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T15:00:15.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1159518039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094] boot 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: osdmap e328: 8 total, 8 up, 7 in 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[102314]: osdmap e329: 8 total, 8 up, 7 in 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: pgmap v1022: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1159518039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094] boot 2024-09-13T15:00:15.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: osdmap e328: 8 total, 8 up, 7 in 2024-09-13T15:00:15.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T15:00:15.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:15 smithi154 ceph-mon[107038]: osdmap e329: 8 total, 8 up, 7 in 2024-09-13T15:00:15.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:15.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:15.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:15.373 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:15.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:15.484 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: pgmap v1022: 89 pgs: 15 active+clean+remapped, 10 active+undersized, 5 active+undersized+degraded, 59 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 1 op/s; 3/84 objects degraded (3.571%); 8/84 objects misplaced (9.524%) 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1159518039' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: osd.2 [v2:172.21.15.154:6818/2560992094,v1:172.21.15.154:6819/2560992094] boot 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: osdmap e328: 8 total, 8 up, 7 in 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-13T15:00:15.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:15 smithi172 ceph-mon[100696]: osdmap e329: 8 total, 8 up, 7 in 2024-09-13T15:00:15.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:15.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:15.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:15.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:15.723 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:15.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:15.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:15.869 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:15.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:15.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:15.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:16.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:16.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:16.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:16.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:16.272 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:00:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:00:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:00:16.272 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:00:16.272 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:00:16.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:16.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:16.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:16.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:16.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:16.617 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:16.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:16.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:16.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:16.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:16.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:00:16.965 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:16 smithi154 ceph-mon[102314]: pgmap v1025: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 3 active+undersized, 2 active+undersized+degraded, 53 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T15:00:16.965 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:16 smithi154 ceph-mon[102314]: osdmap e330: 8 total, 8 up, 7 in 2024-09-13T15:00:16.965 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:16 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:16.965 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:16 smithi154 ceph-mon[107038]: pgmap v1025: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 3 active+undersized, 2 active+undersized+degraded, 53 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T15:00:16.966 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:16 smithi154 ceph-mon[107038]: osdmap e330: 8 total, 8 up, 7 in 2024-09-13T15:00:16.966 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:16 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:16.971 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:16.997 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:16.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:17.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:17.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:17.133 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:17.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:17.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:16 smithi172 ceph-mon[100696]: pgmap v1025: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 3 active+undersized, 2 active+undersized+degraded, 53 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 3/84 objects degraded (3.571%) 2024-09-13T15:00:17.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:16 smithi172 ceph-mon[100696]: osdmap e330: 8 total, 8 up, 7 in 2024-09-13T15:00:17.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:16 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-13T15:00:17.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:17.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.369 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:17.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:17.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:17.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:17.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:17.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:17.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:17.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:17.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:17.863 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:17.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:17.986 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:17.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:18.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:18.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:18.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:18.235 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:18.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:18.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:18.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:18.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:18.523 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:18.524 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:18.534 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:00:18.534 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:18.558 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:00:18.558 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:18.580 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:00:18.580 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T15:00:18.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:18.597 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:00:18.597 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:18.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:18.613 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:00:18.613 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:18.629 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:00:18.630 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:18.646 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:00:18.646 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:18.663 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:00:18.663 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:18.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:18.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:18.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:18.844 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:18.844 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:18.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:18.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:18.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:18.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:19.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:19.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:19.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:19.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[102314]: pgmap v1027: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 2 active+undersized, 1 active+undersized+degraded, 55 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2305922014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[107038]: pgmap v1027: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 2 active+undersized, 1 active+undersized+degraded, 55 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2305922014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:19.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:19.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:19.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:19.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:18 smithi172 ceph-mon[100696]: pgmap v1027: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 2 active+undersized, 1 active+undersized+degraded, 55 active+clean; 4.6 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1/84 objects degraded (1.190%) 2024-09-13T15:00:19.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:18 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:18 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2305922014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:19.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:18 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:00:19.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:18 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:00:19.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:18 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:19.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:19.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:19.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:19.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:00:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:19.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:19.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:19.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:19.668 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:19.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:00:19.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:19.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:19.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:19.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:19.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:19.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:19.956 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:19 smithi154 ceph-mon[102314]: pgmap v1028: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:19.956 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:19 smithi154 ceph-mon[102314]: pgmap v1029: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:19.956 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:19 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded, 2 pgs undersized) 2024-09-13T15:00:19.957 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:19 smithi154 ceph-mon[107038]: pgmap v1028: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:19.957 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:19 smithi154 ceph-mon[107038]: pgmap v1029: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:19.957 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:19 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded, 2 pgs undersized) 2024-09-13T15:00:19.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:19.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:20.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:20.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:20.117 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:19 smithi172 ceph-mon[100696]: pgmap v1028: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:20.117 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:19 smithi172 ceph-mon[100696]: pgmap v1029: 73 pgs: 1 remapped, 7 peering, 7 remapped+peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:20.117 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:19 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/84 objects degraded (1.190%), 1 pg degraded, 2 pgs undersized) 2024-09-13T15:00:20.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:20.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:20.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.343 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:20.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:20.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:20.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:20.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:20.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:20.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.713 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:20.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:20.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:20.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:20.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:20.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:20.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:21.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:21.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:21.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:21.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:21.258 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:21.258 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:21.258 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:21.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:21.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:21.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:21.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:21.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:21.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:21.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:21.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:21.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:21.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:21.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:21.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:00:21.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:21.965 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:21.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:22.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:22.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:22.104 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:22.157 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:21 smithi154 ceph-mon[102314]: pgmap v1030: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:22.158 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:21 smithi154 ceph-mon[107038]: pgmap v1030: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:22.158 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:00:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:00:21] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.51.0" 2024-09-13T15:00:22.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.212 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:22.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:22.224 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:21 smithi172 ceph-mon[100696]: pgmap v1030: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:22.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:22.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:22.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:22.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:22.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:22.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:22.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:22.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:22.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:22.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:22.846 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:22.846 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:22.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:22.961 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_3' created 2024-09-13T15:00:22.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:22.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:22.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:23.058 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:22 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2384903564' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T15:00:23.058 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:22 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2384903564' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T15:00:23.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:23.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:23.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:23.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:00:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:23.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:22 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2384903564' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-13T15:00:23.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:23.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:23.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:23.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:23.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:23.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:23.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:23.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:23.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:23.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-13T15:00:23.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:23.930 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:23.931 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:23.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:23.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:23.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:24.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:24.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:24.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:24.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:24.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:23 smithi154 ceph-mon[107038]: pgmap v1031: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:24.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:23 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2384903564' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T15:00:24.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:23 smithi154 ceph-mon[107038]: osdmap e331: 8 total, 8 up, 7 in 2024-09-13T15:00:24.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:23 smithi154 ceph-mon[102314]: pgmap v1031: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:24.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:23 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2384903564' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T15:00:24.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:23 smithi154 ceph-mon[102314]: osdmap e331: 8 total, 8 up, 7 in 2024-09-13T15:00:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:23 smithi172 ceph-mon[100696]: pgmap v1031: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:23 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2384903564' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-13T15:00:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:23 smithi172 ceph-mon[100696]: osdmap e331: 8 total, 8 up, 7 in 2024-09-13T15:00:24.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T15:00:24.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:24.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:24.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:24.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:24.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:24.577 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:24.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:00:24.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:24.706 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:24.720 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:24.752 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:24.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:24.829 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:24.843 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:24.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:24.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:24.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:25.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:25.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:25.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:24 smithi154 ceph-mon[102314]: osdmap e332: 8 total, 8 up, 7 in 2024-09-13T15:00:25.159 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:24 smithi154 ceph-mon[107038]: osdmap e332: 8 total, 8 up, 7 in 2024-09-13T15:00:25.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:25.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:25.221 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:24 smithi172 ceph-mon[100696]: osdmap e332: 8 total, 8 up, 7 in 2024-09-13T15:00:25.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:25.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:25.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:25.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:25.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:25.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:25.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:25.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:25.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:25.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:25.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:25.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:25.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:26.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:26.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:26.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:26.192 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:25 smithi172 ceph-mon[100696]: pgmap v1034: 89 pgs: 14 unknown, 75 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:25 smithi172 ceph-mon[100696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:00:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:25 smithi172 ceph-mon[100696]: osdmap e333: 8 total, 8 up, 7 in 2024-09-13T15:00:26.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:26.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:26.380 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:26.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.444 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:26.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:25 smithi154 ceph-mon[102314]: pgmap v1034: 89 pgs: 14 unknown, 75 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:26.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:25 smithi154 ceph-mon[102314]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:00:26.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:25 smithi154 ceph-mon[102314]: osdmap e333: 8 total, 8 up, 7 in 2024-09-13T15:00:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:25 smithi154 ceph-mon[107038]: pgmap v1034: 89 pgs: 14 unknown, 75 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:25 smithi154 ceph-mon[107038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:00:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:25 smithi154 ceph-mon[107038]: osdmap e333: 8 total, 8 up, 7 in 2024-09-13T15:00:26.481 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:26.481 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:26.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.569 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:26.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:26.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:26.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:26.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:26.816 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:26.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:26.940 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:26.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:27.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.065 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:00:27.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:27.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:27.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:27.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.329 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:27.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:27.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:27.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:27.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:27.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:27.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:27.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:27.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:27.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:28.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:28.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:28.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:28.190 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:27 smithi172 ceph-mon[100696]: pgmap v1036: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:28.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:28.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:28.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:28.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:27 smithi154 ceph-mon[102314]: pgmap v1036: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:27 smithi154 ceph-mon[107038]: pgmap v1036: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:28.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:28.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:28.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.688 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:28.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:28.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:28.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:28.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:28.934 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:28.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:28.988 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T15:00:29.004 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:29.004 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:29.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:29.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:29.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:29.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:28 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/909896338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:29.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:28 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/909896338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:28 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/909896338' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:00:29.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:29.304 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:29.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:29.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:29.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:29.553 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:29.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:00:29.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.677 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:29.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:29.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:29.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:29.844 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:29.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:29.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:29.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:30.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.171 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:30.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:29 smithi172 ceph-mon[100696]: pgmap v1037: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:29 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/909896338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:00:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:29 smithi172 ceph-mon[100696]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T15:00:30.263 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:29 smithi154 ceph-mon[107038]: pgmap v1037: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:30.263 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:29 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/909896338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:00:30.263 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:29 smithi154 ceph-mon[107038]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T15:00:30.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:29 smithi154 ceph-mon[102314]: pgmap v1037: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:30.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:29 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/909896338' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:00:30.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:29 smithi154 ceph-mon[102314]: osdmap e334: 8 total, 8 up, 7 in 2024-09-13T15:00:30.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:30.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T15:00:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:30.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:30.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:30.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:30.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:30.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:30.668 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:30.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.793 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:30.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:30.893 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:00:30.894 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:30.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:30.904 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:00:30.904 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:30.918 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:30.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:30.920 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:00:30.920 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:30.936 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:00:30.936 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:30.952 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:00:30.952 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:30.968 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:00:30.968 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:30.984 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:00:30.984 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:31.001 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:00:31.001 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:31.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:31.140 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:30 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3830925681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:31.140 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:30 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3830925681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:31.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:31.159 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:30 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3830925681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:31.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:31.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:31.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:31.382 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:00:31.382 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:00:31.382 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:00:31.382 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:00:31.382 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:00:31.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:31.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:31.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:31.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:31.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:31.658 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:31.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:31.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:31.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:31.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:31.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:31.995 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:00:32.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:32.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:32.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:00:32.076 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:32 smithi154 ceph-mon[102314]: pgmap v1039: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:32.076 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2113712671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:32.077 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:32 smithi154 ceph-mon[107038]: pgmap v1039: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:32.077 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2113712671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:32.077 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:00:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:00:31] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T15:00:32.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:32.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:32.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:32.257 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:00:32.279 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:32.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:32 smithi172 ceph-mon[100696]: pgmap v1039: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:32.377 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2113712671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:00:32.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:32.402 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:32.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:32.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:32.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:32.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:32.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:32.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:32.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:32.814 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:32.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:32.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:32.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:32.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:33.016 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-13T15:00:33.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:33.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:33.146 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:33.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:33.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.302 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:33 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:33.302 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2113712671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:33.303 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:33 smithi172 ceph-mon[100696]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T15:00:33.303 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1092986892' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:00:33.303 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:33 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:00:33.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:33.393 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:33.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:33.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2113712671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:33.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[102314]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T15:00:33.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1092986892' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:00:33.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:00:33.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:00:33.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2113712671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:00:33.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[107038]: osdmap e335: 8 total, 8 up, 7 in 2024-09-13T15:00:33.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1092986892' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:00:33.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:33 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:00:33.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:33.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:33.619 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:33.640 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:33.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:33.763 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:33.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:00:33.828 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193505 7.5 7:a65a0552:::benchmark_data_smithi154_225924_object83241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.486000+0000", 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.33647058099999, 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0470585349999999, 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193505 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.486000+0000", 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.486000+0000", 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.485999+0000", 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.486005+0000", 2024-09-13T15:00:33.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.486018+0000", 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2822e-05 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.486026+0000", 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1729999999999995e-06 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.486069+0000", 2024-09-13T15:00:33.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2388999999999997e-05 2024-09-13T15:00:33.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.486077+0000", 2024-09-13T15:00:33.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0979999999999994e-06 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.517079+0000", 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0310020360000001 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.517178+0000", 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9475000000000006e-05 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.517256+0000", 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7218999999999994e-05 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.522939+0000", 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056831599999999996 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.526377+0000", 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034384289999999998 2024-09-13T15:00:33.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.532978+0000", 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066004929999999998 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.533005+0000", 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7348999999999999e-05 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.533059+0000", 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4005999999999998e-05 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193514 7.5 7:ac35c364:::benchmark_data_smithi154_225924_object83250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.492926+0000", 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.329545188, 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045612542, 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.837 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193514 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.492926+0000", 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.492926+0000", 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.492924+0000", 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.492931+0000", 2024-09-13T15:00:33.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6930000000000001e-06 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.492947+0000", 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6311e-05 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.492955+0000", 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8299999999999996e-06 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.493001+0000", 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6504000000000001e-05 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.493011+0000", 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1759999999999999e-06 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.519604+0000", 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0265936040000001 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.519698+0000", 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3262000000000003e-05 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.519769+0000", 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1823999999999995e-05 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.526331+0000", 2024-09-13T15:00:33.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065621469999999999 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538239+0000", 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011907522 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538393+0000", 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015444300000000001 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538440+0000", 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6944999999999999e-05 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538538+0000", 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7952e-05 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.844 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193549 7.5 7:adf92108:::benchmark_data_smithi154_225924_object83285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.525403+0000", 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.29706843700001, 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.013353382, 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193549 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525403+0000", 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525403+0000", 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525401+0000", 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525408+0000", 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8859999999999998e-06 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525423+0000", 2024-09-13T15:00:33.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5594e-05 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525430+0000", 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9909999999999996e-06 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525487+0000", 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6208000000000001e-05 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.525493+0000", 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8190000000000003e-06 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.521389+0000", 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99589534700000004 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.521455+0000", 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5924000000000001e-05 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.521526+0000", 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1283000000000002e-05 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.528173+0000", 2024-09-13T15:00:33.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066466900000000002 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538362+0000", 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01018939 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538604+0000", 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024181499999999999 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538647+0000", 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2639999999999998e-05 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538756+0000", 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109366 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193586 7.5 7:a044e91a:::benchmark_data_smithi154_225924_object83322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.852 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.564896+0000", 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.257575301, 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98211051400000005, 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193586 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564896+0000", 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564896+0000", 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564894+0000", 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564900+0000", 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4779999999999999e-06 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564911+0000", 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1420000000000001e-05 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564919+0000", 2024-09-13T15:00:33.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3869999999999997e-06 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564968+0000", 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9400999999999997e-05 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.564979+0000", 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1122e-05 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.523021+0000", 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95804227099999995 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.523099+0000", 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7558999999999994e-05 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.523165+0000", 2024-09-13T15:00:33.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6048000000000002e-05 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.528180+0000", 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050152520000000004 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.546786+0000", 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018605245999999999 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.546897+0000", 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011184999999999999 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.546936+0000", 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8343999999999997e-05 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.547006+0000", 2024-09-13T15:00:33.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0493000000000004e-05 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193603 7.5 7:a4729de7:::benchmark_data_smithi154_225924_object83339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.580938+0000", 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.241532613, 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96626124300000005, 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193603 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.580938+0000", 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.580938+0000", 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.580937+0000", 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.580943+0000", 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1550000000000004e-06 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.580955+0000", 2024-09-13T15:00:33.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1977000000000001e-05 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.580962+0000", 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1890000000000001e-06 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581018+0000", 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6288e-05 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581029+0000", 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0964e-05 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.524677+0000", 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94364729800000002 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.524796+0000", 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119059 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.524909+0000", 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011281900000000001 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.531461+0000", 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065524720000000002 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.546847+0000", 2024-09-13T15:00:33.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015385797 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.547055+0000", 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020861400000000001 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.547104+0000", 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8531e-05 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.547200+0000", 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5649000000000005e-05 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193604 7.5 7:a3e972ca:::benchmark_data_smithi154_225924_object83340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.581337+0000", 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.241134152, 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98418087700000001, 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.867 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193604 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581337+0000", 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581337+0000", 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581336+0000", 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581340+0000", 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6229999999999999e-06 2024-09-13T15:00:33.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581348+0000", 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5369999999999997e-06 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581353+0000", 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9579999999999998e-06 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581388+0000", 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5076000000000001e-05 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.581397+0000", 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.037e-06 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.526411+0000", 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94501359399999996 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.526524+0000", 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112917 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.526638+0000", 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114047 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.534784+0000", 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081465050000000001 2024-09-13T15:00:33.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.547237+0000", 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012452349999999999 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.565405+0000", 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018168601999999999 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:33.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.565435+0000", 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9839999999999999e-05 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.565518+0000", 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.242e-05 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.876 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193617 7.5 7:ac0a8214:::benchmark_data_smithi154_225924_object83353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.600412+0000", 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.22205898799999, 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96525039999999995, 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193617 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600412+0000", 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600412+0000", 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600410+0000", 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:00:33.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600416+0000", 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0329999999999997e-06 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600432+0000", 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5900999999999999e-05 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600441+0000", 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8920000000000006e-06 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600487+0000", 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6267999999999998e-05 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.600497+0000", 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3379999999999994e-06 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.528215+0000", 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92771821799999998 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.528300+0000", 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4660999999999995e-05 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.528365+0000", 2024-09-13T15:00:33.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5310999999999999e-05 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.534793+0000", 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064278720000000003 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.547271+0000", 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012478652999999999 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.565563+0000", 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018291541000000001 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.565596+0000", 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3349999999999997e-05 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.565662+0000", 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6054000000000001e-05 2024-09-13T15:00:33.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:193630 7.5 7:a8760431:::benchmark_data_smithi154_225924_object83366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:23.619586+0000", 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.202884496, 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95001361299999998, 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.884 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 193630 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619586+0000", 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619586+0000", 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619584+0000", 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999971 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619615+0000", 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1806e-05 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619649+0000", 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3606000000000003e-05 2024-09-13T15:00:33.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619668+0000", 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8709999999999999e-05 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619780+0000", 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011275799999999999 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:23.619787+0000", 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.669e-06 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.529795+0000", 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.91000769299999995 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.529905+0000", 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110654 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.530021+0000", 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115852 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.538196+0000", 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0081749300000000004 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.551261+0000", 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013064922 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.569479+0000", 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018218107000000001 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.569507+0000", 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7991999999999999e-05 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:24.569600+0000", 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2800999999999995e-05 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199569 7.8 7:12dbea7a:::benchmark_data_smithi154_225924_object89305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.657564+0000", 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.16490722899999, 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99737976800000006, 2024-09-13T15:00:33.891 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199569 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657564+0000", 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657564+0000", 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657562+0000", 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657570+0000", 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0499999999999992e-06 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657586+0000", 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5905000000000001e-05 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657594+0000", 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0450000000000001e-06 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657633+0000", 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8779999999999998e-05 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.657640+0000", 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5349999999999999e-06 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.645993+0000", 2024-09-13T15:00:33.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98835311100000001 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.646084+0000", 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0852000000000004e-05 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.646165+0000", 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0728000000000004e-05 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.654704+0000", 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085389939999999994 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.654749+0000", 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5000000000000003e-05 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.654794+0000", 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4892000000000003e-05 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.654843+0000", 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8874e-05 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.654944+0000", 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100973 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199574 7.8 7:1320529c:::benchmark_data_smithi154_225924_object89310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.671523+0000", 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.15094820900001, 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98946841699999999, 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199574 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671523+0000", 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671523+0000", 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671520+0000", 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999976 2024-09-13T15:00:33.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671533+0000", 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2279999999999999e-05 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671563+0000", 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0533000000000001e-05 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671583+0000", 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0169e-05 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671700+0000", 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011713399999999999 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.671725+0000", 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4803e-05 2024-09-13T15:00:33.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.649045+0000", 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97731936799999997 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.649131+0000", 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6633999999999998e-05 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.649259+0000", 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012773799999999999 2024-09-13T15:00:33.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.654987+0000", 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0057283769999999998 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.655071+0000", 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3646000000000005e-05 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.660895+0000", 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058238029999999998 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.660925+0000", 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0497999999999998e-05 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.660991+0000", 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5961000000000001e-05 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199598 7.8 7:1875d6cb:::benchmark_data_smithi154_225924_object89334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.700919+0000", 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.12155229, 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96027941000000006, 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199598 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.906 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700919+0000", 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700919+0000", 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700917+0000", 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700923+0000", 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.049e-06 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700936+0000", 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3174e-05 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700943+0000", 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7560000000000002e-06 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700988+0000", 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4786000000000001e-05 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.700999+0000", 2024-09-13T15:00:33.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0691e-05 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.651938+0000", 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95093896499999997 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.652054+0000", 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116278 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.652186+0000", 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013258200000000001 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.660801+0000", 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086141169999999993 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.660857+0000", 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6790999999999999e-05 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.661044+0000", 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018650200000000001 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.661091+0000", 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7611999999999997e-05 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.661198+0000", 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106686 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199604 7.8 7:1c702bae:::benchmark_data_smithi154_225924_object89340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.709110+0000", 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.113360641, 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95817510800000005, 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.913 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199604 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709110+0000", 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709110+0000", 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709109+0000", 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709115+0000", 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4779999999999999e-06 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709126+0000", 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1629e-05 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709132+0000", 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4990000000000002e-06 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709167+0000", 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4836000000000003e-05 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.709175+0000", 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9160000000000001e-06 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.655091+0000", 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94591608900000002 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.655231+0000", 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014068500000000001 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.655361+0000", 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129987 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.660999+0000", 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005637993 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667046+0000", 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060467289999999998 2024-09-13T15:00:33.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667084+0000", 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7815999999999998e-05 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667119+0000", 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5067999999999997e-05 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667285+0000", 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016656099999999999 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199611 7.8 7:126ba9fd:::benchmark_data_smithi154_225924_object89347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.713547+0000", 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.108923698, 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96836632300000003, 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199611 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713547+0000", 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713547+0000", 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713546+0000", 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713552+0000", 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4090000000000002e-06 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713564+0000", 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2376e-05 2024-09-13T15:00:33.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713569+0000", 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5609999999999998e-06 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713617+0000", 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7416999999999997e-05 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713627+0000", 2024-09-13T15:00:33.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0572e-05 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.657964+0000", 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94433611699999997 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.658084+0000", 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120919 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.658204+0000", 2024-09-13T15:00:33.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119956 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.664251+0000", 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060462349999999996 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667335+0000", 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003084697 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.681821+0000", 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014485431999999999 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.681852+0000", 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1343000000000001e-05 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.681914+0000", 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1478999999999999e-05 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199612 7.8 7:1fb87a3c:::benchmark_data_smithi154_225924_object89348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.713682+0000", 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.108789113, 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96839723600000005, 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199612 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713682+0000", 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713682+0000", 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713681+0000", 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713685+0000", 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4240000000000002e-06 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713692+0000", 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9199999999999998e-06 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713698+0000", 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9290000000000003e-06 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713741+0000", 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3171999999999999e-05 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.713749+0000", 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7199999999999995e-06 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.661235+0000", 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94748602199999998 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.661385+0000", 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149896 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.661515+0000", 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012981700000000001 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.673095+0000", 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011579842 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.681781+0000", 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086854989999999993 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.681958+0000", 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017761400000000001 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.681987+0000", 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9128999999999998e-05 2024-09-13T15:00:33.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682079+0000", 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1836999999999995e-05 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199623 7.8 7:1f6c8901:::benchmark_data_smithi154_225924_object89359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.733030+0000", 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.08944072600001, 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94931194500000005, 2024-09-13T15:00:33.933 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199623 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733030+0000", 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733030+0000", 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733029+0000", 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733035+0000", 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7240000000000003e-06 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733049+0000", 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3414999999999999e-05 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733056+0000", 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1799999999999999e-06 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733106+0000", 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9987000000000001e-05 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.733116+0000", 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0479000000000001e-05 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.664296+0000", 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.93117950100000002 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.664440+0000", 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014437900000000001 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.664575+0000", 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013435699999999999 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682103+0000", 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017528264000000002 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682162+0000", 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8764000000000002e-05 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682200+0000", 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8826999999999999e-05 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682246+0000", 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5955000000000003e-05 2024-09-13T15:00:33.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682342+0000", 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5791999999999994e-05 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:199628 7.8 7:166f517a:::benchmark_data_smithi154_225924_object89364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:32.739166+0000", 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 121.083305379, 2024-09-13T15:00:33.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95163799999999998, 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199628 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739166+0000", 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739166+0000", 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739164+0000", 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739169+0000", 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8230000000000002e-06 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739178+0000", 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7860000000000002e-06 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739182+0000", 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3640000000000001e-06 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739213+0000", 2024-09-13T15:00:33.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0718e-05 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:32.739219+0000", 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.083e-06 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667361+0000", 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92814228600000004 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667523+0000", 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016174700000000001 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,4", 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.667645+0000", 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012167600000000001 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682109+0000", 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014464110000000001 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.682383+0000", 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027370999999999998 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.690653+0000", 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082698070000000005 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.690707+0000", 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4627999999999999e-05 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:33.690804+0000", 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6432000000000001e-05 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:206506 7.5 7:a5b1404c:::benchmark_data_smithi154_225924_object96242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:42.691028+0000", 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.131443300999997, 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98111440400000005, 2024-09-13T15:00:33.947 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 206506 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691028+0000", 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691028+0000", 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691026+0000", 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691033+0000", 2024-09-13T15:00:33.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0759999999999997e-06 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691049+0000", 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5622000000000001e-05 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691056+0000", 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0380000000000002e-06 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691124+0000", 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7541999999999998e-05 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.691135+0000", 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1221e-05 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.612355+0000", 2024-09-13T15:00:33.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92122004999999996 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.612499+0000", 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144406 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.612637+0000", 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013757999999999999 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.619303+0000", 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066659010000000001 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.619311+0000", 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0509999999999997e-06 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.672042+0000", 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.052731306999999998 2024-09-13T15:00:33.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.672075+0000", 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3402999999999999e-05 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.672142+0000", 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6657999999999998e-05 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:206509 7.5 7:afd7fd45:::benchmark_data_smithi154_225924_object96245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:42.695148+0000", 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.127323335, 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97720187800000002, 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 206509 2024-09-13T15:00:33.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695148+0000", 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695148+0000", 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695147+0000", 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695152+0000", 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5180000000000004e-06 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695161+0000", 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5040000000000002e-06 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695165+0000", 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5059999999999997e-06 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695203+0000", 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7599999999999999e-05 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695211+0000", 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9009999999999994e-06 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.615304+0000", 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92009352700000002 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.615380+0000", 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5809999999999994e-05 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.615457+0000", 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6489000000000006e-05 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.619314+0000", 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038570169999999999 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.629082+0000", 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097682150000000002 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.672188+0000", 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043106406999999999 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.672246+0000", 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7527e-05 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.672350+0000", 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103788 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:206510 7.5 7:a783c789:::benchmark_data_smithi154_225924_object96246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:42.695965+0000", 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.126506456000001, 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98508895299999999, 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.962 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 206510 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695965+0000", 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695965+0000", 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695964+0000", 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695967+0000", 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5429999999999998e-06 2024-09-13T15:00:33.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695975+0000", 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5460000000000003e-06 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.695979+0000", 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0219999999999998e-06 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.696008+0000", 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9264999999999998e-05 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.696017+0000", 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2260000000000005e-06 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.617472+0000", 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92145509699999995 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.617546+0000", 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4648999999999996e-05 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.617621+0000", 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5165000000000003e-05 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.621017+0000", 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033958119999999998 2024-09-13T15:00:33.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.629112+0000", 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080947439999999992 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.680898+0000", 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.051786130999999999 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.680952+0000", 2024-09-13T15:00:33.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3511999999999999e-05 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.681053+0000", 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101869 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:206516 7.5 7:ac1f010a:::benchmark_data_smithi154_225924_object96252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:00:33.969 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:42.711335+0000", 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 51.111135787999999, 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97613953600000003, 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 206516 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711335+0000", 2024-09-13T15:00:33.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711335+0000", 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711334+0000", 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711341+0000", 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9469999999999997e-06 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711354+0000", 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3805000000000001e-05 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711362+0000", 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.858e-06 2024-09-13T15:00:33.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711417+0000", 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4494999999999999e-05 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.711429+0000", 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2751e-05 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.619367+0000", 2024-09-13T15:00:33.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.90793784499999997 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.619433+0000", 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5981000000000004e-05 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.619503+0000", 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0156999999999999e-05 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.621121+0000", 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016175149999999999 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.629139+0000", 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080184330000000002 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:00:33.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.687320+0000", 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.058180562999999998 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.687374+0000", 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4434000000000001e-05 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:43.687475+0000", 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100342 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.976 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:00:33.977 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:00:33.977 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:00:33.997 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:33.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:33.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:34.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:34.120 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:34.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:34 smithi172 ceph-mon[100696]: pgmap v1041: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:34 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:00:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:34 smithi172 ceph-mon[100696]: osdmap e336: 8 total, 8 up, 7 in 2024-09-13T15:00:34.244 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:34.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:34.362 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:34 smithi154 ceph-mon[102314]: pgmap v1041: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:34.362 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:34 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:00:34.362 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:34 smithi154 ceph-mon[102314]: osdmap e336: 8 total, 8 up, 7 in 2024-09-13T15:00:34.362 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:34 smithi154 ceph-mon[107038]: pgmap v1041: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:34.362 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:34 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:00:34.362 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:34 smithi154 ceph-mon[107038]: osdmap e336: 8 total, 8 up, 7 in 2024-09-13T15:00:34.368 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:34.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:34.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:34.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:34.615 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:34.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:34.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:00:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:34.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:34.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:00:34.751 DEBUG:teuthology.orchestra.run.smithi154:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-13T15:00:34.785 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T15:00:34.785 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T15:00:34.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:34.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:34.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:34.928 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:hints = 1 2024-09-13T15:00:34.928 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T15:00:34.928 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object prefix: benchmark_data_smithi154_261841 2024-09-13T15:00:34.928 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:00:34.928 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 0 0 0 0 0 0 - 0 2024-09-13T15:00:34.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:35.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:35.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:35.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:35.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:35.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:35.375 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:35 smithi154 ceph-mon[102314]: pgmap v1043: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:35.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:35 smithi154 ceph-mon[102314]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:00:35.376 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:35 smithi154 ceph-mon[107038]: pgmap v1043: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:35.376 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:35 smithi154 ceph-mon[107038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:00:35.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:35.423 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:35.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:35 smithi172 ceph-mon[100696]: pgmap v1043: 89 pgs: 89 active+clean; 4.6 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-13T15:00:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:35 smithi172 ceph-mon[100696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:00:35.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:35.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:35.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:35.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:35.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:35.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:35.928 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 1 16 3130 3114 194.613 194.625 0.00838919 0.00512339 2024-09-13T15:00:35.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:35.987 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:35.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:36.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:36.103 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T15:00:36.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:36.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:36.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:36.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:36.418 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:36.444 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1638531595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:36.444 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:36 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:36.444 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1638531595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:36.445 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:36 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1638531595' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:36 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:00:36.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:36.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:36.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:36.664 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:36.666 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:36.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:36.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:36.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:36.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:36.929 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 2 16 6513 6497 202.955 211.438 0.00326823 0.00491695 2024-09-13T15:00:36.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:36.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:37.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:37.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:37.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:37.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:37.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:37.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:37.381 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:37 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:37.381 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:37 smithi154 ceph-mon[107038]: osdmap e337: 8 total, 8 up, 7 in 2024-09-13T15:00:37.381 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:37 smithi154 ceph-mon[107038]: pgmap v1045: 89 pgs: 89 active+clean; 47 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s wr, 111 op/s 2024-09-13T15:00:37.381 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:37 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:37.381 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:37 smithi154 ceph-mon[102314]: osdmap e337: 8 total, 8 up, 7 in 2024-09-13T15:00:37.381 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:37 smithi154 ceph-mon[102314]: pgmap v1045: 89 pgs: 89 active+clean; 47 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s wr, 111 op/s 2024-09-13T15:00:37.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:37 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:00:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:37 smithi172 ceph-mon[100696]: osdmap e337: 8 total, 8 up, 7 in 2024-09-13T15:00:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:37 smithi172 ceph-mon[100696]: pgmap v1045: 89 pgs: 89 active+clean; 47 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 7.0 MiB/s wr, 111 op/s 2024-09-13T15:00:37.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:37.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:37.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:37.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:37.767 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:37.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:37.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:37.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:37.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:37.929 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 3 15 9650 9635 200.673 196.125 0.00542259 0.00497459 2024-09-13T15:00:37.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:38.027 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:38.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:38.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:38.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:38.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:38.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:38.420 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:38.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:38.560 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:38.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:38.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:38.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:38.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:38.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:00:38.930 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 4 16 12706 12690 198.171 190.938 0.00188105 0.00503744 2024-09-13T15:00:38.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:38.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:38.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:39.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:39.104 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:39.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:39.235 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:39.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:39.368 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:39.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:39.470 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:39.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:39.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:39.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:39.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.788 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:39.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:39.848 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:00:39.848 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:39.883 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:00:39.883 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:39.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:39.916 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:39.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:39.919 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:00:39.919 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:39.930 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 5 16 15156 15140 189.163 153.125 0.00570871 0.00527885 2024-09-13T15:00:39.954 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:00:39.954 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:39.990 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:00:39.990 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:39.991 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:39 smithi154 ceph-mon[102314]: pgmap v1046: 89 pgs: 89 active+clean; 47 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 6.3 MiB/s wr, 100 op/s 2024-09-13T15:00:39.991 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:39 smithi154 ceph-mon[107038]: pgmap v1046: 89 pgs: 89 active+clean; 47 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 6.3 MiB/s wr, 100 op/s 2024-09-13T15:00:39.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:39 smithi172 ceph-mon[100696]: pgmap v1046: 89 pgs: 89 active+clean; 47 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 6.3 MiB/s wr, 100 op/s 2024-09-13T15:00:40.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:40.022 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:00:40.022 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:40.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:40.057 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:00:40.058 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:40.098 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:00:40.098 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:40.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.191 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:40.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:40.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:40.321 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:40.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:40.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:40.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:40.594 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:40.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:40.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:40.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:40.850 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:40.931 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 6 16 17622 17606 183.322 154.125 0.00339242 0.00544874 2024-09-13T15:00:40.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:40.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:40 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3248463944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:40.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:40 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3248463944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:40.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:40.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:41.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:41.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:41.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:40 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3248463944' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:00:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:41.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:41.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:41.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:41.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.527 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:41.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:41.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:41.655 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:41.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:41.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:41.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:41.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:41.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:41.930 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 7 16 19528 19512 174.152 119.125 0.00732838 0.00573567 2024-09-13T15:00:42.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.044 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:42.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:42.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:42.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:41 smithi154 ceph-mon[102314]: pgmap v1047: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T15:00:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:41 smithi154 ceph-mon[107038]: pgmap v1047: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T15:00:42.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:00:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:00:41] "GET /metrics HTTP/1.1" 200 36542 "" "Prometheus/2.51.0" 2024-09-13T15:00:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:41 smithi172 ceph-mon[100696]: pgmap v1047: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-13T15:00:42.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:42.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:42.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:42.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:42.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:42.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:42.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:42.725 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:42.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:42.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:42.930 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 8 16 21733 21717 169.609 137.812 0.00611817 0.00589048 2024-09-13T15:00:42.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:42.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:42.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:43.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:43.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:43.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:43.271 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:43.373 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:43.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:43.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:43.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:43.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:43.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:43.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:43.796 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:43.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:43.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:43.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:43.930 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 9 16 24076 24060 167.034 146.438 0.00161541 0.0059739 2024-09-13T15:00:44.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.066 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:44.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:44.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.199 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:44.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:43 smithi172 ceph-mon[100696]: pgmap v1048: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T15:00:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:44.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:44.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:44.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:44.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:43 smithi154 ceph-mon[102314]: pgmap v1048: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T15:00:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:44.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:43 smithi154 ceph-mon[107038]: pgmap v1048: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-13T15:00:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:44.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:44.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:44.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:44.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:44.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:44.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:44.931 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 10 16 25860 25844 161.48 111.5 0.00145839 0.00614841 2024-09-13T15:00:44.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:44.982 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:44.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:45.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:45.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:45.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:45.244 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:45.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:45.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:45.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:45.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:45.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:45.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:45.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:45.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:45.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:45.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:45.871 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:45.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:45.931 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 11 16 27095 27079 153.818 77.1875 0.00739199 0.00649565 2024-09-13T15:00:45.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:46.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:46.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:46.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:46.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:45 smithi172 ceph-mon[100696]: pgmap v1049: 89 pgs: 89 active+clean; 1.0 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:00:46.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:46.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:46.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:46.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:45 smithi154 ceph-mon[102314]: pgmap v1049: 89 pgs: 89 active+clean; 1.0 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:00:46.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:45 smithi154 ceph-mon[107038]: pgmap v1049: 89 pgs: 89 active+clean; 1.0 GiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-13T15:00:46.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:46.559 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:46.688 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.816 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:46.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:46.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:46.931 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 12 16 29142 29126 151.655 127.938 0.00822685 0.00658732 2024-09-13T15:00:46.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:46.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:47.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:47.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:47.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:47.239 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:47.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:47.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:47.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:47.520 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:47.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:47.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:47.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:47.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:47.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:47.918 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:47.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:47.931 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 13 16 31233 31217 150.041 130.688 0.00436188 0.00665808 2024-09-13T15:00:48.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:48.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:48.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:48.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:47 smithi154 ceph-mon[102314]: pgmap v1050: 89 pgs: 89 active+clean; 1.6 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T15:00:48.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:47 smithi154 ceph-mon[107038]: pgmap v1050: 89 pgs: 89 active+clean; 1.6 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T15:00:48.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:47 smithi172 ceph-mon[100696]: pgmap v1050: 89 pgs: 89 active+clean; 1.6 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-13T15:00:48.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:48.303 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:48.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:48.432 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:48.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:48.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:48.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:48.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:48.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:48.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:48.931 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 14 16 33045 33029 147.413 113.25 0.00595192 0.00677792 2024-09-13T15:00:48.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:48.976 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:48.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:49.014 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:00:49.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:49.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:49.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:49.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:49.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:49.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:49.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1238058656' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:49.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:48 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:49.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1238058656' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:49.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:48 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:49.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:49.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1238058656' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:49.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:48 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:00:49.516 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:49.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:49.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:49.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:49.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:49.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:49.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:49.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:49.931 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 15 16 34883 34867 145.243 114.875 0.00418137 0.00685959 2024-09-13T15:00:50.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:50.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:50.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:50.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[102314]: pgmap v1051: 89 pgs: 89 active+clean; 1.6 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-13T15:00:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[102314]: osdmap e338: 8 total, 8 up, 7 in 2024-09-13T15:00:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[107038]: pgmap v1051: 89 pgs: 89 active+clean; 1.6 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-13T15:00:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:49 smithi154 ceph-mon[107038]: osdmap e338: 8 total, 8 up, 7 in 2024-09-13T15:00:50.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:50.324 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:50.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:50.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:49 smithi172 ceph-mon[100696]: pgmap v1051: 89 pgs: 89 active+clean; 1.6 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-13T15:00:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:49 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:00:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:49 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:00:50.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:49 smithi172 ceph-mon[100696]: osdmap e338: 8 total, 8 up, 7 in 2024-09-13T15:00:50.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:50.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:50.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:50.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:50.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:50.850 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:50.932 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 16 16 36174 36158 141.209 80.6875 0.00226627 0.00706638 2024-09-13T15:00:50.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:50.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:50.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:51.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:51.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:51.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:51.248 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:51.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:51.374 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:51.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:51.499 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:00:51.499 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:00:51.504 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:51.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:51.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:51.747 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:51.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:51.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:00:51.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:51.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:51.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:51.932 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 17 16 37964 37948 139.483 111.875 0.00517695 0.00716296 2024-09-13T15:00:52.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:52.046 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:52.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:52.164 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:52.171 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:52 smithi154 ceph-mon[102314]: pgmap v1053: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T15:00:52.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:00:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:00:51] "GET /metrics HTTP/1.1" 200 36610 "" "Prometheus/2.51.0" 2024-09-13T15:00:52.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:52 smithi154 ceph-mon[107038]: pgmap v1053: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T15:00:52.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:52.301 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:52.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:52.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:52 smithi172 ceph-mon[100696]: pgmap v1053: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T15:00:52.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:52.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:52.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:00:52.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:52.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:52.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:52.888 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:52.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:52.932 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 18 16 39719 39703 137.822 109.688 0.0159896 0.0072488 2024-09-13T15:00:52.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:53.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:53.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:53.162 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:53.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:53.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:53.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:53.426 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:53.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:53.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:53.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:53.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:53.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:53.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:53.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:53.932 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 19 16 41482 41466 136.367 110.188 0.0066732 0.00732761 2024-09-13T15:00:53.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:00:53.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:53.991 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:53.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:54.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:54.118 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:54.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:54.129 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:54.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:54.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:54.249 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:54 smithi154 ceph-mon[102314]: pgmap v1054: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T15:00:54.250 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:54 smithi154 ceph-mon[107038]: pgmap v1054: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T15:00:54.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:54.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:54 smithi172 ceph-mon[100696]: pgmap v1054: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-13T15:00:54.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:54.562 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:54.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:54.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:54.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:54.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:54.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:54.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:54.933 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:00:54.932479+0000 min lat: 0.00096194 max lat: 0.314029 avg lat: 0.00742336 2024-09-13T15:00:54.933 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:00:54.933 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 20 16 43099 43083 134.602 101.062 0.0139781 0.00742336 2024-09-13T15:00:54.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:54.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:55.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:55.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.228 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:55.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:55.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.368 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:55.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:55.494 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:55.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:55.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:55.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:55.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:55.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:55.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:55.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:55.933 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 21 16 44727 44711 133.037 101.75 0.0115763 0.00751155 2024-09-13T15:00:55.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:56.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:56.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:56 smithi172 ceph-mon[100696]: pgmap v1055: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-13T15:00:56.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:56.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:56.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:56.429 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:56 smithi154 ceph-mon[102314]: pgmap v1055: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-13T15:00:56.429 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:56 smithi154 ceph-mon[107038]: pgmap v1055: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-09-13T15:00:56.437 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:56.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:56.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:56.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:56.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:56.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:56.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:56.933 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 22 16 46116 46100 130.936 86.8125 0.0237539 0.00763101 2024-09-13T15:00:56.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:56.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:56.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:57.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:57.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:57.166 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:00:57.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:57.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:57.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:57.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:57.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:57.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:57.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:00:57.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:57.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:00:57.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:57.860 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:57.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:57.933 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 23 16 47387 47371 128.697 79.4375 0.0121518 0.0077646 2024-09-13T15:00:57.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:58.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:58.049 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:00:58.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:00:58.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:58.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:00:58.164 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[102314]: pgmap v1056: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-13T15:00:58.164 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/365920445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:58.164 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:58.164 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:58.164 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[107038]: pgmap v1056: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-13T15:00:58.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/365920445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:58.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:58.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:58.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:58.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:58.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:58.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:58 smithi172 ceph-mon[100696]: pgmap v1056: 89 pgs: 89 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-13T15:00:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/365920445' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:00:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:00:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:00:58.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:58.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:58.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:00:58.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:00:58.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:58.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:58.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:00:58.934 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 24 16 48925 48909 127.34 96.125 0.00282811 0.00784709 2024-09-13T15:00:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:58.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:58.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:00:59.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:59.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:59.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:59.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:59.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:59 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/365920445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:59.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:59 smithi154 ceph-mon[102314]: osdmap e339: 8 total, 8 up, 7 in 2024-09-13T15:00:59.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:00:59 smithi154 ceph-mon[102314]: pgmap v1058: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T15:00:59.207 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:59 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/365920445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:59.207 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:59 smithi154 ceph-mon[107038]: osdmap e339: 8 total, 8 up, 7 in 2024-09-13T15:00:59.207 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:00:59 smithi154 ceph-mon[107038]: pgmap v1058: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T15:00:59.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:00:59.307 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:00:59.307 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:00:59.307 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:00:59.307 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:00:59.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:00:59.346 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:00:59.346 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:00:59.386 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:00:59.386 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:00:59.417 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:00:59.418 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:00:59.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:59.447 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:00:59.447 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:00:59.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:59.475 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:00:59.475 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:00:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:59 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/365920445' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:00:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:59 smithi172 ceph-mon[100696]: osdmap e339: 8 total, 8 up, 7 in 2024-09-13T15:00:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:00:59 smithi172 ceph-mon[100696]: pgmap v1058: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-13T15:00:59.508 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:00:59.509 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:00:59.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:59.582 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:00:59.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:00:59.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:00:59.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:00:59.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:00:59.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:00:59.933 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 25 16 50295 50279 125.671 85.625 0.00969463 0.00795177 2024-09-13T15:00:59.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:00:59.991 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:00:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:00:59.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:00.051 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:01:00.051 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:01:00.051 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:01:00.051 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:01:00.051 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:01:00.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:00.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:00.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:00.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:00.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2783082006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2783082006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2783082006' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:00.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:00.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:00.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:00.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:00.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:00.814 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:00.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:00.934 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 26 16 51863 51847 124.604 98 0.00684184 0.00801884 2024-09-13T15:01:00.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:00.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:01.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:01.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:01.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:01:01.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:01.225 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:01.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:01.351 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:01.358 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:01 smithi172 ceph-mon[100696]: pgmap v1059: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:01:01.387 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:01 smithi154 ceph-mon[102314]: pgmap v1059: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:01:01.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:01 smithi154 ceph-mon[107038]: pgmap v1059: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:01:01.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:01.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:01.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:01.612 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:01.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:01.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:01.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:01.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:01.934 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 27 16 53360 53344 123.454 93.5625 0.0109929 0.00809422 2024-09-13T15:01:01.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:01:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:01:01] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-13T15:01:01.996 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:01.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:02.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:02.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:02.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:01:02.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:02.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:02.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:02.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:02.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:02.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:02.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:02.769 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:02.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:02.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:02.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:02.934 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 28 16 54795 54779 122.248 89.6875 0.0108674 0.00817542 2024-09-13T15:01:03.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:03.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:03.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:03.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:03.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:03.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:03.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:03.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:03.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:03.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:03.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:03.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:03.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:03.768 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:03.793 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:03.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:03.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:01:03.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:03.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:03.935 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 29 16 56214 56198 121.088 88.6875 0.00491506 0.00825246 2024-09-13T15:01:03.944 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:03 smithi154 ceph-mon[102314]: pgmap v1060: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:01:03.945 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:03 smithi154 ceph-mon[107038]: pgmap v1060: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:01:04.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:04.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:04.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:04.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:04.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:03 smithi172 ceph-mon[100696]: pgmap v1060: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-13T15:01:04.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:04.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:04.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:04.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:04.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:04.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:04.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:04.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:04.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:04.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:04.935 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 30 16 57549 57533 119.833 83.4375 0.00260935 0.00833292 2024-09-13T15:01:04.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:05.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:05.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:05.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:05.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:05.285 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:05.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:05.410 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:05.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:05.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:05.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:05.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:05.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:05.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:05.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:05.935 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 31 16 58735 58719 118.359 74.125 0.00795632 0.00844386 2024-09-13T15:01:05.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:05.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:06.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:06.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:06.183 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:06.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:06.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:05 smithi154 ceph-mon[102314]: pgmap v1061: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:01:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:05 smithi154 ceph-mon[107038]: pgmap v1061: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:01:06.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:05 smithi172 ceph-mon[100696]: pgmap v1061: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-13T15:01:06.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:06.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:06.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:06.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:06.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:06.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:06.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:06.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:06.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:06.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:06.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:06.935 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 32 16 59892 59876 116.918 72.3125 0.00625798 0.00854737 2024-09-13T15:01:06.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:07.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:07.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:07.138 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:07.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:07.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.417 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:07.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:07.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:07.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:07.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:07.676 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:07.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:07.800 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:07.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:07.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:07.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:07.935 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 33 16 61027 61011 115.525 70.9375 0.0174278 0.00865151 2024-09-13T15:01:08.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:08.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:08.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:08.178 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:07 smithi154 ceph-mon[102314]: pgmap v1062: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T15:01:08.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:07 smithi154 ceph-mon[107038]: pgmap v1062: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T15:01:08.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:07 smithi172 ceph-mon[100696]: pgmap v1062: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-13T15:01:08.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:08.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:08.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:08.444 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:08.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:08.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:08.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:08.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:08.701 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:08.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:08.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:08.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:08.935 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 34 16 62204 62188 114.291 73.5625 0.00945164 0.00874429 2024-09-13T15:01:08.973 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:08.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:08.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:09.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:09.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:09.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:09.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:09.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:09.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:09.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.496 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:09.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:09.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:09.626 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:09.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:09.754 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:09.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:09.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:09.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:09.936 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 35 16 63069 63053 112.57 54.0625 0.00271692 0.00882992 2024-09-13T15:01:09.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:10.005 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:10.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.144 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:10.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:10.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:09 smithi154 ceph-mon[102314]: pgmap v1063: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T15:01:10.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:09 smithi154 ceph-mon[107038]: pgmap v1063: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T15:01:10.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:09 smithi172 ceph-mon[100696]: pgmap v1063: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T15:01:10.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:10.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:10.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:10.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:10.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:10.530 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:10.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:10.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:10.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:10.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:10.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:10.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:10.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:10.936 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 36 16 64271 64255 111.529 75.125 0.016721 0.00895964 2024-09-13T15:01:11.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:11.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:11.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:11.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:11.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:11.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:11.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:11.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:11.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:11.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:11.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:11.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:11.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:11.857 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:11.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:11 smithi154 ceph-mon[102314]: pgmap v1064: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T15:01:11.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:11 smithi154 ceph-mon[107038]: pgmap v1064: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T15:01:11.936 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 37 16 65479 65463 110.556 75.5 0.00868688 0.00903972 2024-09-13T15:01:11.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:11.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:11.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:12.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:12.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:01:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:01:11] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-13T15:01:12.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:11 smithi172 ceph-mon[100696]: pgmap v1064: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-13T15:01:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:12.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:12.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.400 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:12.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:12.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:12.526 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:12.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:12.668 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:12.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:12.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:12.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:12.936 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 38 16 66658 66642 109.585 73.6875 0.00975891 0.0091172 2024-09-13T15:01:12.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:12.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:12.945 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:01:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:01:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:01:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:01:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206474 7.7 7:ed16db3f:::benchmark_data_smithi154_225924_object96210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.640343+0000", 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.287387697, 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.086262802, 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206474 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640343+0000", 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640343+0000", 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640345+0000", 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.429e-06 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640350+0000", 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7770000000000002e-06 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640358+0000", 2024-09-13T15:01:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6139999999999992e-06 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640366+0000", 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4900000000000003e-06 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640425+0000", 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9626999999999997e-05 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640437+0000", 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1270999999999999e-05 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.630363+0000", 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.989926521 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.630452+0000", 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8906999999999994e-05 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.630833+0000", 2024-09-13T15:01:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038098 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726226+0000", 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.095393194000000001 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726438+0000", 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000211998 2024-09-13T15:01:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726474+0000", 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5805e-05 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726524+0000", 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0071999999999998e-05 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726606+0000", 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2117e-05 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206476 7.6 7:6e2524ad:::benchmark_data_smithi154_225924_object96212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.287277407999994, 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0031420820000001, 2024-09-13T15:01:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206476 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5899999999999995e-07 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640458+0000", 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7950000000000001e-06 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640466+0000", 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1440000000000003e-06 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640485+0000", 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8547999999999999e-05 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.682274+0000", 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041789204000000003 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.682283+0000", 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5620000000000004e-06 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.622873+0000", 2024-09-13T15:01:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.94058993999999996 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.623013+0000", 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000140309 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.623452+0000", 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000438808 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637079+0000", 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013627121000000001 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.639954+0000", 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002875444 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643431+0000", 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003476556 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643486+0000", 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4607000000000001e-05 2024-09-13T15:01:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643596+0000", 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011028500000000001 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206483 7.f 7:fbb8947b:::benchmark_data_smithi154_225924_object96219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.666053+0000", 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.261678450999995, 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0757986070000001, 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206483 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666053+0000", 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666053+0000", 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666054+0000", 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.37e-06 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666059+0000", 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.412e-06 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666071+0000", 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.151e-05 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666079+0000", 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0940000000000008e-06 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666131+0000", 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2080999999999998e-05 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666138+0000", 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.311e-06 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.712568+0000", 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.046429209 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.712690+0000", 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012265 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:01:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.713129+0000", 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043868999999999999 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741477+0000", 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028348172000000001 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741649+0000", 2024-09-13T15:01:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017229699999999999 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741689+0000", 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9153999999999998e-05 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741753+0000", 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3941000000000003e-05 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741851+0000", 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8715999999999995e-05 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206491 7.7 7:ea59bc92:::benchmark_data_smithi154_225924_object96227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.672662+0000", 2024-09-13T15:01:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.255068842, 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1077584599999999, 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206491 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672662+0000", 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672662+0000", 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672664+0000", 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672669+0000", 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1930000000000001e-06 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672687+0000", 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8133000000000001e-05 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672697+0000", 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4590000000000004e-06 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672763+0000", 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5847999999999997e-05 2024-09-13T15:01:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672773+0000", 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8500000000000006e-06 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742530+0000", 2024-09-13T15:01:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.069756988 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742637+0000", 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107477 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742809+0000", 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000171545 2024-09-13T15:01:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.752976+0000", 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010167914 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.756045+0000", 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030688450000000002 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.780240+0000", 2024-09-13T15:01:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024194462 2024-09-13T15:01:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.780318+0000", 2024-09-13T15:01:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8339999999999999e-05 2024-09-13T15:01:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.780421+0000", 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102654 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206499 7.f 7:f6551031:::benchmark_data_smithi154_225924_object96235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.681433+0000", 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.246298277999998, 2024-09-13T15:01:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085242949, 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206499 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681433+0000", 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681433+0000", 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681434+0000", 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.003e-06 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681438+0000", 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3379999999999998e-06 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681450+0000", 2024-09-13T15:01:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2055000000000001e-05 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681465+0000", 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4653e-05 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681546+0000", 2024-09-13T15:01:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1155999999999993e-05 2024-09-13T15:01:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681562+0000", 2024-09-13T15:01:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5843000000000001e-05 2024-09-13T15:01:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741946+0000", 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0603843799999999 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742108+0000", 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016193100000000001 2024-09-13T15:01:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742232+0000", 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012353700000000001 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745195+0000", 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029633070000000001 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.752971+0000", 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077759500000000002 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.765956+0000", 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012985294 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.766039+0000", 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2367999999999995e-05 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.766676+0000", 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00063713400000000003 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206520 7.7 7:e2c3601b:::benchmark_data_smithi154_225924_object96256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.759086+0000", 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.168645584000004, 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0266216669999999, 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206520 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759086+0000", 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759086+0000", 2024-09-13T15:01:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759087+0000", 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3349999999999999e-06 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759093+0000", 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3609999999999997e-06 2024-09-13T15:01:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759107+0000", 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3456e-05 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759114+0000", 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8040000000000004e-06 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759168+0000", 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4752e-05 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759177+0000", 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3950000000000001e-06 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745626+0000", 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98644944899999998 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745746+0000", 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120185 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:01:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745871+0000", 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012463900000000001 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.752981+0000", 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007110384 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.762491+0000", 2024-09-13T15:01:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095095030000000007 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785513+0000", 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023022154 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785604+0000", 2024-09-13T15:01:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0752000000000001e-05 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785707+0000", 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103498 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206524 7.6 7:66f18573:::benchmark_data_smithi154_225924_object96260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.767148+0000", 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.160583036000006, 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.87661840599999996, 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206524 2024-09-13T15:01:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767148+0000", 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767148+0000", 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767149+0000", 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1480000000000001e-06 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767152+0000", 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.948e-06 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767160+0000", 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1820000000000006e-06 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767166+0000", 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999996e-06 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767230+0000", 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4321999999999998e-05 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767240+0000", 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0498e-05 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637108+0000", 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86986720900000003 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637235+0000", 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012727999999999999 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:01:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637369+0000", 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013369400000000001 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.639995+0000", 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002626729 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643141+0000", 2024-09-13T15:01:13.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003145283 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643627+0000", 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048677600000000002 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643671+0000", 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3912999999999998e-05 2024-09-13T15:01:13.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643767+0000", 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5163999999999999e-05 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206531 7.6 7:68c59cad:::benchmark_data_smithi154_225924_object96267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:01:13.011 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.794323+0000", 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.133408305000003, 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.849595451, 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206531 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.012 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794323+0000", 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794323+0000", 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794324+0000", 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.097e-06 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794328+0000", 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1999999999999996e-06 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794339+0000", 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0617e-05 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794345+0000", 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3350000000000003e-06 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.015 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794406+0000", 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0608e-05 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794418+0000", 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2455e-05 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.640076+0000", 2024-09-13T15:01:13.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.84565835300000003 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.640222+0000", 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000145988 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.640353+0000", 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013003900000000001 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643276+0000", 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029229880000000001 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643338+0000", 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2345e-05 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643773+0000", 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043504699999999998 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643813+0000", 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0383000000000003e-05 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643918+0000", 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104996 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206534 7.7 7:e3d4229e:::benchmark_data_smithi154_225924_object96270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:43.644006+0000", 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 89.283724887000005, 2024-09-13T15:01:13.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14271251600000001, 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206534 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644006+0000", 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644006+0000", 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644007+0000", 2024-09-13T15:01:13.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4499999999999995e-07 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644012+0000", 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0540000000000002e-06 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644023+0000", 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0325e-05 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644029+0000", 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5400000000000001e-06 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.749539+0000", 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.10551031600000001 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.749650+0000", 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011041500000000001 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.749794+0000", 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014454800000000001 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.756007+0000", 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062123350000000003 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.762527+0000", 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006519988 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.786560+0000", 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024033591 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.786614+0000", 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3634999999999997e-05 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.786719+0000", 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104824 2024-09-13T15:01:13.027 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206535 7.f 7:f7b09c47:::benchmark_data_smithi154_225924_object96271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:43.650650+0000", 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 89.277080909999995, 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.134818876, 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.028 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206535 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.650650+0000", 2024-09-13T15:01:13.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.650650+0000", 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.650653+0000", 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.807e-06 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.650663+0000", 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8749999999999995e-06 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.650678+0000", 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5277e-05 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.650690+0000", 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1531e-05 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.748635+0000", 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.097945779999999996 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.748756+0000", 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012087 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.748887+0000", 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130946 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.752959+0000", 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040714890000000002 2024-09-13T15:01:13.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.755999+0000", 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030399770000000001 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785262+0000", 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029263471999999999 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785354+0000", 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1952000000000003e-05 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785469+0000", 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0001149 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.035 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:213192 7.6 7:678df65b:::benchmark_data_smithi154_225924_object102928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:51.215821+0000", 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 81.711910004000003, 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.171609816, 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.036 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213192 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215821+0000", 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215821+0000", 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215822+0000", 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.252e-06 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215826+0000", 2024-09-13T15:01:13.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.738e-06 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215843+0000", 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6759e-05 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215853+0000", 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0030000000000001e-05 2024-09-13T15:01:13.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215944+0000", 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0888000000000001e-05 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.216082+0000", 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137749 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.216206+0000", 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012477300000000001 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.219412+0000", 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032052399999999998 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.219425+0000", 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3569e-05 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.387328+0000", 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.167902411 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.387386+0000", 2024-09-13T15:01:13.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8041000000000002e-05 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.387431+0000", 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5365999999999999e-05 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:213201 7.f 7:f4496a28:::benchmark_data_smithi154_225924_object102937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:51.223062+0000", 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 81.704669181, 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.166977974, 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:01:13.044 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213201 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223062+0000", 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223062+0000", 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223063+0000", 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1799999999999999e-06 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223068+0000", 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5739999999999997e-06 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223082+0000", 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4725e-05 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223089+0000", 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6989999999999997e-06 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223144+0000", 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5009999999999997e-05 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223254+0000", 2024-09-13T15:01:13.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109583 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223354+0000", 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100076 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.228032+0000", 2024-09-13T15:01:13.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046784319999999997 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.228141+0000", 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108785 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.389936+0000", 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.161794783 2024-09-13T15:01:13.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.389994+0000", 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8616000000000003e-05 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.390040+0000", 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5510999999999998e-05 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.051 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:58592 8.6 8:66025f50:::benchmark_data_smithi154_261841_object58591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.713052+0000", 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.2146792599999996, 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.135725176, 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.052 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 58592 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713052+0000", 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:13.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713052+0000", 2024-09-13T15:01:13.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713053+0000", 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0800000000000003e-07 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713073+0000", 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0619e-05 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713083+0000", 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6730000000000004e-06 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713089+0000", 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3679999999999998e-06 2024-09-13T15:01:13.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.714597+0000", 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015075799999999999 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.714701+0000", 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103541 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.714822+0000", 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121755 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721300+0000", 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064776909999999998 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848568+0000", 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12726848599999999 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848669+0000", 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100338 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848728+0000", 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8811000000000003e-05 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848777+0000", 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9406000000000001e-05 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:58595 8.6 8:614e9bb7:::benchmark_data_smithi154_261841_object58594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.713590+0000", 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.2141407019999999, 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13530942500000001, 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:01:13.062 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 58595 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713590+0000", 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713590+0000", 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713591+0000", 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9000000000000005e-07 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713612+0000", 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1036e-05 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713625+0000", 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2054e-05 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713632+0000", 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5939999999999996e-06 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.716759+0000", 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003127037 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.716868+0000", 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108725 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.716984+0000", 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116328 2024-09-13T15:01:13.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721304+0000", 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043200670000000004 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848612+0000", 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12730782400000001 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848819+0000", 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000206806 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848867+0000", 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7988000000000002e-05 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848900+0000", 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2975999999999997e-05 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:58600 8.6 8:6c80b0c1:::benchmark_data_smithi154_261841_object58599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.717523+0000", 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.2102081140000003, 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.131599155, 2024-09-13T15:01:13.069 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 58600 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.717523+0000", 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.717523+0000", 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.717524+0000", 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3999999999999999e-06 2024-09-13T15:01:13.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.717542+0000", 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8012999999999998e-05 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.717554+0000", 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1908e-05 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.717562+0000", 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3490000000000003e-06 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.719004+0000", 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014427140000000001 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.719112+0000", 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010723899999999999 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.719233+0000", 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012137399999999999 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848789+0000", 2024-09-13T15:01:13.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.129555635 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848937+0000", 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000148426 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848972+0000", 2024-09-13T15:01:13.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5139999999999999e-05 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.849090+0000", 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011816600000000001 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.849122+0000", 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1791000000000001e-05 2024-09-13T15:01:13.076 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.35787.0:58592 8.6 e339/331 ondisk, result = 0)", 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.721570+0000", 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.2061608039999996, 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.127006428, 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.077 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721570+0000", 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721570+0000", 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721571+0000", 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2399999999999997e-07 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721577+0000", 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4410000000000001e-06 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721588+0000", 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1579e-05 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721596+0000", 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3710000000000002e-06 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848550+0000", 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12695443200000001 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848561+0000", 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0791999999999999e-05 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848577+0000", 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5988999999999999e-05 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.35787.0:58595 8.6 e339/331 ondisk, result = 0)", 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.721615+0000", 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.2061160940000004, 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12700320000000001, 2024-09-13T15:01:13.082 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721615+0000", 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721615+0000", 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721615+0000", 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.72e-07 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721617+0000", 2024-09-13T15:01:13.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0329999999999998e-06 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721623+0000", 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7379999999999999e-06 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721627+0000", 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3579999999999996e-06 2024-09-13T15:01:13.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848601+0000", 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12697393800000001 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848607+0000", 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7119999999999997e-06 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848618+0000", 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1148999999999999e-05 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.35787.0:58592 8.6 e339/331 ondisk, result = 0)", 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.721949+0000", 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.2057825900000001, 2024-09-13T15:01:13.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.126841231, 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721949+0000", 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721949+0000", 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721949+0000", 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2799999999999996e-07 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721953+0000", 2024-09-13T15:01:13.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5410000000000001e-06 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848614+0000", 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.126661573 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848622+0000", 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.622e-06 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848659+0000", 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7255000000000002e-05 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848665+0000", 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4110000000000004e-06 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848790+0000", 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000125201 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.35787.0:58600 8.6 e339/331 ondisk, result = 0)", 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.722939+0000", 2024-09-13T15:01:13.092 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.2047920750000003, 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.126188827, 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.722939+0000", 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.722939+0000", 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.722940+0000", 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4799999999999998e-07 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.722943+0000", 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1e-06 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848470+0000", 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.125527165 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848488+0000", 2024-09-13T15:01:13.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8076000000000001e-05 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848962+0000", 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047398299999999998 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848968+0000", 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9950000000000002e-06 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.849128+0000", 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015996000000000001 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:62804 8.c 8:34586847:::benchmark_data_smithi154_261841_object62803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:09.463695+0000", 2024-09-13T15:01:13.097 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.464035967, 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12047263, 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 62804 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.463695+0000", 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.463695+0000", 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.463697+0000", 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4190000000000001e-06 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.463716+0000", 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9474e-05 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.463728+0000", 2024-09-13T15:01:13.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1625e-05 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.463735+0000", 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1439999999999997e-06 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.465437+0000", 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017022540000000001 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.465556+0000", 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011843799999999999 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.465642+0000", 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6119000000000007e-05 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:01:13.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.467680+0000", 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020383630000000001 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.468644+0000", 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096418399999999998 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.584069+0000", 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.11542472099999999 2024-09-13T15:01:13.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.584126+0000", 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7460999999999998e-05 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:09.584168+0000", 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1427999999999997e-05 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.104 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.105 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.105 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:13.105 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:13.105 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:01:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.107 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:13.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:13.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:13.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:13.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:13.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:13.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:13.527 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:13.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:13.658 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:13.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:13.783 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:13.813 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:01:13.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:13.913 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:13.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:13.936 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 39 16 67873 67857 108.722 75.9375 0.00996566 0.00919296 2024-09-13T15:01:14.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:14.042 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:13 smithi172 ceph-mon[100696]: pgmap v1065: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:01:14.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:14.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3348847006' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:14.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]} : dispatch 2024-09-13T15:01:14.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:14.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:14.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[102314]: pgmap v1065: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:01:14.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:14.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3348847006' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:14.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]} : dispatch 2024-09-13T15:01:14.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[107038]: pgmap v1065: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:01:14.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:14.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3348847006' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:14.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]} : dispatch 2024-09-13T15:01:14.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:14.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:14.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:14.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:14.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:14.587 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:14.617 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:14.719 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:14.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:14.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:01:14.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:14.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:14.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:14.936 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:14 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:14.936 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3348847006' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:14.936 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]': finished 2024-09-13T15:01:14.936 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:14 smithi172 ceph-mon[100696]: osdmap e340: 8 total, 8 up, 7 in 2024-09-13T15:01:14.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:01:14.936164+0000 min lat: 0.00096194 max lat: 0.314029 avg lat: 0.00926055 2024-09-13T15:01:14.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:01:14.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 40 16 69105 69089 107.929 77 0.00681108 0.00926055 2024-09-13T15:01:14.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:14.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:14.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3348847006' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:14.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]': finished 2024-09-13T15:01:14.993 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[102314]: osdmap e340: 8 total, 8 up, 7 in 2024-09-13T15:01:14.994 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:14.994 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3348847006' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:14.994 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 2]}]': finished 2024-09-13T15:01:14.994 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:14 smithi154 ceph-mon[107038]: osdmap e340: 8 total, 8 up, 7 in 2024-09-13T15:01:15.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.119 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:15.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:15.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:15.277 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:15.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:15.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:15.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:15.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:15.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:15.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:15.777 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:15.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:15.826 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:15 smithi154 ceph-mon[102314]: pgmap v1067: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T15:01:15.826 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:15 smithi154 ceph-mon[102314]: osdmap e341: 8 total, 8 up, 7 in 2024-09-13T15:01:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:15.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:15.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:15.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 41 16 70247 70231 107.037 71.375 0.0130709 0.0093364 2024-09-13T15:01:15.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:16.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:16.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:16.147 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:15 smithi154 ceph-mon[107038]: pgmap v1067: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T15:01:16.148 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:15 smithi154 ceph-mon[107038]: osdmap e341: 8 total, 8 up, 7 in 2024-09-13T15:01:16.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:16.206 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:15 smithi172 ceph-mon[100696]: pgmap v1067: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T15:01:16.207 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:15 smithi172 ceph-mon[100696]: osdmap e341: 8 total, 8 up, 7 in 2024-09-13T15:01:16.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:16.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:16.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:16.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:16.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.557 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:16.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:16.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:16.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:16.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:16.828 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:01:16.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:16.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:16.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 42 16 71363 71347 106.149 69.75 0.0134986 0.00941599 2024-09-13T15:01:16.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:16.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:16.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:17.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:17.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:17.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:17.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/428902723' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:01:17.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/428902723' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:01:17.235 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:17.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/428902723' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:01:17.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:01:17.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:17.363 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:17.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:17.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:17.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:17.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:17.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:17.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:17.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:17.907 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:17.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:17.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 43 16 72410 72394 105.202 65.4375 0.0155774 0.00950048 2024-09-13T15:01:18.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:18.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:18.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:18.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:18.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[102314]: pgmap v1069: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-13T15:01:18.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/428902723' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:01:18.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[102314]: osdmap e342: 8 total, 8 up, 7 in 2024-09-13T15:01:18.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[102314]: osdmap e343: 8 total, 8 up, 7 in 2024-09-13T15:01:18.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[107038]: pgmap v1069: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-13T15:01:18.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/428902723' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:01:18.183 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[107038]: osdmap e342: 8 total, 8 up, 7 in 2024-09-13T15:01:18.183 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:17 smithi154 ceph-mon[107038]: osdmap e343: 8 total, 8 up, 7 in 2024-09-13T15:01:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:17 smithi172 ceph-mon[100696]: pgmap v1069: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-13T15:01:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:17 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/428902723' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:01:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:17 smithi172 ceph-mon[100696]: osdmap e342: 8 total, 8 up, 7 in 2024-09-13T15:01:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:17 smithi172 ceph-mon[100696]: osdmap e343: 8 total, 8 up, 7 in 2024-09-13T15:01:18.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:18.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:18.397 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:18.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:18.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:18.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:18.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:18.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:18.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:18.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:18.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:18.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:18.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:18.939 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 44 16 73597 73581 104.497 74.1875 0.0167752 0.00956495 2024-09-13T15:01:18.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:18.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:19.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:19.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:19.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:19.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:18 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:18 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:19.237 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:18 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:01:19.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:19.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:19.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:19.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:19.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:19.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:19.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:19.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:19.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:19.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 45 15 74621 74606 103.598 64.0625 0.0181023 0.00964774 2024-09-13T15:01:19.954 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:19 smithi154 ceph-mon[102314]: pgmap v1072: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T15:01:19.954 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:19 smithi154 ceph-mon[107038]: pgmap v1072: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T15:01:19.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:19.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:20.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:20.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:20.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:20.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:20.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:20.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:19 smithi172 ceph-mon[100696]: pgmap v1072: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T15:01:20.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:20.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:20.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:20.509 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:20.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:20.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:20.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:20.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:20.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:20.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:20.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:20.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:20.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 46 16 75722 75706 102.841 68.75 0.00775304 0.00971866 2024-09-13T15:01:20.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:21.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:21.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:21.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:21.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:21.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:21.370 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:01:21.370 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:21.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:21.404 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:01:21.404 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:21.416 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:21.443 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:01:21.444 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:21.477 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:01:21.478 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:21.509 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:01:21.510 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:21.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:21.543 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:01:21.543 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:21.560 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:21.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:21.586 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:01:21.587 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:21.629 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:01:21.629 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:21.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:21.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:21.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:21.814 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:21.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:21.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 47 16 76826 76810 102.121 69 0.0104654 0.00978719 2024-09-13T15:01:21.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:21.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:21.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:21 smithi154 ceph-mon[102314]: pgmap v1073: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T15:01:21.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3547643209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:21.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:21 smithi154 ceph-mon[107038]: pgmap v1073: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T15:01:21.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3547643209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:21.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:01:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:01:21] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-09-13T15:01:22.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:22.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:22.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:22.305 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:21 smithi172 ceph-mon[100696]: pgmap v1073: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-13T15:01:22.305 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3547643209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:22.338 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:22.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:22.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T15:01:22.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:22.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:22.593 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:22.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:22.721 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:22.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:22.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:22.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 48 16 77984 77968 101.501 72.375 0.0062304 0.00984667 2024-09-13T15:01:22.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:22.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:22.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:23.116 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:23.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:23.233 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:23.233 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:23.242 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:23.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:23.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:23.516 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:23.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:23.675 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:23.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:23.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:23.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:23.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 49 16 79062 79046 100.804 67.375 0.0187325 0.0099147 2024-09-13T15:01:23.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:23.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:23 smithi154 ceph-mon[102314]: pgmap v1074: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T15:01:23.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:23 smithi154 ceph-mon[107038]: pgmap v1074: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T15:01:23.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:23.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:24.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:24.108 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:24.148 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:24.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:01:24.210 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:23 smithi172 ceph-mon[100696]: pgmap v1074: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-13T15:01:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:24.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:24.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:24.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:24.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:24.557 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:24.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:24.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:24.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:24.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:24.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:24.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:01:24.937 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 50 16 80109 80093 100.097 65.4375 0.0174383 0.00998519 2024-09-13T15:01:24.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:24.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:25.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:25.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:25.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:25.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:25.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:24 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:25.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:24 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:25.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:24 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:25.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:24 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:25.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:24 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:25.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:24 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:25.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:25.368 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:25.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:25.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:25.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:25.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:25.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:25.771 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:25.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:25.908 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:25.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:25.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 51 16 81218 81202 99.4931 69.3125 0.0125591 0.0100454 2024-09-13T15:01:26.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:26.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:26.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.171 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:26.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:25 smithi172 ceph-mon[100696]: pgmap v1075: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-13T15:01:26.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.294 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:26.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:26.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:26.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:25 smithi154 ceph-mon[102314]: pgmap v1075: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-13T15:01:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:25 smithi154 ceph-mon[107038]: pgmap v1075: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 49 MiB/s wr, 778 op/s 2024-09-13T15:01:26.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:26.556 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:26.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:26.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:26.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:26.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:26.812 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:26.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:26.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:26.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 52 16 82068 82052 98.6002 53.125 0.0212247 0.0101364 2024-09-13T15:01:26.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:26.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:27.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:27.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:27.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:27.228 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:27.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:27.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:27.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:27.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:27.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.638 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:27.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:27.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:27.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:27.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:27.916 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:27.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:27.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 53 16 83184 83168 98.0558 69.75 0.0139198 0.0101928 2024-09-13T15:01:28.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:28.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:28.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:28.154 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:27 smithi154 ceph-mon[102314]: pgmap v1076: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:28.154 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:27 smithi154 ceph-mon[107038]: pgmap v1076: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:28.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:27 smithi172 ceph-mon[100696]: pgmap v1076: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-13T15:01:28.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:28.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:28.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:28.442 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:28.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.614 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:28.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:28.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:28.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:28.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:28.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:28.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:28.939 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 54 16 84225 84209 97.4448 65.0625 0.0166533 0.0102571 2024-09-13T15:01:28.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:29.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:29.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:29.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:29.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:29.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:29.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:29.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:29.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:29.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:29.562 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:29.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:29.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:29.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:29.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:29.823 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:29.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:29.939 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 55 16 85365 85349 96.9675 71.25 0.025909 0.0103074 2024-09-13T15:01:29.965 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:29.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:30.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:30.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:30.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:30.228 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:30.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:29 smithi154 ceph-mon[102314]: pgmap v1077: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T15:01:30.281 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:29 smithi154 ceph-mon[107038]: pgmap v1077: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T15:01:30.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:30.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:30.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:29 smithi172 ceph-mon[100696]: pgmap v1077: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-13T15:01:30.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:30.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:30.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:30.759 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:30.821 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:30.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:30.874 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:30.926 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:30.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:30.939 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 56 16 86202 86186 96.17 52.3125 0.0421005 0.01039 2024-09-13T15:01:31.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:31.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:31.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:31.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:31.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:31.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:31.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:31.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:31.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:31.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:31.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:31.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:31.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:31.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:31.939 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 57 16 87068 87052 95.4323 54.125 0.048243 0.010472 2024-09-13T15:01:31.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:31.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:32.003 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T15:01:32.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:32.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:32.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:32.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:32.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:32 smithi154 ceph-mon[102314]: pgmap v1078: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T15:01:32.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/986420158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:01:32.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:32 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:01:32.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:01:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:01:31] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-13T15:01:32.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:32 smithi154 ceph-mon[107038]: pgmap v1078: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T15:01:32.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/986420158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:01:32.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:32 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:01:32.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.333 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:32.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:32.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:32.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:32 smithi172 ceph-mon[100696]: pgmap v1078: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-13T15:01:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/986420158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:01:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:32 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:01:32.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:32.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:32.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:32.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:32.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:32.893 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:32.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:32.939 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 58 16 87886 87870 94.6684 51.125 0.0215982 0.0105582 2024-09-13T15:01:32.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:33.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:33.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:33.159 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:33.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:33.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:33.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:33 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:01:33.350 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:33 smithi154 ceph-mon[102314]: osdmap e344: 8 total, 8 up, 7 in 2024-09-13T15:01:33.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:33 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:01:33.350 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:33 smithi154 ceph-mon[107038]: osdmap e344: 8 total, 8 up, 7 in 2024-09-13T15:01:33.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:33.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:33 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:01:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:33 smithi172 ceph-mon[100696]: osdmap e344: 8 total, 8 up, 7 in 2024-09-13T15:01:33.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:33.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:33.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:33.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:33.697 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:33.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:33.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:33.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:33.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 59 16 88895 88879 94.1327 63.0625 0.0139803 0.010618 2024-09-13T15:01:33.951 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:33.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:34.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:34.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:34.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:34.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:34.310 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:34 smithi154 ceph-mon[102314]: pgmap v1080: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:01:34.311 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:34 smithi154 ceph-mon[107038]: pgmap v1080: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:01:34.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:34.363 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:34.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:34 smithi172 ceph-mon[100696]: pgmap v1080: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-13T15:01:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:34.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:34.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:34.641 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:34.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:34.780 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:34.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:34.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:34.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:34.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:34.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:01:34.939524+0000 min lat: 0.00096194 max lat: 0.314029 avg lat: 0.0106798 2024-09-13T15:01:34.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:01:34.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 60 16 89873 89857 93.5825 61.125 0.0110615 0.0106798 2024-09-13T15:01:35.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:35.043 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:01:35.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:35.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:35.196 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:35.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:01:35.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:35.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:35.379 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4184696937' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:35.380 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4184696937' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:35.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4184696937' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:01:35.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:35.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:35.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:35.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:35.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:35.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:35.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:35.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:35.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 61 16 90860 90844 93.0577 61.6875 0.00563636 0.0107403 2024-09-13T15:01:36.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:36.036 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:36.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:36.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:36.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:36.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:36.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:36.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:36.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[102314]: pgmap v1081: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:01:36.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:36.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4184696937' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:36.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[102314]: osdmap e345: 8 total, 8 up, 7 in 2024-09-13T15:01:36.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[107038]: pgmap v1081: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:01:36.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:36.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4184696937' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:36.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:36 smithi154 ceph-mon[107038]: osdmap e345: 8 total, 8 up, 7 in 2024-09-13T15:01:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:36 smithi172 ceph-mon[100696]: pgmap v1081: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-13T15:01:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:36 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:01:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4184696937' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:01:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:36 smithi172 ceph-mon[100696]: osdmap e345: 8 total, 8 up, 7 in 2024-09-13T15:01:36.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:36.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:36.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:36.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:36.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:36.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:36.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 62 16 91886 91870 92.591 64.125 0.0100654 0.0107945 2024-09-13T15:01:36.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:36.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:36.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:37.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.118 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:37.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:37.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:37.266 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:37.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:37.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:37.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:37.547 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:37.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:37.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:37.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:37.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:37.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:37.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 63 16 92867 92851 92.0945 61.3125 0.00776211 0.0108526 2024-09-13T15:01:37.947 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:37.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:38.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:38.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:38.093 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:38.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:38.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:38.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:38.343 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:38.428 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:38 smithi154 ceph-mon[102314]: pgmap v1083: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:01:38.428 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:38 smithi154 ceph-mon[107038]: pgmap v1083: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:01:38.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:38.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:38 smithi172 ceph-mon[100696]: pgmap v1083: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-13T15:01:38.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:38.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:38.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:01:38.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:38.752 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:38.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:38.877 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:38.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:38.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 64 16 93820 93804 91.5862 59.5625 0.0190679 0.0109131 2024-09-13T15:01:38.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:39.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:39.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:39.157 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:39.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:39.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:39.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:39.427 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:39.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:39.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:39.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:39.684 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:39.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:39.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:39.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:39.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 65 16 94775 94759 91.0954 59.6875 0.0296362 0.0109717 2024-09-13T15:01:39.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:39.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:40.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:40.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:40.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:40.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:40 smithi154 ceph-mon[107038]: pgmap v1084: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-13T15:01:40.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:40 smithi154 ceph-mon[102314]: pgmap v1084: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-13T15:01:40.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:40.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:40.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:40.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:40.478 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:40 smithi172 ceph-mon[100696]: pgmap v1084: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-13T15:01:40.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:40.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:40.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:40.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:40.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:40.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:40.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:40.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 66 16 95731 95715 90.6205 59.75 0.0304893 0.0110296 2024-09-13T15:01:40.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:41.022 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:41.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.166 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:41.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:41.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:41.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:41.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:41.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:41.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:41.559 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:41.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:41.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:41.751 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:01:41.751 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:01:41.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:41.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:41.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:41.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 67 16 96608 96592 90.086 54.8125 0.0147017 0.0110954 2024-09-13T15:01:41.947 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:41.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:42.033 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:01:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:01:41] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-13T15:01:42.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:42.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:42.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:42.216 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:42.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:42.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:42.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:42.445 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:42 smithi154 ceph-mon[102314]: pgmap v1085: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-13T15:01:42.446 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1788465053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:42.446 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:42 smithi154 ceph-mon[107038]: pgmap v1085: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-13T15:01:42.446 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1788465053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:42.468 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:42 smithi172 ceph-mon[100696]: pgmap v1085: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-13T15:01:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1788465053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:01:42.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:42.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:42.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:42.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:42.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:42.879 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:42.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:42.942 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 68 16 97585 97569 89.6591 61.0625 0.0111332 0.0111483 2024-09-13T15:01:42.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:43.008 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:43.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:43.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:43.154 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:43.154 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:43.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:01:43.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:43.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:43.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:43.390 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:01:43.390 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:01:43.390 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:01:43.390 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:01:43.390 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:01:43.391 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:01:43.392 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:01:43.392 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:01:43.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:43.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:43.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:43.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.677 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:43.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:43.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:43.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:43.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:43.942 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 69 16 98437 98421 89.1315 53.25 0.0100402 0.011214 2024-09-13T15:01:43.943 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:43.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:44.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:44.072 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:01:44.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:44.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:44.202 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:44.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:44.343 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:44 smithi154 ceph-mon[102314]: pgmap v1086: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:01:44.343 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1639956479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:01:44.343 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:44 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:01:44.344 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:44 smithi154 ceph-mon[107038]: pgmap v1086: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:01:44.344 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1639956479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:01:44.344 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:44 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:01:44.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:44.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:44 smithi172 ceph-mon[100696]: pgmap v1086: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:01:44.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1639956479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:01:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:44 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:01:44.455 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:44.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:44.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:44.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:44.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:44.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:44.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:44.891 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:44.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:44.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:44.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 70 16 99379 99363 88.6929 58.875 0.0255078 0.0112689 2024-09-13T15:01:45.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:45.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:45.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:45.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:45 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:01:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:45 smithi154 ceph-mon[102314]: osdmap e346: 8 total, 8 up, 7 in 2024-09-13T15:01:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:45 smithi154 ceph-mon[102314]: pgmap v1088: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 858 op/s 2024-09-13T15:01:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:45 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:01:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:45 smithi154 ceph-mon[107038]: osdmap e346: 8 total, 8 up, 7 in 2024-09-13T15:01:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:45 smithi154 ceph-mon[107038]: pgmap v1088: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 858 op/s 2024-09-13T15:01:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:45 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:01:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:45 smithi172 ceph-mon[100696]: osdmap e346: 8 total, 8 up, 7 in 2024-09-13T15:01:45.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:45 smithi172 ceph-mon[100696]: pgmap v1088: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 858 op/s 2024-09-13T15:01:45.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:45.322 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:45.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:45.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:45.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:01:45.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:45.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:45.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:45.717 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:45.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:45.843 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:45.945 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:45.948 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 71 16 100302 100286 88.2558 57.6875 0.018368 0.011325 2024-09-13T15:01:45.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:45.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:46.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:46.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:46.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:46.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:46.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:01:46.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:46.420 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:46.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:46.551 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:46.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:46.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:46.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:46.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:46.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:46.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 72 16 101188 101172 87.7992 55.375 0.0130157 0.0113847 2024-09-13T15:01:46.950 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:46.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:47.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:47.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:47.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:47.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:47.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:47.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:47.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:47.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:47.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:47.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:47.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:47.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:47.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:47.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:47.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:47.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 73 16 101986 101970 87.2797 49.875 0.0072704 0.0114517 2024-09-13T15:01:47.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:47 smithi172 ceph-mon[100696]: pgmap v1089: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 953 op/s 2024-09-13T15:01:48.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:48.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:48.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:48.168 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:48.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:47 smithi154 ceph-mon[102314]: pgmap v1089: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 953 op/s 2024-09-13T15:01:48.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:47 smithi154 ceph-mon[107038]: pgmap v1089: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 953 op/s 2024-09-13T15:01:48.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:48.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:48.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:48.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:48.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:48.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:48.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:48.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:48.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:48.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:48.943 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:48.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 74 16 102829 102813 86.8122 52.6875 0.0346931 0.0115132 2024-09-13T15:01:48.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:48.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:49.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:49.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:49.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:49.231 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:49.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:49.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:49.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:49.480 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:49.485 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:49.587 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:49.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:49.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:49.745 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:49.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:49.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:49.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:49.948 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 75 16 103728 103712 86.4034 56.1875 0.01156 0.0115678 2024-09-13T15:01:50.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:50.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:50.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:50.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:50.179 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:49 smithi154 ceph-mon[102314]: pgmap v1090: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 953 op/s 2024-09-13T15:01:50.180 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:49 smithi154 ceph-mon[107038]: pgmap v1090: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 953 op/s 2024-09-13T15:01:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:49 smithi172 ceph-mon[100696]: pgmap v1090: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 953 op/s 2024-09-13T15:01:50.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:50.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:50.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:50.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:50.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:50.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:50.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:50.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:50.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:50.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:50.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:50.948 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 76 16 104639 104623 86.0157 56.9375 0.00653653 0.0116203 2024-09-13T15:01:50.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:51.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:51.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:51.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:51.230 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:51.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:51.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:51.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:51.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:51.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:51.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:51.647 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:51.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:51.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:51.778 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:01:51.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:51.904 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:51.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:51.949 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 77 16 105490 105474 85.5884 53.1875 0.0340159 0.0116788 2024-09-13T15:01:52.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:52.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:52.142 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:51 smithi154 ceph-mon[102314]: pgmap v1091: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s 2024-09-13T15:01:52.143 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2226770877' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:52.143 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:51 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:52.143 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:01:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:01:51] "GET /metrics HTTP/1.1" 200 36596 "" "Prometheus/2.51.0" 2024-09-13T15:01:52.143 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:51 smithi154 ceph-mon[107038]: pgmap v1091: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s 2024-09-13T15:01:52.144 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2226770877' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:52.144 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:51 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:52.154 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:52.177 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:51 smithi172 ceph-mon[100696]: pgmap v1091: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 901 op/s 2024-09-13T15:01:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2226770877' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:51 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:01:52.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:52.304 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:52.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:52.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:52.435 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:52.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:52.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:52.586 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:52.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:52.737 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:52.825 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:52 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:01:52.826 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:52 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:52.826 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:52 smithi154 ceph-mon[107038]: osdmap e347: 8 total, 8 up, 7 in 2024-09-13T15:01:52.826 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:52 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:01:52.826 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:52 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:52.826 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:52 smithi154 ceph-mon[102314]: osdmap e347: 8 total, 8 up, 7 in 2024-09-13T15:01:52.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:52.868 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:52.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:52.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 78 16 106374 106358 85.1995 55.25 0.0238676 0.0117315 2024-09-13T15:01:52.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:53.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:53.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:53.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:52 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:01:53.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:52 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:01:53.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:52 smithi172 ceph-mon[100696]: osdmap e347: 8 total, 8 up, 7 in 2024-09-13T15:01:53.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:53.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:53.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:53.478 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:53.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:53.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:53.632 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:53.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:53.759 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:53.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:53.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:53.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:53.949 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 79 16 107198 107182 84.7729 51.5 0.00596806 0.0117908 2024-09-13T15:01:53.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:54.012 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:54.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:54.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:54.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:53 smithi154 ceph-mon[102314]: pgmap v1093: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:01:54.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:53 smithi154 ceph-mon[107038]: pgmap v1093: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:01:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:53 smithi172 ceph-mon[100696]: pgmap v1093: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:01:54.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:54.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:54.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:54.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:54.431 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:01:54.431 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:01:54.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:01:54.431 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:01:54.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.431 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:96331 8.e e345/331 8:73f73a06:::benchmark_data_smithi154_261841_object96330:head v 345'6064, mlcod=345'6060)", 2024-09-13T15:01:54.431 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:41.682221+0000", 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.75513039, 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032163114, 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.682221+0000", 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.682221+0000", 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.682222+0000", 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8700000000000005e-07 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.682250+0000", 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8000999999999998e-05 2024-09-13T15:01:54.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.682260+0000", 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0417999999999999e-05 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.682269+0000", 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3030000000000005e-06 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.689672+0000", 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074022580000000001 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.689690+0000", 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8236999999999998e-05 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.712360+0000", 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022670314 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:41.714384+0000", 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020238959999999999 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.436 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:97218 8.3 e345/331 8:c3413088:::benchmark_data_smithi154_261841_object97217:head v 345'6051, mlcod=345'6049)", 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:42.549520+0000", 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.887831019, 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033629531999999997, 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.549520+0000", 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.549520+0000", 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.549521+0000", 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5300000000000003e-07 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.549543+0000", 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2019999999999999e-05 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.549559+0000", 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6127e-05 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.549568+0000", 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8410000000000006e-06 2024-09-13T15:01:54.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.553449+0000", 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038804460000000001 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.553502+0000", 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3545000000000003e-05 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.583105+0000", 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029602454 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:42.583150+0000", 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5246e-05 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:97884 8.6 e345/331 8:6a65b8b3:::benchmark_data_smithi154_261841_object97883:head v 345'6137, mlcod=345'6133)", 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:43.275227+0000", 2024-09-13T15:01:54.441 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.162124575, 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031824910999999997, 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.275227+0000", 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.275227+0000", 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.275228+0000", 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.252e-06 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.275267+0000", 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8656999999999999e-05 2024-09-13T15:01:54.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.275290+0000", 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3453000000000001e-05 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.275298+0000", 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8779999999999998e-06 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.277229+0000", 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001930862 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.280576+0000", 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003347638 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.306929+0000", 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026352415000000001 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.307052+0000", 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012275599999999999 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:97954 8.7 e345/331 8:eea85f5f:::benchmark_data_smithi154_261841_object97953:head v 345'6117, mlcod=345'6115)", 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:43.367389+0000", 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.069962629000001, 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031731769999999999, 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.446 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.367389+0000", 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.367389+0000", 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.367389+0000", 2024-09-13T15:01:54.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2599999999999998e-07 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.367420+0000", 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1272000000000001e-05 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.367427+0000", 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9179999999999997e-06 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.367432+0000", 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0490000000000003e-06 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.368243+0000", 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081099999999999998 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.368292+0000", 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8390000000000003e-05 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.399079+0000", 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030787141000000001 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.399120+0000", 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1674000000000001e-05 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:97957 8.a e345/331 8:5450e94e:::benchmark_data_smithi154_261841_object97956:head v 345'6134, mlcod=345'6132)", 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:43.385994+0000", 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.05135727, 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031698781000000002, 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.385994+0000", 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.385994+0000", 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.385994+0000", 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4400000000000001e-07 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.386015+0000", 2024-09-13T15:01:54.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0693999999999999e-05 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.386021+0000", 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.784e-06 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.386025+0000", 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1810000000000003e-06 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.386107+0000", 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2223000000000002e-05 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.386130+0000", 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2609999999999999e-05 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.411078+0000", 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024948142999999999 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.417693+0000", 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066149019999999998 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:97963 8.a e345/331 8:53c66bb3:::benchmark_data_smithi154_261841_object97962:head v 345'6135, mlcod=345'6132)", 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:43.403129+0000", 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.034222779, 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038073914, 2024-09-13T15:01:54.455 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403129+0000", 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403129+0000", 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403130+0000", 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.046e-06 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403154+0000", 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4593999999999999e-05 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403167+0000", 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2481e-05 2024-09-13T15:01:54.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403174+0000", 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2760000000000003e-06 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403362+0000", 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018806399999999999 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.403410+0000", 2024-09-13T15:01:54.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8167999999999997e-05 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.441022+0000", 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037611908999999999 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.441202+0000", 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000180376 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:98025 8.4 e345/331 8:2111a7e7:::benchmark_data_smithi154_261841_object98024:head v 345'6101, mlcod=345'6099)", 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:43.474091+0000", 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.9632607180000008, 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035000813999999998, 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474091+0000", 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474091+0000", 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474091+0000", 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9499999999999998e-07 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474126+0000", 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5244999999999999e-05 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474133+0000", 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5529999999999998e-06 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474137+0000", 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6779999999999999e-06 2024-09-13T15:01:54.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474553+0000", 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041586699999999999 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.474593+0000", 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0055999999999997e-05 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.508909+0000", 2024-09-13T15:01:54.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034316091 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.509091+0000", 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000182029 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:98026 8.4 e345/331 8:28286643:::benchmark_data_smithi154_261841_object98025:head v 345'6102, mlcod=345'6099)", 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:43.476947+0000", 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.9604039960000001, 2024-09-13T15:01:54.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032174626999999997, 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.476947+0000", 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.476947+0000", 2024-09-13T15:01:54.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.476948+0000", 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4e-07 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.476990+0000", 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1996999999999998e-05 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477006+0000", 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6030999999999999e-05 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477014+0000", 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8599999999999993e-06 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477752+0000", 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073792300000000003 2024-09-13T15:01:54.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477774+0000", 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1526e-05 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.509095+0000", 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031321406000000003 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.509122+0000", 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6944000000000002e-05 2024-09-13T15:01:54.468 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:98027 8.4 e345/331 8:2ee7b739:::benchmark_data_smithi154_261841_object98026:head v 345'6103, mlcod=345'6099)", 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:43.477028+0000", 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.9603229889999998, 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032116743000000003, 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477028+0000", 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477028+0000", 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477029+0000", 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3099999999999999e-07 2024-09-13T15:01:54.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477058+0000", 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9102999999999999e-05 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477064+0000", 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3969999999999999e-06 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.477069+0000", 2024-09-13T15:01:54.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5730000000000001e-06 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.489025+0000", 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011956273 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.489049+0000", 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4128999999999999e-05 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.509125+0000", 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020075407 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:43.509145+0000", 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0530000000000002e-05 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:99324 8.7 e346/331 8:e03772fa:::benchmark_data_smithi154_261841_object99323:head v 346'6191, mlcod=346'6189)", 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:44.879841+0000", 2024-09-13T15:01:54.473 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.5575102790000006, 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036813991999999997, 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.879841+0000", 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.879841+0000", 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.879842+0000", 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4200000000000004e-07 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.879875+0000", 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3108000000000002e-05 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.879887+0000", 2024-09-13T15:01:54.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1839e-05 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.879895+0000", 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9570000000000002e-06 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.887273+0000", 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073779190000000001 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.887292+0000", 2024-09-13T15:01:54.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8947999999999999e-05 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.916614+0000", 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029322124000000001 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.916655+0000", 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1155000000000002e-05 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:99325 8.f e346/331 8:fe4fbc33:::benchmark_data_smithi154_261841_object99324:head v 346'6123, mlcod=346'6121)", 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:44.880046+0000", 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.557304877, 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036562773999999999, 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.880046+0000", 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.880046+0000", 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.880047+0000", 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5899999999999996e-07 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.880072+0000", 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4910999999999999e-05 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.880081+0000", 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.414e-06 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.880086+0000", 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0579999999999997e-06 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.891522+0000", 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011435715000000001 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.891543+0000", 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1115999999999999e-05 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.913035+0000", 2024-09-13T15:01:54.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021491605 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.916609+0000", 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035743960000000001 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:99330 8.7 e346/331 8:e75582d8:::benchmark_data_smithi154_261841_object99329:head v 346'6192, mlcod=346'6189)", 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:44.885000+0000", 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.5523511649999993, 2024-09-13T15:01:54.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.031678757000000002, 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.885000+0000", 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.885000+0000", 2024-09-13T15:01:54.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.885000+0000", 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3900000000000001e-07 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.885018+0000", 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7912000000000001e-05 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.885024+0000", 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1160000000000004e-06 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.885029+0000", 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2529999999999998e-06 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.901248+0000", 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016218871999999999 2024-09-13T15:01:54.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.901271+0000", 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3220000000000001e-05 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.916658+0000", 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015387256 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:44.916679+0000", 2024-09-13T15:01:54.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0888999999999999e-05 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:100004 8.4 e346/331 8:2c9e0c27:::benchmark_data_smithi154_261841_object100003:head v 346'6230, mlcod=346'6227)", 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:45.613447+0000", 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.8239039999999997, 2024-09-13T15:01:54.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038248378, 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.613447+0000", 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.613447+0000", 2024-09-13T15:01:54.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.613448+0000", 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3500000000000002e-07 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.613482+0000", 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4826000000000002e-05 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.613489+0000", 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4570000000000002e-06 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.613494+0000", 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1739999999999999e-06 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.626183+0000", 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012689261 2024-09-13T15:01:54.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.626217+0000", 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3376e-05 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.651566+0000", 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025349226999999998 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.651696+0000", 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000129722 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:100063 8.e e346/331 8:706cea1d:::benchmark_data_smithi154_261841_object100062:head v 346'6330, mlcod=346'6325)", 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:45.695564+0000", 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.7417873940000002, 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030966626000000001, 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.492 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.695564+0000", 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.695564+0000", 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.695564+0000", 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.39e-07 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.695586+0000", 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1885000000000001e-05 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.695592+0000", 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1489999999999999e-06 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.695597+0000", 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9389999999999996e-06 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.705571+0000", 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099740369999999998 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.707639+0000", 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020677489999999998 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.726445+0000", 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018806471000000002 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:45.726531+0000", 2024-09-13T15:01:54.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5056999999999996e-05 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:103459 8.7 e346/331 8:ea6377a1:::benchmark_data_smithi154_261841_object103458:head v 346'6453, mlcod=346'6451)", 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:49.668127+0000", 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.7692238169999999, 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032684422999999997, 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.497 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668127+0000", 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668127+0000", 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668128+0000", 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.569999999999999e-07 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668150+0000", 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2048e-05 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668164+0000", 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.384e-05 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668172+0000", 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4050000000000003e-06 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668358+0000", 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000186255 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.668445+0000", 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7151999999999995e-05 2024-09-13T15:01:54.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.700767+0000", 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032322286999999998 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.700812+0000", 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4478999999999999e-05 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.501 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.502 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.502 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:103525 8.e e346/331 8:760230e7:::benchmark_data_smithi154_261841_object103524:head v 346'6566, mlcod=346'6563)", 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:49.731974+0000", 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.7053772629999999, 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036958340999999999, 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.731974+0000", 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.731974+0000", 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.731975+0000", 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9200000000000001e-07 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.731994+0000", 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.946e-05 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.732006+0000", 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1697e-05 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.732012+0000", 2024-09-13T15:01:54.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4110000000000001e-06 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.738545+0000", 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065331219999999997 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.738596+0000", 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0921000000000003e-05 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.768788+0000", 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030192044000000001 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:49.768932+0000", 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014409399999999999 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.507 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:105276 8.7 e346/331 8:e81e099a:::benchmark_data_smithi154_261841_object105275:head v 346'6558, mlcod=346'6555)", 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:51.698252+0000", 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.7390996059999999, 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035329590000000001, 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.698252+0000", 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.698252+0000", 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.698253+0000", 2024-09-13T15:01:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4399999999999998e-07 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.698277+0000", 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4085999999999999e-05 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.701379+0000", 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031024780000000001 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.701391+0000", 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1420000000000001e-05 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.712963+0000", 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011572345 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.712982+0000", 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9313999999999998e-05 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.733499+0000", 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020516464000000002 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:51.733581+0000", 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2539000000000004e-05 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:105822 8.7 e347/331 8:e0af96d6:::benchmark_data_smithi154_261841_object105821:head v 347'6602, mlcod=347'6598)", 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:52.339456+0000", 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.0978958009999999, 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033950608, 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339456+0000", 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339456+0000", 2024-09-13T15:01:54.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339457+0000", 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1850000000000001e-06 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339484+0000", 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7520000000000001e-05 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339497+0000", 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3093e-05 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339504+0000", 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.114e-06 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339540+0000", 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5818000000000002e-05 2024-09-13T15:01:54.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.339560+0000", 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9626000000000001e-05 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.373232+0000", 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033671718000000003 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.373406+0000", 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017453400000000001 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:105828 8.7 e347/331 8:e31647d2:::benchmark_data_smithi154_261841_object105827:head v 347'6603, mlcod=347'6600)", 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:52.340818+0000", 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.0965329020000001, 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032726457, 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.520 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.340818+0000", 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.340818+0000", 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.340819+0000", 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2200000000000003e-07 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.340848+0000", 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8717999999999998e-05 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.340863+0000", 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5356e-05 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.340870+0000", 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2990000000000002e-06 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.349806+0000", 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089363370000000008 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.349840+0000", 2024-09-13T15:01:54.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3764999999999999e-05 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.373409+0000", 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023569254000000001 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:52.373545+0000", 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013600600000000001 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.524 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:106684 8.f e347/331 8:f1065ae7:::benchmark_data_smithi154_261841_object106683:head v 347'6577, mlcod=347'6574)", 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:53.334710+0000", 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.102641441, 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039072124999999999, 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.334710+0000", 2024-09-13T15:01:54.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.334710+0000", 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.334711+0000", 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8899999999999998e-07 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.334732+0000", 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0990000000000001e-05 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.334746+0000", 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4404e-05 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.334752+0000", 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4409999999999998e-06 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.337181+0000", 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002428936 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.337225+0000", 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3616999999999999e-05 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.373633+0000", 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036407676 2024-09-13T15:01:54.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:53.373782+0000", 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014927199999999999 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:01:54.529 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:01:54.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:54.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:54.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:01:54.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:54.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:54.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:54.859 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:54.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:01:54.950408+0000 min lat: 0.00096194 max lat: 0.314029 avg lat: 0.0118532 2024-09-13T15:01:54.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:01:54.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 80 16 107982 107966 84.3245 49 0.0244126 0.0118532 2024-09-13T15:01:54.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:54.984 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:54.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:55.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:55.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:55.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:55.242 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:55.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:55.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:55.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.509 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:55.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:55.589 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:55.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:55.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:55.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:55.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:55.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:55.896 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:55.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:55.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 81 16 108848 108832 83.9516 54.125 0.0339069 0.0119061 2024-09-13T15:01:55.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:55 smithi154 ceph-mon[107038]: pgmap v1094: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-13T15:01:55.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:55 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:55.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:55 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:55.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:55 smithi154 ceph-mon[102314]: pgmap v1094: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-13T15:01:55.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:55 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:55.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:55 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:55.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:56.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:56.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:56.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:55 smithi172 ceph-mon[100696]: pgmap v1094: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-13T15:01:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:55 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:55 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:56.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:56.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:56.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:56.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:56.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:56.590 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:56.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:01:56.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:56.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:56.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:56.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:56.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:56.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 82 16 109667 109651 83.5516 51.1875 0.0274783 0.0119631 2024-09-13T15:01:56.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:57.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:57.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:57.177 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:57.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:57.303 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:57.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:57.432 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:01:57.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:57.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:57.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:57.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:57.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:57.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:57.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:57.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:57.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 83 16 110520 110504 83.1872 53.3125 0.0161884 0.0120158 2024-09-13T15:01:57.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[107038]: pgmap v1095: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-13T15:01:57.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[107038]: pgmap v1096: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[102314]: pgmap v1095: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[102314]: pgmap v1096: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:01:57.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:57 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:57.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:57 smithi172 ceph-mon[100696]: pgmap v1095: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-13T15:01:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:01:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:01:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:57 smithi172 ceph-mon[100696]: pgmap v1096: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:01:57.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:57 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:58.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:58.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:58.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:58.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:58.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:58.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:01:58.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:01:58.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:01:58.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:01:58.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:58.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:58.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.863 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:58.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:58.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 84 16 111374 111358 82.8323 53.375 0.0114933 0.0120675 2024-09-13T15:01:58.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:01:58.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[107038]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-13T15:01:58.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:01:58.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:58.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:58.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[102314]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-13T15:01:58.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:01:58.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:58.972 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:58.991 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:58.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:01:59.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:59.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:59.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:59.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:58 smithi172 ceph-mon[100696]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-13T15:01:59.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:58 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:01:59.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:01:59.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:01:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:59.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:01:59.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:01:59.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:01:59.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:01:59.546 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:01:59.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:01:59.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:01:59.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:01:59.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:01:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:01:59.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:01:59.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 85 16 112255 112239 82.5057 55.0625 0.0107696 0.0121147 2024-09-13T15:01:59.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:01:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:01:59.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:00.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:00.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:00.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:00.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:01:59 smithi154 ceph-mon[102314]: pgmap v1097: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 39 MiB/s wr, 621 op/s 2024-09-13T15:02:00.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:01:59 smithi154 ceph-mon[107038]: pgmap v1097: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 39 MiB/s wr, 621 op/s 2024-09-13T15:02:00.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:00.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:01:59 smithi172 ceph-mon[100696]: pgmap v1097: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 39 MiB/s wr, 621 op/s 2024-09-13T15:02:00.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:00.347 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:00.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:00.474 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:00.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:00.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:00.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:00.725 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:00.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:00.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:00.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 86 16 112997 112981 82.0856 46.375 0.00372497 0.0121717 2024-09-13T15:02:00.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:00.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:00.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:01.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:01.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:01.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:01.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:01.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:01.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:01.369 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:01.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:01.509 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:01.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:01.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:01.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:01.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:01.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:01.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:01.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:01.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 87 16 113606 113590 81.5796 38.0625 0.0348578 0.0122519 2024-09-13T15:02:02.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:02.020 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:02:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:02:01] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-13T15:02:02.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:02.100 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:02:02.100 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:02.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:02.139 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:02:02.140 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:02.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:02.176 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:02:02.177 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:02.216 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:02:02.217 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:02.249 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:02:02.249 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:02.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:02.283 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:02:02.283 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:02.284 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:02 smithi154 ceph-mon[102314]: pgmap v1098: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T15:02:02.284 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:02 smithi154 ceph-mon[107038]: pgmap v1098: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T15:02:02.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:02.319 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:02:02.319 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:02.356 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:02:02.356 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:02.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:02.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:02 smithi172 ceph-mon[100696]: pgmap v1098: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T15:02:02.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:02.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:02.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:02.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:02.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:02.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:02.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:02.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 88 16 114369 114353 81.1945 47.6875 0.0140829 0.0123105 2024-09-13T15:02:02.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:02.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:03.018 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:03.045 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:02:03.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:03.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:03.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:03.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:03.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3419371056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:03.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/484967479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:03.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:03 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:03.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3419371056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:03.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/484967479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:03.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:03 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:03.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:03.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:03.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:03.484 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3419371056' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/484967479' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:03 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:02:03.587 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:03.612 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:03.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:03.740 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:03.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:03.866 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:03.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:03.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 89 16 115190 115174 80.8587 51.3125 0.0128241 0.0123614 2024-09-13T15:02:03.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:04.011 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:04.067 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:02:04.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:04.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:04.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:04.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:04.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:04.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[102314]: pgmap v1099: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 52 MiB/s wr, 830 op/s 2024-09-13T15:02:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:02:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[102314]: osdmap e348: 8 total, 8 up, 7 in 2024-09-13T15:02:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3549255388' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[107038]: pgmap v1099: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 52 MiB/s wr, 830 op/s 2024-09-13T15:02:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:02:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[107038]: osdmap e348: 8 total, 8 up, 7 in 2024-09-13T15:02:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:04 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3549255388' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:04 smithi172 ceph-mon[100696]: pgmap v1099: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 52 MiB/s wr, 830 op/s 2024-09-13T15:02:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:04 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:02:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:04 smithi172 ceph-mon[100696]: osdmap e348: 8 total, 8 up, 7 in 2024-09-13T15:02:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:04 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3549255388' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:02:04.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:04.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:04.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:04.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:04.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:04.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:04.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total time run: 90.0089 2024-09-13T15:02:04.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total writes made: 116036 2024-09-13T15:02:04.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Write size: 65536 2024-09-13T15:02:04.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object size: 65536 2024-09-13T15:02:04.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Bandwidth (MB/sec): 80.5726 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Bandwidth: 36.1989 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max bandwidth (MB/sec): 211.438 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min bandwidth (MB/sec): 38.0625 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average IOPS: 1289 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev IOPS: 579.182 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max IOPS: 3383 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min IOPS: 609 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average Latency(s): 0.0124068 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Latency(s): 0.0118036 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max latency(s): 0.314029 2024-09-13T15:02:04.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min latency(s): 0.00096194 2024-09-13T15:02:04.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:04.956 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T15:02:04.982 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:04.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:05.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:05.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:05.184 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:05 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:05.185 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:05 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3549255388' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:05.185 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:05 smithi154 ceph-mon[107038]: osdmap e349: 8 total, 8 up, 7 in 2024-09-13T15:02:05.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:05 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:05.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:05 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3549255388' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:05.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:05 smithi154 ceph-mon[102314]: osdmap e349: 8 total, 8 up, 7 in 2024-09-13T15:02:05.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:05.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:05.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:05.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:05 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:05 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3549255388' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:02:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:05 smithi172 ceph-mon[100696]: osdmap e349: 8 total, 8 up, 7 in 2024-09-13T15:02:05.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:05.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:05.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:02:05.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:05.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:05.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:05.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:05.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:05.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:06.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:06.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:06.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:06.256 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:02:06.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:02:06.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.361 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:06.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:06.386 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:06 smithi154 ceph-mon[102314]: pgmap v1102: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 44 MiB/s wr, 710 op/s 2024-09-13T15:02:06.386 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:06 smithi154 ceph-mon[107038]: pgmap v1102: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 44 MiB/s wr, 710 op/s 2024-09-13T15:02:06.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:06 smithi172 ceph-mon[100696]: pgmap v1102: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 44 MiB/s wr, 710 op/s 2024-09-13T15:02:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:06.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:06.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:06.641 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:06.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:06.767 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:06.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:06.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:06.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:07.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:07.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:07.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:07.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:07.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:07.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:07.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:07.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:07.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:07 smithi154 ceph-mon[102314]: pgmap v1103: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 61 MiB/s wr, 1.01k op/s 2024-09-13T15:02:07.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:07 smithi154 ceph-mon[107038]: pgmap v1103: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 61 MiB/s wr, 1.01k op/s 2024-09-13T15:02:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:07 smithi172 ceph-mon[100696]: pgmap v1103: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 61 MiB/s wr, 1.01k op/s 2024-09-13T15:02:07.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:07.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:07.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:07.712 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:07.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:02:07.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:07.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.841 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:07.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:07.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:07.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:07.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:08.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:08.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:08.116 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:08.116 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:08.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:08.239 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:08.338 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:08.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:08.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:08.413 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:08.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:08.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:08.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:08.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:08.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:08.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:08.889 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:08.914 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:08.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:09.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:09.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:09.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:09.188 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:09.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:02:09.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:09.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:09.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:09.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:09.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:09.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:09.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:09.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:09.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:09.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:09.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:10.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:10.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:10.127 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:10 smithi154 ceph-mon[102314]: pgmap v1104: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 28 MiB/s wr, 478 op/s 2024-09-13T15:02:10.127 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:10 smithi154 ceph-mon[107038]: pgmap v1104: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 28 MiB/s wr, 478 op/s 2024-09-13T15:02:10.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:10.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:10.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:10.304 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:10.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:10.432 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:10.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:10 smithi172 ceph-mon[100696]: pgmap v1104: 89 pgs: 89 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 28 MiB/s wr, 478 op/s 2024-09-13T15:02:10.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:10.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:10.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:10.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:10.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:10.884 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:10.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:10.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:11.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:11.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:11.175 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:11.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:11.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:11.300 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:11.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:11.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:11.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:11.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:11.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:11.721 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:11.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:11.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:11.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:11.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:12.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:12.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:12.142 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:12.156 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:12 smithi154 ceph-mon[102314]: pgmap v1105: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 28 MiB/s wr, 1.00k op/s 2024-09-13T15:02:12.156 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:02:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:02:11] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-13T15:02:12.156 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:12 smithi154 ceph-mon[107038]: pgmap v1105: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 28 MiB/s wr, 1.00k op/s 2024-09-13T15:02:12.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:12.271 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:12.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:12.421 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:12 smithi172 ceph-mon[100696]: pgmap v1105: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 28 MiB/s wr, 1.00k op/s 2024-09-13T15:02:12.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:12.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:12.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:12.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:12.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:12.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:12.906 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:02:12.906 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:12.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:12.921 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:02:12.922 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:12.948 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:12.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:12.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:12.960 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:02:12.961 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:12.997 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:02:12.997 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:13.028 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:02:13.029 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:13.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:13.072 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:02:13.072 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:13.075 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:13.111 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:02:13.112 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:13.147 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:02:13.147 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:13.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:13.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:13.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4151365499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:02:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4151365499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:02:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:13.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:13.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:13.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:13.471 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4151365499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:02:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:13.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:13.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:13.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:13.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:13.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:13.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:13.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:13.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:14.008 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:14.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:14.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:14.229 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:14.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:14.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:14.340 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:14 smithi154 ceph-mon[102314]: pgmap v1106: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 103 B/s rd, 23 MiB/s wr, 808 op/s 2024-09-13T15:02:14.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:14 smithi154 ceph-mon[107038]: pgmap v1106: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 103 B/s rd, 23 MiB/s wr, 808 op/s 2024-09-13T15:02:14.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:14 smithi172 ceph-mon[100696]: pgmap v1106: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 103 B/s rd, 23 MiB/s wr, 808 op/s 2024-09-13T15:02:14.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:14.431 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:14.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:14.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:14.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:14.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:14.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:14.857 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:14.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:14.984 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:14.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:15.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:15.107 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:15.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:15.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:15.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:15.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:15.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:15.506 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:15.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:15.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:15.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:15.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:15.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:15.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:15.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:15.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:16.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:16.051 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:16.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:16.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:16.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:02:16.262 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:02:16.263 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:02:16.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:16.300 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:16.342 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:16 smithi154 ceph-mon[102314]: pgmap v1107: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 93 B/s rd, 21 MiB/s wr, 887 op/s 2024-09-13T15:02:16.342 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:16 smithi154 ceph-mon[107038]: pgmap v1107: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 93 B/s rd, 21 MiB/s wr, 887 op/s 2024-09-13T15:02:16.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:16.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:16 smithi172 ceph-mon[100696]: pgmap v1107: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 93 B/s rd, 21 MiB/s wr, 887 op/s 2024-09-13T15:02:16.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:16.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:16.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:16.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:16.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:16.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:16.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:16.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:17.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:17.075 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:02:17.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.171 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:17.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:17.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:17.297 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:17.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:17.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:17 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2322429236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:17.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:17 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:17 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2322429236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:17 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:17 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2322429236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:17 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:17.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:17.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:17.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:17.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:17.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:17.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:17.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:17.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:17.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:18.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:18.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:18.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:18.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:18.336 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:18.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:18.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[102314]: from='client.35862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[102314]: pgmap v1108: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 11 MiB/s wr, 912 op/s 2024-09-13T15:02:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:18.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[102314]: osdmap e350: 8 total, 8 up, 7 in 2024-09-13T15:02:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[107038]: from='client.35862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[107038]: pgmap v1108: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 11 MiB/s wr, 912 op/s 2024-09-13T15:02:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:18.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:18 smithi154 ceph-mon[107038]: osdmap e350: 8 total, 8 up, 7 in 2024-09-13T15:02:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:18 smithi172 ceph-mon[100696]: from='client.35862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:18 smithi172 ceph-mon[100696]: pgmap v1108: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 11 MiB/s wr, 912 op/s 2024-09-13T15:02:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:18 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:18 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:18 smithi172 ceph-mon[100696]: osdmap e350: 8 total, 8 up, 7 in 2024-09-13T15:02:18.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:18.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.631 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:18.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:18.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:18.756 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:18.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:18.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:18.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:18.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:19.011 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:19.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:19.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:19.181 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:19.192 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1107,"stamp":"2024-09-13T15:02:14.976843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7212898301,"num_objects":110016,"num_object_clones":0,"num_object_copies":330048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110016,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":123209,"num_write_kb":7441346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":3564500,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117568,"ondisk_log_size":117568,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23851000,"kb_used_data":21179840,"kb_used_omap":39,"kb_used_meta":2670808,"kb_avail":632365064,"statfs":{"total":671965249536,"available":647541825536,"internally_reserved":0,"allocated":21688156160,"data_stored":21674929534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40275,"internal_metadata":2734908077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-158728168,"num_objects":-2421,"num_object_clones":0,"num_object_copies":-7263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2421,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9677,"num_write_kb":232129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.902769"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.085727+0000","last_change":"2024-09-13T15:01:14.837554+0000","last_active":"2024-09-13T15:02:04.085727+0000","last_peered":"2024-09-13T15:02:04.085727+0000","last_clean":"2024-09-13T15:02:04.085727+0000","last_became_active":"2024-09-13T15:01:14.837236+0000","last_became_peered":"2024-09-13T15:01:14.837236+0000","last_unstale":"2024-09-13T15:02:04.085727+0000","last_undegraded":"2024-09-13T15:02:04.085727+0000","last_fullsized":"2024-09-13T15:02:04.085727+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:18.510952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":91,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.039887+0000","last_change":"2024-09-13T15:00:15.944713+0000","last_active":"2024-09-13T15:02:03.039887+0000","last_peered":"2024-09-13T15:02:03.039887+0000","last_clean":"2024-09-13T15:02:03.039887+0000","last_became_active":"2024-09-13T15:00:15.944353+0000","last_became_peered":"2024-09-13T15:00:15.944353+0000","last_unstale":"2024-09-13T15:02:03.039887+0000","last_undegraded":"2024-09-13T15:02:03.039887+0000","last_fullsized":"2024-09-13T15:02:03.039887+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:22.078466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":197,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032897+0000","last_change":"2024-09-13T14:57:56.258528+0000","last_active":"2024-09-13T15:02:03.032897+0000","last_peered":"2024-09-13T15:02:03.032897+0000","last_clean":"2024-09-13T15:02:03.032897+0000","last_became_active":"2024-09-13T14:57:56.258116+0000","last_became_peered":"2024-09-13T14:57:56.258116+0000","last_unstale":"2024-09-13T15:02:03.032897+0000","last_undegraded":"2024-09-13T15:02:03.032897+0000","last_fullsized":"2024-09-13T15:02:03.032897+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:13.267817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":520,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.083076+0000","last_change":"2024-09-13T14:59:42.555312+0000","last_active":"2024-09-13T15:02:04.083076+0000","last_peered":"2024-09-13T15:02:04.083076+0000","last_clean":"2024-09-13T15:02:04.083076+0000","last_became_active":"2024-09-13T14:59:42.554397+0000","last_became_peered":"2024-09-13T14:59:42.554397+0000","last_unstale":"2024-09-13T15:02:04.083076+0000","last_undegraded":"2024-09-13T15:02:04.083076+0000","last_fullsized":"2024-09-13T15:02:04.083076+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:26.418960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":524,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.087389+0000","last_change":"2024-09-13T14:59:42.572046+0000","last_active":"2024-09-13T15:02:04.087389+0000","last_peered":"2024-09-13T15:02:04.087389+0000","last_clean":"2024-09-13T15:02:04.087389+0000","last_became_active":"2024-09-13T14:59:42.565943+0000","last_became_peered":"2024-09-13T14:59:42.565943+0000","last_unstale":"2024-09-13T15:02:04.087389+0000","last_undegraded":"2024-09-13T15:02:04.087389+0000","last_fullsized":"2024-09-13T15:02:04.087389+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:46.240539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":94,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.039644+0000","last_change":"2024-09-13T14:59:43.546919+0000","last_active":"2024-09-13T15:02:03.039644+0000","last_peered":"2024-09-13T15:02:03.039644+0000","last_clean":"2024-09-13T15:02:03.039644+0000","last_became_active":"2024-09-13T14:59:43.546633+0000","last_became_peered":"2024-09-13T14:59:43.546633+0000","last_unstale":"2024-09-13T15:02:03.039644+0000","last_undegraded":"2024-09-13T15:02:03.039644+0000","last_fullsized":"2024-09-13T15:02:03.039644+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:34.351065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.075009+0000","last_change":"2024-09-13T15:01:17.867842+0000","last_active":"2024-09-13T15:02:04.075009+0000","last_peered":"2024-09-13T15:02:04.075009+0000","last_clean":"2024-09-13T15:02:04.075009+0000","last_became_active":"2024-09-13T15:01:17.867444+0000","last_became_peered":"2024-09-13T15:01:17.867444+0000","last_unstale":"2024-09-13T15:02:04.075009+0000","last_undegraded":"2024-09-13T15:02:04.075009+0000","last_fullsized":"2024-09-13T15:02:04.075009+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:14.910223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":127,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.082088+0000","last_change":"2024-09-13T14:59:43.549396+0000","last_active":"2024-09-13T15:02:04.082088+0000","last_peered":"2024-09-13T15:02:04.082088+0000","last_clean":"2024-09-13T15:02:04.082088+0000","last_became_active":"2024-09-13T14:59:43.548994+0000","last_became_peered":"2024-09-13T14:59:43.548994+0000","last_unstale":"2024-09-13T15:02:04.082088+0000","last_undegraded":"2024-09-13T15:02:04.082088+0000","last_fullsized":"2024-09-13T15:02:04.082088+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:51.433399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"349'7793","reported_seq":7820,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:14.752027+0000","last_change":"2024-09-13T15:00:24.023321+0000","last_active":"2024-09-13T15:02:14.752027+0000","last_peered":"2024-09-13T15:02:14.752027+0000","last_clean":"2024-09-13T15:02:14.752027+0000","last_became_active":"2024-09-13T15:00:24.022952+0000","last_became_peered":"2024-09-13T15:00:24.022952+0000","last_unstale":"2024-09-13T15:02:14.752027+0000","last_undegraded":"2024-09-13T15:02:14.752027+0000","last_fullsized":"2024-09-13T15:02:14.752027+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7793,"log_dups_size":0,"ondisk_log_size":7793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:34.183228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442564608,"num_objects":6753,"num_object_clones":0,"num_object_copies":20259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7793,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":105,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.040104+0000","last_change":"2024-09-13T15:00:15.946088+0000","last_active":"2024-09-13T15:02:03.040104+0000","last_peered":"2024-09-13T15:02:03.040104+0000","last_clean":"2024-09-13T15:02:03.040104+0000","last_became_active":"2024-09-13T15:00:15.945399+0000","last_became_peered":"2024-09-13T15:00:15.945399+0000","last_unstale":"2024-09-13T15:02:03.040104+0000","last_undegraded":"2024-09-13T15:02:03.040104+0000","last_fullsized":"2024-09-13T15:02:03.040104+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:20.043020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":234,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032636+0000","last_change":"2024-09-13T14:59:04.217165+0000","last_active":"2024-09-13T15:02:03.032636+0000","last_peered":"2024-09-13T15:02:03.032636+0000","last_clean":"2024-09-13T15:02:03.032636+0000","last_became_active":"2024-09-13T14:59:04.216859+0000","last_became_peered":"2024-09-13T14:59:04.216859+0000","last_unstale":"2024-09-13T15:02:03.032636+0000","last_undegraded":"2024-09-13T15:02:03.032636+0000","last_fullsized":"2024-09-13T15:02:03.032636+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:00.417627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"349'7740","reported_seq":7764,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:14.389308+0000","last_change":"2024-09-13T15:00:24.023029+0000","last_active":"2024-09-13T15:02:14.389308+0000","last_peered":"2024-09-13T15:02:14.389308+0000","last_clean":"2024-09-13T15:02:14.389308+0000","last_became_active":"2024-09-13T15:00:24.022751+0000","last_became_peered":"2024-09-13T15:00:24.022751+0000","last_unstale":"2024-09-13T15:02:14.389308+0000","last_undegraded":"2024-09-13T15:02:14.389308+0000","last_fullsized":"2024-09-13T15:02:14.389308+0000","mapping_epoch":331,"log_start":"336'200","ondisk_log_start":"336'200","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:57.881826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441450496,"num_objects":6736,"num_object_clones":0,"num_object_copies":20208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7740,"num_write_kb":463232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":78,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.040041+0000","last_change":"2024-09-13T14:57:56.273194+0000","last_active":"2024-09-13T15:02:03.040041+0000","last_peered":"2024-09-13T15:02:03.040041+0000","last_clean":"2024-09-13T15:02:03.040041+0000","last_became_active":"2024-09-13T14:57:56.259881+0000","last_became_peered":"2024-09-13T14:57:56.259881+0000","last_unstale":"2024-09-13T15:02:03.040041+0000","last_undegraded":"2024-09-13T15:02:03.040041+0000","last_fullsized":"2024-09-13T15:02:03.040041+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:28.683153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":411,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.035991+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:02:03.035991+0000","last_peered":"2024-09-13T15:02:03.035991+0000","last_clean":"2024-09-13T15:02:03.035991+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:02:03.035991+0000","last_undegraded":"2024-09-13T15:02:03.035991+0000","last_fullsized":"2024-09-13T15:02:03.035991+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"349'7560","reported_seq":7607,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.698771+0000","last_change":"2024-09-13T15:00:24.024661+0000","last_active":"2024-09-13T15:02:10.698771+0000","last_peered":"2024-09-13T15:02:10.698771+0000","last_clean":"2024-09-13T15:02:10.698771+0000","last_became_active":"2024-09-13T15:00:24.023841+0000","last_became_peered":"2024-09-13T15:00:24.023841+0000","last_unstale":"2024-09-13T15:02:10.698771+0000","last_undegraded":"2024-09-13T15:02:10.698771+0000","last_fullsized":"2024-09-13T15:02:10.698771+0000","mapping_epoch":331,"log_start":"337'400","ondisk_log_start":"337'400","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7160,"log_dups_size":0,"ondisk_log_size":7160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:34.065023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454164480,"num_objects":6930,"num_object_clones":0,"num_object_copies":20790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":91,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.039752+0000","last_change":"2024-09-13T14:59:04.223552+0000","last_active":"2024-09-13T15:02:03.039752+0000","last_peered":"2024-09-13T15:02:03.039752+0000","last_clean":"2024-09-13T15:02:03.039752+0000","last_became_active":"2024-09-13T14:59:04.216127+0000","last_became_peered":"2024-09-13T14:59:04.216127+0000","last_unstale":"2024-09-13T15:02:03.039752+0000","last_undegraded":"2024-09-13T15:02:03.039752+0000","last_fullsized":"2024-09-13T15:02:03.039752+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:48.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":252,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.073948+0000","last_change":"2024-09-13T14:59:04.211285+0000","last_active":"2024-09-13T15:02:04.073948+0000","last_peered":"2024-09-13T15:02:04.073948+0000","last_clean":"2024-09-13T15:02:04.073948+0000","last_became_active":"2024-09-13T14:59:04.210929+0000","last_became_peered":"2024-09-13T14:59:04.210929+0000","last_unstale":"2024-09-13T15:02:04.073948+0000","last_undegraded":"2024-09-13T15:02:04.073948+0000","last_fullsized":"2024-09-13T15:02:04.073948+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:11.009140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"349'7564","reported_seq":7611,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.439063+0000","last_change":"2024-09-13T15:00:24.022660+0000","last_active":"2024-09-13T15:02:10.439063+0000","last_peered":"2024-09-13T15:02:10.439063+0000","last_clean":"2024-09-13T15:02:10.439063+0000","last_became_active":"2024-09-13T15:00:24.022328+0000","last_became_peered":"2024-09-13T15:00:24.022328+0000","last_unstale":"2024-09-13T15:02:10.439063+0000","last_undegraded":"2024-09-13T15:02:10.439063+0000","last_fullsized":"2024-09-13T15:02:10.439063+0000","mapping_epoch":331,"log_start":"337'500","ondisk_log_start":"337'500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:12.849154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457441280,"num_objects":6980,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7564,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":214,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.033048+0000","last_change":"2024-09-13T14:59:04.216946+0000","last_active":"2024-09-13T15:02:03.033048+0000","last_peered":"2024-09-13T15:02:03.033048+0000","last_clean":"2024-09-13T15:02:03.033048+0000","last_became_active":"2024-09-13T14:59:04.216628+0000","last_became_peered":"2024-09-13T14:59:04.216628+0000","last_unstale":"2024-09-13T15:02:03.033048+0000","last_undegraded":"2024-09-13T15:02:03.033048+0000","last_fullsized":"2024-09-13T15:02:03.033048+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:50.914653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":446,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.082246+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:02:04.082246+0000","last_peered":"2024-09-13T15:02:04.082246+0000","last_clean":"2024-09-13T15:02:04.082246+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:02:04.082246+0000","last_undegraded":"2024-09-13T15:02:04.082246+0000","last_fullsized":"2024-09-13T15:02:04.082246+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"349'7506","reported_seq":7553,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.167519+0000","last_change":"2024-09-13T15:00:23.973213+0000","last_active":"2024-09-13T15:02:10.167519+0000","last_peered":"2024-09-13T15:02:10.167519+0000","last_clean":"2024-09-13T15:02:10.167519+0000","last_became_active":"2024-09-13T15:00:23.972885+0000","last_became_peered":"2024-09-13T15:00:23.972885+0000","last_unstale":"2024-09-13T15:02:10.167519+0000","last_undegraded":"2024-09-13T15:02:10.167519+0000","last_fullsized":"2024-09-13T15:02:10.167519+0000","mapping_epoch":331,"log_start":"337'600","ondisk_log_start":"337'600","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:22.288214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457572352,"num_objects":6982,"num_object_clones":0,"num_object_copies":20946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7506,"num_write_kb":463616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":572,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.076741+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:02:04.076741+0000","last_peered":"2024-09-13T15:02:04.076741+0000","last_clean":"2024-09-13T15:02:04.076741+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:02:04.076741+0000","last_undegraded":"2024-09-13T15:02:04.076741+0000","last_fullsized":"2024-09-13T15:02:04.076741+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":345,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077530+0000","last_change":"2024-09-13T14:59:04.213062+0000","last_active":"2024-09-13T15:02:04.077530+0000","last_peered":"2024-09-13T15:02:04.077530+0000","last_clean":"2024-09-13T15:02:04.077530+0000","last_became_active":"2024-09-13T14:59:04.212776+0000","last_became_peered":"2024-09-13T14:59:04.212776+0000","last_unstale":"2024-09-13T15:02:04.077530+0000","last_undegraded":"2024-09-13T15:02:04.077530+0000","last_fullsized":"2024-09-13T15:02:04.077530+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:49.898651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"349'7645","reported_seq":7692,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.430541+0000","last_change":"2024-09-13T15:00:24.023172+0000","last_active":"2024-09-13T15:02:10.430541+0000","last_peered":"2024-09-13T15:02:10.430541+0000","last_clean":"2024-09-13T15:02:10.430541+0000","last_became_active":"2024-09-13T15:00:24.022574+0000","last_became_peered":"2024-09-13T15:00:24.022574+0000","last_unstale":"2024-09-13T15:02:10.430541+0000","last_undegraded":"2024-09-13T15:02:10.430541+0000","last_fullsized":"2024-09-13T15:02:10.430541+0000","mapping_epoch":331,"log_start":"337'600","ondisk_log_start":"337'600","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:06.934046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460128256,"num_objects":7021,"num_object_clones":0,"num_object_copies":21063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":142,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036412+0000","last_change":"2024-09-13T14:59:04.217247+0000","last_active":"2024-09-13T15:02:03.036412+0000","last_peered":"2024-09-13T15:02:03.036412+0000","last_clean":"2024-09-13T15:02:03.036412+0000","last_became_active":"2024-09-13T14:59:04.216574+0000","last_became_peered":"2024-09-13T14:59:04.216574+0000","last_unstale":"2024-09-13T15:02:03.036412+0000","last_undegraded":"2024-09-13T15:02:03.036412+0000","last_fullsized":"2024-09-13T15:02:03.036412+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:00:26.362085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":348,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077924+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:02:04.077924+0000","last_peered":"2024-09-13T15:02:04.077924+0000","last_clean":"2024-09-13T15:02:04.077924+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:02:04.077924+0000","last_undegraded":"2024-09-13T15:02:04.077924+0000","last_fullsized":"2024-09-13T15:02:04.077924+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"349'7832","reported_seq":7859,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:14.357391+0000","last_change":"2024-09-13T15:00:23.972644+0000","last_active":"2024-09-13T15:02:14.357391+0000","last_peered":"2024-09-13T15:02:14.357391+0000","last_clean":"2024-09-13T15:02:14.357391+0000","last_became_active":"2024-09-13T15:00:23.972370+0000","last_became_peered":"2024-09-13T15:00:23.972370+0000","last_unstale":"2024-09-13T15:02:14.357391+0000","last_undegraded":"2024-09-13T15:02:14.357391+0000","last_fullsized":"2024-09-13T15:02:14.357391+0000","mapping_epoch":331,"log_start":"337'300","ondisk_log_start":"337'300","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7532,"log_dups_size":0,"ondisk_log_size":7532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:55.756118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442499072,"num_objects":6752,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7832,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":354,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036069+0000","last_change":"2024-09-13T15:00:15.945718+0000","last_active":"2024-09-13T15:02:03.036069+0000","last_peered":"2024-09-13T15:02:03.036069+0000","last_clean":"2024-09-13T15:02:03.036069+0000","last_became_active":"2024-09-13T15:00:15.944400+0000","last_became_peered":"2024-09-13T15:00:15.944400+0000","last_unstale":"2024-09-13T15:02:03.036069+0000","last_undegraded":"2024-09-13T15:02:03.036069+0000","last_fullsized":"2024-09-13T15:02:03.036069+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:23:47.992626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":372,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077723+0000","last_change":"2024-09-13T14:59:04.211136+0000","last_active":"2024-09-13T15:02:04.077723+0000","last_peered":"2024-09-13T15:02:04.077723+0000","last_clean":"2024-09-13T15:02:04.077723+0000","last_became_active":"2024-09-13T14:59:04.210848+0000","last_became_peered":"2024-09-13T14:59:04.210848+0000","last_unstale":"2024-09-13T15:02:04.077723+0000","last_undegraded":"2024-09-13T15:02:04.077723+0000","last_fullsized":"2024-09-13T15:02:04.077723+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:07.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"349'7498","reported_seq":7545,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.437509+0000","last_change":"2024-09-13T15:00:24.023758+0000","last_active":"2024-09-13T15:02:10.437509+0000","last_peered":"2024-09-13T15:02:10.437509+0000","last_clean":"2024-09-13T15:02:10.437509+0000","last_became_active":"2024-09-13T15:00:24.022468+0000","last_became_peered":"2024-09-13T15:00:24.022468+0000","last_unstale":"2024-09-13T15:02:10.437509+0000","last_undegraded":"2024-09-13T15:02:10.437509+0000","last_fullsized":"2024-09-13T15:02:10.437509+0000","mapping_epoch":331,"log_start":"337'500","ondisk_log_start":"337'500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:31.767944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455081984,"num_objects":6944,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.074364+0000","last_change":"2024-09-13T15:01:17.859549+0000","last_active":"2024-09-13T15:02:04.074364+0000","last_peered":"2024-09-13T15:02:04.074364+0000","last_clean":"2024-09-13T15:02:04.074364+0000","last_became_active":"2024-09-13T15:01:17.858796+0000","last_became_peered":"2024-09-13T15:01:17.858796+0000","last_unstale":"2024-09-13T15:02:04.074364+0000","last_undegraded":"2024-09-13T15:02:04.074364+0000","last_fullsized":"2024-09-13T15:02:04.074364+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:32.898026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":317,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.087631+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:02:04.087631+0000","last_peered":"2024-09-13T15:02:04.087631+0000","last_clean":"2024-09-13T15:02:04.087631+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:02:04.087631+0000","last_undegraded":"2024-09-13T15:02:04.087631+0000","last_fullsized":"2024-09-13T15:02:04.087631+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"349'7723","reported_seq":7750,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:14.769944+0000","last_change":"2024-09-13T15:00:23.972864+0000","last_active":"2024-09-13T15:02:14.769944+0000","last_peered":"2024-09-13T15:02:14.769944+0000","last_clean":"2024-09-13T15:02:14.769944+0000","last_became_active":"2024-09-13T15:00:23.972539+0000","last_became_peered":"2024-09-13T15:00:23.972539+0000","last_unstale":"2024-09-13T15:02:14.769944+0000","last_undegraded":"2024-09-13T15:02:14.769944+0000","last_fullsized":"2024-09-13T15:02:14.769944+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:22.430857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439156736,"num_objects":6701,"num_object_clones":0,"num_object_copies":20103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7723,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":737,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.039444+0000","last_change":"2024-09-13T14:59:43.552438+0000","last_active":"2024-09-13T15:02:03.039444+0000","last_peered":"2024-09-13T15:02:03.039444+0000","last_clean":"2024-09-13T15:02:03.039444+0000","last_became_active":"2024-09-13T14:59:43.551943+0000","last_became_peered":"2024-09-13T14:59:43.551943+0000","last_unstale":"2024-09-13T15:02:03.039444+0000","last_undegraded":"2024-09-13T15:02:03.039444+0000","last_fullsized":"2024-09-13T15:02:03.039444+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:18.178351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":969,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032827+0000","last_change":"2024-09-13T15:00:15.945425+0000","last_active":"2024-09-13T15:02:03.032827+0000","last_peered":"2024-09-13T15:02:03.032827+0000","last_clean":"2024-09-13T15:02:03.032827+0000","last_became_active":"2024-09-13T15:00:15.944987+0000","last_became_peered":"2024-09-13T15:00:15.944987+0000","last_unstale":"2024-09-13T15:02:03.032827+0000","last_undegraded":"2024-09-13T15:02:03.032827+0000","last_fullsized":"2024-09-13T15:02:03.032827+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:58.158525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":526,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.076531+0000","last_change":"2024-09-13T14:59:04.214511+0000","last_active":"2024-09-13T15:02:04.076531+0000","last_peered":"2024-09-13T15:02:04.076531+0000","last_clean":"2024-09-13T15:02:04.076531+0000","last_became_active":"2024-09-13T14:59:04.214185+0000","last_became_peered":"2024-09-13T14:59:04.214185+0000","last_unstale":"2024-09-13T15:02:04.076531+0000","last_undegraded":"2024-09-13T15:02:04.076531+0000","last_fullsized":"2024-09-13T15:02:04.076531+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:33.504798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":172,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.037833+0000","last_change":"2024-09-13T14:59:43.549931+0000","last_active":"2024-09-13T15:02:03.037833+0000","last_peered":"2024-09-13T15:02:03.037833+0000","last_clean":"2024-09-13T15:02:03.037833+0000","last_became_active":"2024-09-13T14:59:43.549564+0000","last_became_peered":"2024-09-13T14:59:43.549564+0000","last_unstale":"2024-09-13T15:02:03.037833+0000","last_undegraded":"2024-09-13T15:02:03.037833+0000","last_fullsized":"2024-09-13T15:02:03.037833+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:10.194065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"349'7518","reported_seq":7565,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.371175+0000","last_change":"2024-09-13T15:00:24.024002+0000","last_active":"2024-09-13T15:02:10.371175+0000","last_peered":"2024-09-13T15:02:10.371175+0000","last_clean":"2024-09-13T15:02:10.371175+0000","last_became_active":"2024-09-13T15:00:24.023708+0000","last_became_peered":"2024-09-13T15:00:24.023708+0000","last_unstale":"2024-09-13T15:02:10.371175+0000","last_undegraded":"2024-09-13T15:02:10.371175+0000","last_fullsized":"2024-09-13T15:02:10.371175+0000","mapping_epoch":331,"log_start":"337'500","ondisk_log_start":"337'500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:46.507079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454164480,"num_objects":6930,"num_object_clones":0,"num_object_copies":20790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7518,"num_write_kb":462336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":108,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.039730+0000","last_change":"2024-09-13T14:59:42.568409+0000","last_active":"2024-09-13T15:02:03.039730+0000","last_peered":"2024-09-13T15:02:03.039730+0000","last_clean":"2024-09-13T15:02:03.039730+0000","last_became_active":"2024-09-13T14:59:42.567972+0000","last_became_peered":"2024-09-13T14:59:42.567972+0000","last_unstale":"2024-09-13T15:02:03.039730+0000","last_undegraded":"2024-09-13T15:02:03.039730+0000","last_fullsized":"2024-09-13T15:02:03.039730+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:21.721692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":520,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.090767+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:02:04.090767+0000","last_peered":"2024-09-13T15:02:04.090767+0000","last_clean":"2024-09-13T15:02:04.090767+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:02:04.090767+0000","last_undegraded":"2024-09-13T15:02:04.090767+0000","last_fullsized":"2024-09-13T15:02:04.090767+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":515,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.086985+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:02:04.086985+0000","last_peered":"2024-09-13T15:02:04.086985+0000","last_clean":"2024-09-13T15:02:04.086985+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:02:04.086985+0000","last_undegraded":"2024-09-13T15:02:04.086985+0000","last_fullsized":"2024-09-13T15:02:04.086985+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"349'7608","reported_seq":7653,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.120672+0000","last_change":"2024-09-13T15:00:23.973296+0000","last_active":"2024-09-13T15:02:10.120672+0000","last_peered":"2024-09-13T15:02:10.120672+0000","last_clean":"2024-09-13T15:02:10.120672+0000","last_became_active":"2024-09-13T15:00:23.972661+0000","last_became_peered":"2024-09-13T15:00:23.972661+0000","last_unstale":"2024-09-13T15:02:10.120672+0000","last_undegraded":"2024-09-13T15:02:10.120672+0000","last_fullsized":"2024-09-13T15:02:10.120672+0000","mapping_epoch":331,"log_start":"337'700","ondisk_log_start":"337'700","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:58.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752000,"num_objects":7000,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7608,"num_write_kb":467456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":954,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.085531+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:02:04.085531+0000","last_peered":"2024-09-13T15:02:04.085531+0000","last_clean":"2024-09-13T15:02:04.085531+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:02:04.085531+0000","last_undegraded":"2024-09-13T15:02:04.085531+0000","last_fullsized":"2024-09-13T15:02:04.085531+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":576,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.083305+0000","last_change":"2024-09-13T15:00:15.945376+0000","last_active":"2024-09-13T15:02:04.083305+0000","last_peered":"2024-09-13T15:02:04.083305+0000","last_clean":"2024-09-13T15:02:04.083305+0000","last_became_active":"2024-09-13T15:00:15.944813+0000","last_became_peered":"2024-09-13T15:00:15.944813+0000","last_unstale":"2024-09-13T15:02:04.083305+0000","last_undegraded":"2024-09-13T15:02:04.083305+0000","last_fullsized":"2024-09-13T15:02:04.083305+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:02.237663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":321,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.073766+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:02:04.073766+0000","last_peered":"2024-09-13T15:02:04.073766+0000","last_clean":"2024-09-13T15:02:04.073766+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:02:04.073766+0000","last_undegraded":"2024-09-13T15:02:04.073766+0000","last_fullsized":"2024-09-13T15:02:04.073766+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"349'7760","reported_seq":7787,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:14.773746+0000","last_change":"2024-09-13T15:00:24.022623+0000","last_active":"2024-09-13T15:02:14.773746+0000","last_peered":"2024-09-13T15:02:14.773746+0000","last_clean":"2024-09-13T15:02:14.773746+0000","last_became_active":"2024-09-13T15:00:24.022333+0000","last_became_peered":"2024-09-13T15:00:24.022333+0000","last_unstale":"2024-09-13T15:02:14.773746+0000","last_undegraded":"2024-09-13T15:02:14.773746+0000","last_fullsized":"2024-09-13T15:02:14.773746+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:24.771552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437911552,"num_objects":6682,"num_object_clones":0,"num_object_copies":20046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":245,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032977+0000","last_change":"2024-09-13T14:59:43.578518+0000","last_active":"2024-09-13T15:02:03.032977+0000","last_peered":"2024-09-13T15:02:03.032977+0000","last_clean":"2024-09-13T15:02:03.032977+0000","last_became_active":"2024-09-13T14:59:43.552215+0000","last_became_peered":"2024-09-13T14:59:43.552215+0000","last_unstale":"2024-09-13T15:02:03.032977+0000","last_undegraded":"2024-09-13T15:02:03.032977+0000","last_fullsized":"2024-09-13T15:02:03.032977+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:29.168795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":364,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036505+0000","last_change":"2024-09-13T14:59:43.550004+0000","last_active":"2024-09-13T15:02:03.036505+0000","last_peered":"2024-09-13T15:02:03.036505+0000","last_clean":"2024-09-13T15:02:03.036505+0000","last_became_active":"2024-09-13T14:59:43.549533+0000","last_became_peered":"2024-09-13T14:59:43.549533+0000","last_unstale":"2024-09-13T15:02:03.036505+0000","last_undegraded":"2024-09-13T15:02:03.036505+0000","last_fullsized":"2024-09-13T15:02:03.036505+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:00.462528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":132,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032745+0000","last_change":"2024-09-13T15:00:14.949848+0000","last_active":"2024-09-13T15:02:03.032745+0000","last_peered":"2024-09-13T15:02:03.032745+0000","last_clean":"2024-09-13T15:02:03.032745+0000","last_became_active":"2024-09-13T15:00:14.949644+0000","last_became_peered":"2024-09-13T15:00:14.949644+0000","last_unstale":"2024-09-13T15:02:03.032745+0000","last_undegraded":"2024-09-13T15:02:03.032745+0000","last_fullsized":"2024-09-13T15:02:03.032745+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:17.762542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"349'7671","reported_seq":7718,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.824212+0000","last_change":"2024-09-13T15:00:24.024125+0000","last_active":"2024-09-13T15:02:10.824212+0000","last_peered":"2024-09-13T15:02:10.824212+0000","last_clean":"2024-09-13T15:02:10.824212+0000","last_became_active":"2024-09-13T15:00:24.023801+0000","last_became_peered":"2024-09-13T15:00:24.023801+0000","last_unstale":"2024-09-13T15:02:10.824212+0000","last_undegraded":"2024-09-13T15:02:10.824212+0000","last_fullsized":"2024-09-13T15:02:10.824212+0000","mapping_epoch":331,"log_start":"337'500","ondisk_log_start":"337'500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7171,"log_dups_size":0,"ondisk_log_size":7171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.418853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459997184,"num_objects":7019,"num_object_clones":0,"num_object_copies":21057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7671,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":222,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032868+0000","last_change":"2024-09-13T15:00:14.948015+0000","last_active":"2024-09-13T15:02:03.032868+0000","last_peered":"2024-09-13T15:02:03.032868+0000","last_clean":"2024-09-13T15:02:03.032868+0000","last_became_active":"2024-09-13T15:00:14.947799+0000","last_became_peered":"2024-09-13T15:00:14.947799+0000","last_unstale":"2024-09-13T15:02:03.032868+0000","last_undegraded":"2024-09-13T15:02:03.032868+0000","last_fullsized":"2024-09-13T15:02:03.032868+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:37.576480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1163,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.073910+0000","last_change":"2024-09-13T14:59:42.570065+0000","last_active":"2024-09-13T15:02:04.073910+0000","last_peered":"2024-09-13T15:02:04.073910+0000","last_clean":"2024-09-13T15:02:04.073910+0000","last_became_active":"2024-09-13T14:59:42.569452+0000","last_became_peered":"2024-09-13T14:59:42.569452+0000","last_unstale":"2024-09-13T15:02:04.073910+0000","last_undegraded":"2024-09-13T15:02:04.073910+0000","last_fullsized":"2024-09-13T15:02:04.073910+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:05.305260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":165,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.082913+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:02:04.082913+0000","last_peered":"2024-09-13T15:02:04.082913+0000","last_clean":"2024-09-13T15:02:04.082913+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:02:04.082913+0000","last_undegraded":"2024-09-13T15:02:04.082913+0000","last_fullsized":"2024-09-13T15:02:04.082913+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"349'7432","reported_seq":7479,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.195079+0000","last_change":"2024-09-13T15:00:24.024834+0000","last_active":"2024-09-13T15:02:10.195079+0000","last_peered":"2024-09-13T15:02:10.195079+0000","last_clean":"2024-09-13T15:02:10.195079+0000","last_became_active":"2024-09-13T15:00:24.024491+0000","last_became_peered":"2024-09-13T15:00:24.024491+0000","last_unstale":"2024-09-13T15:02:10.195079+0000","last_undegraded":"2024-09-13T15:02:10.195079+0000","last_fullsized":"2024-09-13T15:02:10.195079+0000","mapping_epoch":331,"log_start":"337'600","ondisk_log_start":"337'600","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6832,"log_dups_size":0,"ondisk_log_size":6832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:35.133306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452460544,"num_objects":6904,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7432,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":575,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.073308+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:02:04.073308+0000","last_peered":"2024-09-13T15:02:04.073308+0000","last_clean":"2024-09-13T15:02:04.073308+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:02:04.073308+0000","last_undegraded":"2024-09-13T15:02:04.073308+0000","last_fullsized":"2024-09-13T15:02:04.073308+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":108,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.089051+0000","last_change":"2024-09-13T14:59:42.576338+0000","last_active":"2024-09-13T15:02:04.089051+0000","last_peered":"2024-09-13T15:02:04.089051+0000","last_clean":"2024-09-13T15:02:04.089051+0000","last_became_active":"2024-09-13T14:59:42.575875+0000","last_became_peered":"2024-09-13T14:59:42.575875+0000","last_unstale":"2024-09-13T15:02:04.089051+0000","last_undegraded":"2024-09-13T15:02:04.089051+0000","last_fullsized":"2024-09-13T15:02:04.089051+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:51.037045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":510,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.082361+0000","last_change":"2024-09-13T14:59:42.572653+0000","last_active":"2024-09-13T15:02:04.082361+0000","last_peered":"2024-09-13T15:02:04.082361+0000","last_clean":"2024-09-13T15:02:04.082361+0000","last_became_active":"2024-09-13T14:59:42.572326+0000","last_became_peered":"2024-09-13T14:59:42.572326+0000","last_unstale":"2024-09-13T15:02:04.082361+0000","last_undegraded":"2024-09-13T15:02:04.082361+0000","last_fullsized":"2024-09-13T15:02:04.082361+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:02.284316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"349'7536","reported_seq":7583,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:10.784604+0000","last_change":"2024-09-13T15:00:24.023353+0000","last_active":"2024-09-13T15:02:10.784604+0000","last_peered":"2024-09-13T15:02:10.784604+0000","last_clean":"2024-09-13T15:02:10.784604+0000","last_became_active":"2024-09-13T15:00:24.023074+0000","last_became_peered":"2024-09-13T15:00:24.023074+0000","last_unstale":"2024-09-13T15:02:10.784604+0000","last_undegraded":"2024-09-13T15:02:10.784604+0000","last_fullsized":"2024-09-13T15:02:10.784604+0000","mapping_epoch":331,"log_start":"337'300","ondisk_log_start":"337'300","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7236,"log_dups_size":0,"ondisk_log_size":7236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:50.987256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454557696,"num_objects":6936,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7536,"num_write_kb":463104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":605,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.080551+0000","last_change":"2024-09-13T14:59:42.561566+0000","last_active":"2024-09-13T15:02:04.080551+0000","last_peered":"2024-09-13T15:02:04.080551+0000","last_clean":"2024-09-13T15:02:04.080551+0000","last_became_active":"2024-09-13T14:59:42.561224+0000","last_became_peered":"2024-09-13T14:59:42.561224+0000","last_unstale":"2024-09-13T15:02:04.080551+0000","last_undegraded":"2024-09-13T15:02:04.080551+0000","last_fullsized":"2024-09-13T15:02:04.080551+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.918624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1037,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.073744+0000","last_change":"2024-09-13T14:59:07.221776+0000","last_active":"2024-09-13T15:02:04.073744+0000","last_peered":"2024-09-13T15:02:04.073744+0000","last_clean":"2024-09-13T15:02:04.073744+0000","last_became_active":"2024-09-13T14:59:07.221231+0000","last_became_peered":"2024-09-13T14:59:07.221231+0000","last_unstale":"2024-09-13T15:02:04.073744+0000","last_undegraded":"2024-09-13T15:02:04.073744+0000","last_fullsized":"2024-09-13T15:02:04.073744+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:18.106031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":317,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.074800+0000","last_change":"2024-09-13T14:59:42.561395+0000","last_active":"2024-09-13T15:02:04.074800+0000","last_peered":"2024-09-13T15:02:04.074800+0000","last_clean":"2024-09-13T15:02:04.074800+0000","last_became_active":"2024-09-13T14:59:42.561100+0000","last_became_peered":"2024-09-13T14:59:42.561100+0000","last_unstale":"2024-09-13T15:02:04.074800+0000","last_undegraded":"2024-09-13T15:02:04.074800+0000","last_fullsized":"2024-09-13T15:02:04.074800+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:31.681550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"349'7700","reported_seq":7728,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:14.778605+0000","last_change":"2024-09-13T15:00:24.023000+0000","last_active":"2024-09-13T15:02:14.778605+0000","last_peered":"2024-09-13T15:02:14.778605+0000","last_clean":"2024-09-13T15:02:14.778605+0000","last_became_active":"2024-09-13T15:00:24.022727+0000","last_became_peered":"2024-09-13T15:00:24.022727+0000","last_unstale":"2024-09-13T15:02:14.778605+0000","last_undegraded":"2024-09-13T15:02:14.778605+0000","last_fullsized":"2024-09-13T15:02:14.778605+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:35.731087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440205336,"num_objects":6718,"num_object_clones":0,"num_object_copies":20154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7700,"num_write_kb":461313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":451,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.087501+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:02:04.087501+0000","last_peered":"2024-09-13T15:02:04.087501+0000","last_clean":"2024-09-13T15:02:04.087501+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:02:04.087501+0000","last_undegraded":"2024-09-13T15:02:04.087501+0000","last_fullsized":"2024-09-13T15:02:04.087501+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":930,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.039989+0000","last_change":"2024-09-13T15:00:15.946328+0000","last_active":"2024-09-13T15:02:03.039989+0000","last_peered":"2024-09-13T15:02:03.039989+0000","last_clean":"2024-09-13T15:02:03.039989+0000","last_became_active":"2024-09-13T15:00:15.944579+0000","last_became_peered":"2024-09-13T15:00:15.944579+0000","last_unstale":"2024-09-13T15:02:03.039989+0000","last_undegraded":"2024-09-13T15:02:03.039989+0000","last_fullsized":"2024-09-13T15:02:03.039989+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:03.065364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":360,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036017+0000","last_change":"2024-09-13T14:59:43.550071+0000","last_active":"2024-09-13T15:02:03.036017+0000","last_peered":"2024-09-13T15:02:03.036017+0000","last_clean":"2024-09-13T15:02:03.036017+0000","last_became_active":"2024-09-13T14:59:43.549357+0000","last_became_peered":"2024-09-13T14:59:43.549357+0000","last_unstale":"2024-09-13T15:02:03.036017+0000","last_undegraded":"2024-09-13T15:02:03.036017+0000","last_fullsized":"2024-09-13T15:02:03.036017+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:11.778618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":344,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036478+0000","last_change":"2024-09-13T14:59:04.214207+0000","last_active":"2024-09-13T15:02:03.036478+0000","last_peered":"2024-09-13T15:02:03.036478+0000","last_clean":"2024-09-13T15:02:03.036478+0000","last_became_active":"2024-09-13T14:59:04.213866+0000","last_became_peered":"2024-09-13T14:59:04.213866+0000","last_unstale":"2024-09-13T15:02:03.036478+0000","last_undegraded":"2024-09-13T15:02:03.036478+0000","last_fullsized":"2024-09-13T15:02:03.036478+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:19.940621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":428,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036207+0000","last_change":"2024-09-13T15:00:15.945494+0000","last_active":"2024-09-13T15:02:03.036207+0000","last_peered":"2024-09-13T15:02:03.036207+0000","last_clean":"2024-09-13T15:02:03.036207+0000","last_became_active":"2024-09-13T15:00:15.945117+0000","last_became_peered":"2024-09-13T15:00:15.945117+0000","last_unstale":"2024-09-13T15:02:03.036207+0000","last_undegraded":"2024-09-13T15:02:03.036207+0000","last_fullsized":"2024-09-13T15:02:03.036207+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:20.401090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19024,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":580,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.074243+0000","last_change":"2024-09-13T15:00:15.945220+0000","last_active":"2024-09-13T15:02:04.074243+0000","last_peered":"2024-09-13T15:02:04.074243+0000","last_clean":"2024-09-13T15:02:04.074243+0000","last_became_active":"2024-09-13T15:00:15.944818+0000","last_became_peered":"2024-09-13T15:00:15.944818+0000","last_unstale":"2024-09-13T15:02:04.074243+0000","last_undegraded":"2024-09-13T15:02:04.074243+0000","last_fullsized":"2024-09-13T15:02:04.074243+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:24.846652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":80,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.039935+0000","last_change":"2024-09-13T14:57:56.271513+0000","last_active":"2024-09-13T15:02:03.039935+0000","last_peered":"2024-09-13T15:02:03.039935+0000","last_clean":"2024-09-13T15:02:03.039935+0000","last_became_active":"2024-09-13T14:57:56.258172+0000","last_became_peered":"2024-09-13T14:57:56.258172+0000","last_unstale":"2024-09-13T15:02:03.039935+0000","last_undegraded":"2024-09-13T15:02:03.039935+0000","last_fullsized":"2024-09-13T15:02:03.039935+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:41.410545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":224,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032909+0000","last_change":"2024-09-13T14:57:56.258522+0000","last_active":"2024-09-13T15:02:03.032909+0000","last_peered":"2024-09-13T15:02:03.032909+0000","last_clean":"2024-09-13T15:02:03.032909+0000","last_became_active":"2024-09-13T14:57:56.258243+0000","last_became_peered":"2024-09-13T14:57:56.258243+0000","last_unstale":"2024-09-13T15:02:03.032909+0000","last_undegraded":"2024-09-13T15:02:03.032909+0000","last_fullsized":"2024-09-13T15:02:03.032909+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:02.030552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":553,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.075102+0000","last_change":"2024-09-13T14:59:04.211006+0000","last_active":"2024-09-13T15:02:04.075102+0000","last_peered":"2024-09-13T15:02:04.075102+0000","last_clean":"2024-09-13T15:02:04.075102+0000","last_became_active":"2024-09-13T14:59:04.210670+0000","last_became_peered":"2024-09-13T14:59:04.210670+0000","last_unstale":"2024-09-13T15:02:04.075102+0000","last_undegraded":"2024-09-13T15:02:04.075102+0000","last_fullsized":"2024-09-13T15:02:04.075102+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:50.748684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":543,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.082964+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:02:04.082964+0000","last_peered":"2024-09-13T15:02:04.082964+0000","last_clean":"2024-09-13T15:02:04.082964+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:02:04.082964+0000","last_undegraded":"2024-09-13T15:02:04.082964+0000","last_fullsized":"2024-09-13T15:02:04.082964+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":597,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.074924+0000","last_change":"2024-09-13T14:59:04.214486+0000","last_active":"2024-09-13T15:02:04.074924+0000","last_peered":"2024-09-13T15:02:04.074924+0000","last_clean":"2024-09-13T15:02:04.074924+0000","last_became_active":"2024-09-13T14:59:04.214144+0000","last_became_peered":"2024-09-13T14:59:04.214144+0000","last_unstale":"2024-09-13T15:02:04.074924+0000","last_undegraded":"2024-09-13T15:02:04.074924+0000","last_fullsized":"2024-09-13T15:02:04.074924+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:26.351108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":142,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.033028+0000","last_change":"2024-09-13T14:59:43.552494+0000","last_active":"2024-09-13T15:02:03.033028+0000","last_peered":"2024-09-13T15:02:03.033028+0000","last_clean":"2024-09-13T15:02:03.033028+0000","last_became_active":"2024-09-13T14:59:43.552209+0000","last_became_peered":"2024-09-13T14:59:43.552209+0000","last_unstale":"2024-09-13T15:02:03.033028+0000","last_undegraded":"2024-09-13T15:02:03.033028+0000","last_fullsized":"2024-09-13T15:02:03.033028+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:21.140470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":568,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.075004+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:02:04.075004+0000","last_peered":"2024-09-13T15:02:04.075004+0000","last_clean":"2024-09-13T15:02:04.075004+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:02:04.075004+0000","last_undegraded":"2024-09-13T15:02:04.075004+0000","last_fullsized":"2024-09-13T15:02:04.075004+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":525,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.087157+0000","last_change":"2024-09-13T15:01:17.859504+0000","last_active":"2024-09-13T15:02:04.087157+0000","last_peered":"2024-09-13T15:02:04.087157+0000","last_clean":"2024-09-13T15:02:04.087157+0000","last_became_active":"2024-09-13T15:01:17.849652+0000","last_became_peered":"2024-09-13T15:01:17.849652+0000","last_unstale":"2024-09-13T15:02:04.087157+0000","last_undegraded":"2024-09-13T15:02:04.087157+0000","last_fullsized":"2024-09-13T15:02:04.087157+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:04.098478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":541,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.074171+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:02:04.074171+0000","last_peered":"2024-09-13T15:02:04.074171+0000","last_clean":"2024-09-13T15:02:04.074171+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:02:04.074171+0000","last_undegraded":"2024-09-13T15:02:04.074171+0000","last_fullsized":"2024-09-13T15:02:04.074171+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":406,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.074532+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:02:04.074532+0000","last_peered":"2024-09-13T15:02:04.074532+0000","last_clean":"2024-09-13T15:02:04.074532+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:02:04.074532+0000","last_undegraded":"2024-09-13T15:02:04.074532+0000","last_fullsized":"2024-09-13T15:02:04.074532+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":348,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036512+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:02:03.036512+0000","last_peered":"2024-09-13T15:02:03.036512+0000","last_clean":"2024-09-13T15:02:03.036512+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:02:03.036512+0000","last_undegraded":"2024-09-13T15:02:03.036512+0000","last_fullsized":"2024-09-13T15:02:03.036512+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":157,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.085279+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:02:04.085279+0000","last_peered":"2024-09-13T15:02:04.085279+0000","last_clean":"2024-09-13T15:02:04.085279+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:02:04.085279+0000","last_undegraded":"2024-09-13T15:02:04.085279+0000","last_fullsized":"2024-09-13T15:02:04.085279+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":513,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.082432+0000","last_change":"2024-09-13T14:59:42.573105+0000","last_active":"2024-09-13T15:02:04.082432+0000","last_peered":"2024-09-13T15:02:04.082432+0000","last_clean":"2024-09-13T15:02:04.082432+0000","last_became_active":"2024-09-13T14:59:42.572543+0000","last_became_peered":"2024-09-13T14:59:42.572543+0000","last_unstale":"2024-09-13T15:02:04.082432+0000","last_undegraded":"2024-09-13T15:02:04.082432+0000","last_fullsized":"2024-09-13T15:02:04.082432+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:00.772607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":358,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036412+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:02:03.036412+0000","last_peered":"2024-09-13T15:02:03.036412+0000","last_clean":"2024-09-13T15:02:03.036412+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:02:03.036412+0000","last_undegraded":"2024-09-13T15:02:03.036412+0000","last_fullsized":"2024-09-13T15:02:03.036412+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":190,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032967+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T15:02:03.032967+0000","last_peered":"2024-09-13T15:02:03.032967+0000","last_clean":"2024-09-13T15:02:03.032967+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T15:02:03.032967+0000","last_undegraded":"2024-09-13T15:02:03.032967+0000","last_fullsized":"2024-09-13T15:02:03.032967+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":346,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.036434+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:02:03.036434+0000","last_peered":"2024-09-13T15:02:03.036434+0000","last_clean":"2024-09-13T15:02:03.036434+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:02:03.036434+0000","last_undegraded":"2024-09-13T15:02:03.036434+0000","last_fullsized":"2024-09-13T15:02:03.036434+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":205,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-13T15:02:03.032956+0000","last_change":"2024-09-13T14:59:04.223059+0000","last_active":"2024-09-13T15:02:03.032956+0000","last_peered":"2024-09-13T15:02:03.032956+0000","last_clean":"2024-09-13T15:02:03.032956+0000","last_became_active":"2024-09-13T14:59:04.218050+0000","last_became_peered":"2024-09-13T14:59:04.218050+0000","last_unstale":"2024-09-13T15:02:03.032956+0000","last_undegraded":"2024-09-13T15:02:03.032956+0000","last_fullsized":"2024-09-13T15:02:03.032956+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:37.540453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":578,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.079535+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:02:04.079535+0000","last_peered":"2024-09-13T15:02:04.079535+0000","last_clean":"2024-09-13T15:02:04.079535+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:02:04.079535+0000","last_undegraded":"2024-09-13T15:02:04.079535+0000","last_fullsized":"2024-09-13T15:02:04.079535+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":593,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.074751+0000","last_change":"2024-09-13T15:00:15.945414+0000","last_active":"2024-09-13T15:02:04.074751+0000","last_peered":"2024-09-13T15:02:04.074751+0000","last_clean":"2024-09-13T15:02:04.074751+0000","last_became_active":"2024-09-13T15:00:15.945043+0000","last_became_peered":"2024-09-13T15:00:15.945043+0000","last_unstale":"2024-09-13T15:02:04.074751+0000","last_undegraded":"2024-09-13T15:02:04.074751+0000","last_fullsized":"2024-09-13T15:02:04.074751+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:23.243083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":371,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077352+0000","last_change":"2024-09-13T14:59:04.212433+0000","last_active":"2024-09-13T15:02:04.077352+0000","last_peered":"2024-09-13T15:02:04.077352+0000","last_clean":"2024-09-13T15:02:04.077352+0000","last_became_active":"2024-09-13T14:59:04.212152+0000","last_became_peered":"2024-09-13T14:59:04.212152+0000","last_unstale":"2024-09-13T15:02:04.077352+0000","last_undegraded":"2024-09-13T15:02:04.077352+0000","last_fullsized":"2024-09-13T15:02:04.077352+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:04.818877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":567,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.082674+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:02:04.082674+0000","last_peered":"2024-09-13T15:02:04.082674+0000","last_clean":"2024-09-13T15:02:04.082674+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:02:04.082674+0000","last_undegraded":"2024-09-13T15:02:04.082674+0000","last_fullsized":"2024-09-13T15:02:04.082674+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7208108056,"num_objects":109988,"num_object_clones":0,"num_object_copies":329964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109988,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122086,"num_write_kb":7426305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21546086400,"data_stored":21546074184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116386,"ondisk_log_size":116386,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":22254,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":21452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2487812,"kb_used_data":2172028,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91257340,"statfs":{"total":95995035648,"available":93447516160,"internally_reserved":0,"allocated":2224156672,"data_stored":2222255182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":323353035},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757689,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3910896,"kb_used_data":3452392,"kb_used_omap":1,"kb_used_meta":458494,"kb_avail":89834256,"statfs":{"total":95995035648,"available":91990278144,"internally_reserved":0,"allocated":3535249408,"data_stored":3533363611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":469498311},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019329,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2672696,"kb_used_data":2176432,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91072456,"statfs":{"total":95995035648,"available":93258194944,"internally_reserved":0,"allocated":2228666368,"data_stored":2226776837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468213,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3728044,"kb_used_data":3146596,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":90017108,"statfs":{"total":95995035648,"available":92177518592,"internally_reserved":0,"allocated":3222114304,"data_stored":3220232353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":595324068},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273113,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4076992,"kb_used_data":3938936,"kb_used_omap":12,"kb_used_meta":137907,"kb_avail":89668160,"statfs":{"total":95995035648,"available":91820195840,"internally_reserved":0,"allocated":4033470464,"data_stored":4031573291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":141217664},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795265,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3693356,"kb_used_data":3148004,"kb_used_omap":7,"kb_used_meta":545336,"kb_avail":90051796,"statfs":{"total":95995035648,"available":92213039104,"internally_reserved":0,"allocated":3223556096,"data_stored":3221674548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":558424690},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502047,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3281204,"kb_used_data":3145452,"kb_used_omap":10,"kb_used_meta":135669,"kb_avail":90463948,"statfs":{"total":95995035648,"available":92635082752,"internally_reserved":0,"allocated":3220942848,"data_stored":3219053712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":138925759},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3191406592,"data_stored":3191406592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3193110528,"data_stored":3193110528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4021686272,"data_stored":4021682200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3192258560,"data_stored":3192258560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2218000384,"data_stored":2218000384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3518631936,"data_stored":3518627864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2210992128,"data_stored":2210988056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:02:19.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:02:19.199 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:19 smithi154 ceph-mon[102314]: pgmap v1110: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 557 KiB/s wr, 858 op/s 2024-09-13T15:02:19.199 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:19 smithi154 ceph-mon[107038]: pgmap v1110: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 557 KiB/s wr, 858 op/s 2024-09-13T15:02:19.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:19.277 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:19.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:19.407 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:19 smithi172 ceph-mon[100696]: pgmap v1110: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 557 KiB/s wr, 858 op/s 2024-09-13T15:02:19.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:19.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:19.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:19.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:19.773 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:02:19.773 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:02:19.774 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:02:19.774 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:02:19.774 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:02:19.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:19.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:19.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:19.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:19.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:20.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:20.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:02:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:20.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:20.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:20.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:20.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:20.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:20.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:20.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:20.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:20.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:20.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:20.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:20.938 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:20.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:20.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:21.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:21.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:21.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:21.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:21.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:21.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:21.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:02:21.452 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:21.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:21.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:21.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:21.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:21.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:21.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:21.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:21.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:21.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:22.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:22.076 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:22.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:22.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:22 smithi154 ceph-mon[102314]: pgmap v1111: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:02:22.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:02:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:02:21] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.51.0" 2024-09-13T15:02:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:22 smithi154 ceph-mon[107038]: pgmap v1111: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:02:22.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:22.325 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:22.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:22.450 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:22.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:22 smithi172 ceph-mon[100696]: pgmap v1111: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:02:22.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:22.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:22.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:22.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:22.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:22.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:22.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:22.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:23.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:23.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:23.139 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:23.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:23.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:23.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:23.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:23.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:23.530 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:23.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:23.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:23.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:23.808 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:23.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:24.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:24.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:24.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:24.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:24.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:24.322 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:24.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:24.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:24 smithi154 ceph-mon[102314]: pgmap v1112: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:02:24.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:24 smithi154 ceph-mon[107038]: pgmap v1112: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:02:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:24 smithi172 ceph-mon[100696]: pgmap v1112: 89 pgs: 89 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:02:24.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:24.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:24.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:24.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:24.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:24.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:24.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:24.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:24.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:25.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:25.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:25.135 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:25.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:25.264 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:25.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:25.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:25.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:25.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:25.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.673 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:25.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:25.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.803 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:25.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:25.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:25.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:25.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:25.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:26.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:26.060 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:26.116 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:26.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:26.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:26.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:26.332 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:26.425 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:26 smithi154 ceph-mon[102314]: pgmap v1113: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 855 op/s 2024-09-13T15:02:26.425 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:26 smithi154 ceph-mon[107038]: pgmap v1113: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 855 op/s 2024-09-13T15:02:26.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:26 smithi172 ceph-mon[100696]: pgmap v1113: 89 pgs: 89 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 855 op/s 2024-09-13T15:02:26.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:26.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:26.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:26.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:26.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:26.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:26.924 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:26.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:27.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:27.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:27.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:27.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:27.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:27.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:27.436 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:27.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:27.505 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:02:27.506 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:27.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:27.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:27.544 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:02:27.545 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:27.576 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:02:27.577 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:27.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:27.617 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:02:27.617 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:27.638 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:27.655 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:02:27.655 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:27.686 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:02:27.686 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:27.719 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:02:27.720 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:27.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:27.759 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:02:27.760 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:27.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:27.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:27.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:27.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:27.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:28.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:28.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:28.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:28.176 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:28 smithi154 ceph-mon[102314]: pgmap v1114: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-09-13T15:02:28.176 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:28 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/996284495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:28.176 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:28.177 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:28 smithi154 ceph-mon[107038]: pgmap v1114: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-09-13T15:02:28.177 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:28 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/996284495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:28.177 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:28.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:28.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:28.332 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:28.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:28.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:28 smithi172 ceph-mon[100696]: pgmap v1114: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 843 op/s 2024-09-13T15:02:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:28 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/996284495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:28.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:28.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:28.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:28.733 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:28.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:28.859 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:28.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:28.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:28.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:29.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:29.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:29.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:29.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:29.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:29.364 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:29.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:29.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:29.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:29.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:29.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:29.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:29.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:29.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:29.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:30.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:30.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:30.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:30.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:30.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:30.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:30.387 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:30.423 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:30 smithi154 ceph-mon[102314]: pgmap v1115: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 708 op/s 2024-09-13T15:02:30.423 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:30 smithi154 ceph-mon[107038]: pgmap v1115: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 708 op/s 2024-09-13T15:02:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:30.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:30 smithi172 ceph-mon[100696]: pgmap v1115: 89 pgs: 89 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 708 op/s 2024-09-13T15:02:30.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:30.561 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:30.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:30.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:30.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:30.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:30.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:30.937 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:30.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:31.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:31.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:31.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:31.190 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:02:31.192 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:02:31.230 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:31.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:31.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:31.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:31.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:31.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:31.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:31.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:31.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:31.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:31.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:31.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:31.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:02:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:02:31] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-09-13T15:02:32.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:32.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:32.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:32.177 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:32.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:32.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:32.345 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:32 smithi154 ceph-mon[102314]: pgmap v1116: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T15:02:32.345 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:32 smithi154 ceph-mon[107038]: pgmap v1116: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T15:02:32.384 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1116,"stamp":"2024-09-13T15:02:30.986514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6239361021,"num_objects":95161,"num_object_clones":0,"num_object_copies":285483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95161,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":138064,"num_write_kb":7441346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":3564500,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120423,"ondisk_log_size":120423,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":21078156,"kb_used_data":18405652,"kb_used_omap":39,"kb_used_meta":2672344,"kb_avail":635137908,"statfs":{"total":671965249536,"available":650381217792,"internally_reserved":0,"allocated":18847387648,"data_stored":18834133638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40275,"internal_metadata":2736480941},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-793968640,"num_objects":-12115,"num_object_clones":0,"num_object_copies":-36345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12115,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008545"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.087419+0000","last_change":"2024-09-13T15:01:14.837554+0000","last_active":"2024-09-13T15:02:17.087419+0000","last_peered":"2024-09-13T15:02:17.087419+0000","last_clean":"2024-09-13T15:02:17.087419+0000","last_became_active":"2024-09-13T15:01:14.837236+0000","last_became_peered":"2024-09-13T15:01:14.837236+0000","last_unstale":"2024-09-13T15:02:17.087419+0000","last_undegraded":"2024-09-13T15:02:17.087419+0000","last_fullsized":"2024-09-13T15:02:17.087419+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:18.510952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":92,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.086751+0000","last_change":"2024-09-13T15:00:15.944713+0000","last_active":"2024-09-13T15:02:04.086751+0000","last_peered":"2024-09-13T15:02:04.086751+0000","last_clean":"2024-09-13T15:02:04.086751+0000","last_became_active":"2024-09-13T15:00:15.944353+0000","last_became_peered":"2024-09-13T15:00:15.944353+0000","last_unstale":"2024-09-13T15:02:04.086751+0000","last_undegraded":"2024-09-13T15:02:04.086751+0000","last_fullsized":"2024-09-13T15:02:04.086751+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:22.078466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":198,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078484+0000","last_change":"2024-09-13T14:57:56.258528+0000","last_active":"2024-09-13T15:02:04.078484+0000","last_peered":"2024-09-13T15:02:04.078484+0000","last_clean":"2024-09-13T15:02:04.078484+0000","last_became_active":"2024-09-13T14:57:56.258116+0000","last_became_peered":"2024-09-13T14:57:56.258116+0000","last_unstale":"2024-09-13T15:02:04.078484+0000","last_undegraded":"2024-09-13T15:02:04.078484+0000","last_fullsized":"2024-09-13T15:02:04.078484+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:13.267817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":522,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.106824+0000","last_change":"2024-09-13T14:59:42.555312+0000","last_active":"2024-09-13T15:02:17.106824+0000","last_peered":"2024-09-13T15:02:17.106824+0000","last_clean":"2024-09-13T15:02:17.106824+0000","last_became_active":"2024-09-13T14:59:42.554397+0000","last_became_peered":"2024-09-13T14:59:42.554397+0000","last_unstale":"2024-09-13T15:02:17.106824+0000","last_undegraded":"2024-09-13T15:02:17.106824+0000","last_fullsized":"2024-09-13T15:02:17.106824+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:26.418960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":526,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.102854+0000","last_change":"2024-09-13T14:59:42.572046+0000","last_active":"2024-09-13T15:02:17.102854+0000","last_peered":"2024-09-13T15:02:17.102854+0000","last_clean":"2024-09-13T15:02:17.102854+0000","last_became_active":"2024-09-13T14:59:42.565943+0000","last_became_peered":"2024-09-13T14:59:42.565943+0000","last_unstale":"2024-09-13T15:02:17.102854+0000","last_undegraded":"2024-09-13T15:02:17.102854+0000","last_fullsized":"2024-09-13T15:02:17.102854+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:46.240539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":95,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.086408+0000","last_change":"2024-09-13T14:59:43.546919+0000","last_active":"2024-09-13T15:02:04.086408+0000","last_peered":"2024-09-13T15:02:04.086408+0000","last_clean":"2024-09-13T15:02:04.086408+0000","last_became_active":"2024-09-13T14:59:43.546633+0000","last_became_peered":"2024-09-13T14:59:43.546633+0000","last_unstale":"2024-09-13T15:02:04.086408+0000","last_undegraded":"2024-09-13T15:02:04.086408+0000","last_fullsized":"2024-09-13T15:02:04.086408+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:34.351065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.096660+0000","last_change":"2024-09-13T15:01:17.867842+0000","last_active":"2024-09-13T15:02:17.096660+0000","last_peered":"2024-09-13T15:02:17.096660+0000","last_clean":"2024-09-13T15:02:17.096660+0000","last_became_active":"2024-09-13T15:01:17.867444+0000","last_became_peered":"2024-09-13T15:01:17.867444+0000","last_unstale":"2024-09-13T15:02:17.096660+0000","last_undegraded":"2024-09-13T15:02:17.096660+0000","last_fullsized":"2024-09-13T15:02:17.096660+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:14.910223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":129,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.098777+0000","last_change":"2024-09-13T14:59:43.549396+0000","last_active":"2024-09-13T15:02:17.098777+0000","last_peered":"2024-09-13T15:02:17.098777+0000","last_clean":"2024-09-13T15:02:17.098777+0000","last_became_active":"2024-09-13T14:59:43.548994+0000","last_became_peered":"2024-09-13T14:59:43.548994+0000","last_unstale":"2024-09-13T15:02:17.098777+0000","last_undegraded":"2024-09-13T15:02:17.098777+0000","last_fullsized":"2024-09-13T15:02:17.098777+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:43:51.433399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"350'8524","reported_seq":8553,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:29.529577+0000","last_change":"2024-09-13T15:00:24.023321+0000","last_active":"2024-09-13T15:02:29.529577+0000","last_peered":"2024-09-13T15:02:29.529577+0000","last_clean":"2024-09-13T15:02:29.529577+0000","last_became_active":"2024-09-13T15:00:24.022952+0000","last_became_peered":"2024-09-13T15:00:24.022952+0000","last_unstale":"2024-09-13T15:02:29.529577+0000","last_undegraded":"2024-09-13T15:02:29.529577+0000","last_fullsized":"2024-09-13T15:02:29.529577+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":8524,"log_dups_size":0,"ondisk_log_size":8524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:34.183228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394657792,"num_objects":6022,"num_object_clones":0,"num_object_copies":18066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8524,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":107,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.085472+0000","last_change":"2024-09-13T15:00:15.946088+0000","last_active":"2024-09-13T15:02:17.085472+0000","last_peered":"2024-09-13T15:02:17.085472+0000","last_clean":"2024-09-13T15:02:17.085472+0000","last_became_active":"2024-09-13T15:00:15.945399+0000","last_became_peered":"2024-09-13T15:00:15.945399+0000","last_unstale":"2024-09-13T15:02:17.085472+0000","last_undegraded":"2024-09-13T15:02:17.085472+0000","last_fullsized":"2024-09-13T15:02:17.085472+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:20.043020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":235,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078203+0000","last_change":"2024-09-13T14:59:04.217165+0000","last_active":"2024-09-13T15:02:04.078203+0000","last_peered":"2024-09-13T15:02:04.078203+0000","last_clean":"2024-09-13T15:02:04.078203+0000","last_became_active":"2024-09-13T14:59:04.216859+0000","last_became_peered":"2024-09-13T14:59:04.216859+0000","last_unstale":"2024-09-13T15:02:04.078203+0000","last_undegraded":"2024-09-13T15:02:04.078203+0000","last_fullsized":"2024-09-13T15:02:04.078203+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:55:00.417627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"350'8536","reported_seq":8565,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:29.148550+0000","last_change":"2024-09-13T15:00:24.023029+0000","last_active":"2024-09-13T15:02:29.148550+0000","last_peered":"2024-09-13T15:02:29.148550+0000","last_clean":"2024-09-13T15:02:29.148550+0000","last_became_active":"2024-09-13T15:00:24.022751+0000","last_became_peered":"2024-09-13T15:00:24.022751+0000","last_unstale":"2024-09-13T15:02:29.148550+0000","last_undegraded":"2024-09-13T15:02:29.148550+0000","last_fullsized":"2024-09-13T15:02:29.148550+0000","mapping_epoch":331,"log_start":"337'1000","ondisk_log_start":"337'1000","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:57.881826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389283840,"num_objects":5940,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":463232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":80,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.085616+0000","last_change":"2024-09-13T14:57:56.273194+0000","last_active":"2024-09-13T15:02:17.085616+0000","last_peered":"2024-09-13T15:02:17.085616+0000","last_clean":"2024-09-13T15:02:17.085616+0000","last_became_active":"2024-09-13T14:57:56.259881+0000","last_became_peered":"2024-09-13T14:57:56.259881+0000","last_unstale":"2024-09-13T15:02:17.085616+0000","last_undegraded":"2024-09-13T15:02:17.085616+0000","last_fullsized":"2024-09-13T15:02:17.085616+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:28.683153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":412,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077464+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:02:04.077464+0000","last_peered":"2024-09-13T15:02:04.077464+0000","last_clean":"2024-09-13T15:02:04.077464+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:02:04.077464+0000","last_undegraded":"2024-09-13T15:02:04.077464+0000","last_fullsized":"2024-09-13T15:02:04.077464+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"350'8555","reported_seq":8604,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.822077+0000","last_change":"2024-09-13T15:00:24.024661+0000","last_active":"2024-09-13T15:02:30.822077+0000","last_peered":"2024-09-13T15:02:30.822077+0000","last_clean":"2024-09-13T15:02:30.822077+0000","last_became_active":"2024-09-13T15:00:24.023841+0000","last_became_peered":"2024-09-13T15:00:24.023841+0000","last_unstale":"2024-09-13T15:02:30.822077+0000","last_undegraded":"2024-09-13T15:02:30.822077+0000","last_fullsized":"2024-09-13T15:02:30.822077+0000","mapping_epoch":331,"log_start":"337'1400","ondisk_log_start":"337'1400","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7155,"log_dups_size":0,"ondisk_log_size":7155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:34.065023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388956160,"num_objects":5935,"num_object_clones":0,"num_object_copies":17805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":93,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.085598+0000","last_change":"2024-09-13T14:59:04.223552+0000","last_active":"2024-09-13T15:02:17.085598+0000","last_peered":"2024-09-13T15:02:17.085598+0000","last_clean":"2024-09-13T15:02:17.085598+0000","last_became_active":"2024-09-13T14:59:04.216127+0000","last_became_peered":"2024-09-13T14:59:04.216127+0000","last_unstale":"2024-09-13T15:02:17.085598+0000","last_undegraded":"2024-09-13T15:02:17.085598+0000","last_fullsized":"2024-09-13T15:02:17.085598+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:48.321535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":254,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.094982+0000","last_change":"2024-09-13T14:59:04.211285+0000","last_active":"2024-09-13T15:02:17.094982+0000","last_peered":"2024-09-13T15:02:17.094982+0000","last_clean":"2024-09-13T15:02:17.094982+0000","last_became_active":"2024-09-13T14:59:04.210929+0000","last_became_peered":"2024-09-13T14:59:04.210929+0000","last_unstale":"2024-09-13T15:02:17.094982+0000","last_undegraded":"2024-09-13T15:02:17.094982+0000","last_fullsized":"2024-09-13T15:02:17.094982+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:11.009140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"350'8610","reported_seq":8659,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.440399+0000","last_change":"2024-09-13T15:00:24.022660+0000","last_active":"2024-09-13T15:02:30.440399+0000","last_peered":"2024-09-13T15:02:30.440399+0000","last_clean":"2024-09-13T15:02:30.440399+0000","last_became_active":"2024-09-13T15:00:24.022328+0000","last_became_peered":"2024-09-13T15:00:24.022328+0000","last_unstale":"2024-09-13T15:02:30.440399+0000","last_undegraded":"2024-09-13T15:02:30.440399+0000","last_fullsized":"2024-09-13T15:02:30.440399+0000","mapping_epoch":331,"log_start":"337'1600","ondisk_log_start":"337'1600","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7010,"log_dups_size":0,"ondisk_log_size":7010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:12.849154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388890624,"num_objects":5934,"num_object_clones":0,"num_object_copies":17802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8610,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":215,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078513+0000","last_change":"2024-09-13T14:59:04.216946+0000","last_active":"2024-09-13T15:02:04.078513+0000","last_peered":"2024-09-13T15:02:04.078513+0000","last_clean":"2024-09-13T15:02:04.078513+0000","last_became_active":"2024-09-13T14:59:04.216628+0000","last_became_peered":"2024-09-13T14:59:04.216628+0000","last_unstale":"2024-09-13T15:02:04.078513+0000","last_undegraded":"2024-09-13T15:02:04.078513+0000","last_fullsized":"2024-09-13T15:02:04.078513+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:31:50.914653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":448,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.098608+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:02:17.098608+0000","last_peered":"2024-09-13T15:02:17.098608+0000","last_clean":"2024-09-13T15:02:17.098608+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:02:17.098608+0000","last_undegraded":"2024-09-13T15:02:17.098608+0000","last_fullsized":"2024-09-13T15:02:17.098608+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"350'8546","reported_seq":8596,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.221036+0000","last_change":"2024-09-13T15:00:23.973213+0000","last_active":"2024-09-13T15:02:30.221036+0000","last_peered":"2024-09-13T15:02:30.221036+0000","last_clean":"2024-09-13T15:02:30.221036+0000","last_became_active":"2024-09-13T15:00:23.972885+0000","last_became_peered":"2024-09-13T15:00:23.972885+0000","last_unstale":"2024-09-13T15:02:30.221036+0000","last_undegraded":"2024-09-13T15:02:30.221036+0000","last_fullsized":"2024-09-13T15:02:30.221036+0000","mapping_epoch":331,"log_start":"337'1700","ondisk_log_start":"337'1700","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6846,"log_dups_size":0,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:22.288214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389414912,"num_objects":5942,"num_object_clones":0,"num_object_copies":17826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":463616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":574,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.084943+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:02:17.084943+0000","last_peered":"2024-09-13T15:02:17.084943+0000","last_clean":"2024-09-13T15:02:17.084943+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:02:17.084943+0000","last_undegraded":"2024-09-13T15:02:17.084943+0000","last_fullsized":"2024-09-13T15:02:17.084943+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":346,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.393392+0000","last_change":"2024-09-13T14:59:04.213062+0000","last_active":"2024-09-13T15:02:04.393392+0000","last_peered":"2024-09-13T15:02:04.393392+0000","last_clean":"2024-09-13T15:02:04.393392+0000","last_became_active":"2024-09-13T14:59:04.212776+0000","last_became_peered":"2024-09-13T14:59:04.212776+0000","last_unstale":"2024-09-13T15:02:04.393392+0000","last_undegraded":"2024-09-13T15:02:04.393392+0000","last_fullsized":"2024-09-13T15:02:04.393392+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:07:49.898651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"350'8693","reported_seq":8740,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.491222+0000","last_change":"2024-09-13T15:00:24.023172+0000","last_active":"2024-09-13T15:02:30.491222+0000","last_peered":"2024-09-13T15:02:30.491222+0000","last_clean":"2024-09-13T15:02:30.491222+0000","last_became_active":"2024-09-13T15:00:24.022574+0000","last_became_peered":"2024-09-13T15:00:24.022574+0000","last_unstale":"2024-09-13T15:02:30.491222+0000","last_undegraded":"2024-09-13T15:02:30.491222+0000","last_fullsized":"2024-09-13T15:02:30.491222+0000","mapping_epoch":331,"log_start":"337'1700","ondisk_log_start":"337'1700","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6993,"log_dups_size":0,"ondisk_log_size":6993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:06.934046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391446528,"num_objects":5973,"num_object_clones":0,"num_object_copies":17919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":143,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077565+0000","last_change":"2024-09-13T14:59:04.217247+0000","last_active":"2024-09-13T15:02:04.077565+0000","last_peered":"2024-09-13T15:02:04.077565+0000","last_clean":"2024-09-13T15:02:04.077565+0000","last_became_active":"2024-09-13T14:59:04.216574+0000","last_became_peered":"2024-09-13T14:59:04.216574+0000","last_unstale":"2024-09-13T15:02:04.077565+0000","last_undegraded":"2024-09-13T15:02:04.077565+0000","last_fullsized":"2024-09-13T15:02:04.077565+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:00:26.362085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":349,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.395485+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:02:04.395485+0000","last_peered":"2024-09-13T15:02:04.395485+0000","last_clean":"2024-09-13T15:02:04.395485+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:02:04.395485+0000","last_undegraded":"2024-09-13T15:02:04.395485+0000","last_fullsized":"2024-09-13T15:02:04.395485+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"350'8623","reported_seq":8652,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:29.159261+0000","last_change":"2024-09-13T15:00:23.972644+0000","last_active":"2024-09-13T15:02:29.159261+0000","last_peered":"2024-09-13T15:02:29.159261+0000","last_clean":"2024-09-13T15:02:29.159261+0000","last_became_active":"2024-09-13T15:00:23.972370+0000","last_became_peered":"2024-09-13T15:00:23.972370+0000","last_unstale":"2024-09-13T15:02:29.159261+0000","last_undegraded":"2024-09-13T15:02:29.159261+0000","last_fullsized":"2024-09-13T15:02:29.159261+0000","mapping_epoch":331,"log_start":"337'1100","ondisk_log_start":"337'1100","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7523,"log_dups_size":0,"ondisk_log_size":7523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:55.756118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390660096,"num_objects":5961,"num_object_clones":0,"num_object_copies":17883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8623,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":355,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077462+0000","last_change":"2024-09-13T15:00:15.945718+0000","last_active":"2024-09-13T15:02:04.077462+0000","last_peered":"2024-09-13T15:02:04.077462+0000","last_clean":"2024-09-13T15:02:04.077462+0000","last_became_active":"2024-09-13T15:00:15.944400+0000","last_became_peered":"2024-09-13T15:00:15.944400+0000","last_unstale":"2024-09-13T15:02:04.077462+0000","last_undegraded":"2024-09-13T15:02:04.077462+0000","last_fullsized":"2024-09-13T15:02:04.077462+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:23:47.992626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":373,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.395503+0000","last_change":"2024-09-13T14:59:04.211136+0000","last_active":"2024-09-13T15:02:04.395503+0000","last_peered":"2024-09-13T15:02:04.395503+0000","last_clean":"2024-09-13T15:02:04.395503+0000","last_became_active":"2024-09-13T14:59:04.210848+0000","last_became_peered":"2024-09-13T14:59:04.210848+0000","last_unstale":"2024-09-13T15:02:04.395503+0000","last_undegraded":"2024-09-13T15:02:04.395503+0000","last_fullsized":"2024-09-13T15:02:04.395503+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:07.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"350'8506","reported_seq":8555,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.504587+0000","last_change":"2024-09-13T15:00:24.023758+0000","last_active":"2024-09-13T15:02:30.504587+0000","last_peered":"2024-09-13T15:02:30.504587+0000","last_clean":"2024-09-13T15:02:30.504587+0000","last_became_active":"2024-09-13T15:00:24.022468+0000","last_became_peered":"2024-09-13T15:00:24.022468+0000","last_unstale":"2024-09-13T15:02:30.504587+0000","last_undegraded":"2024-09-13T15:02:30.504587+0000","last_fullsized":"2024-09-13T15:02:30.504587+0000","mapping_epoch":331,"log_start":"337'1500","ondisk_log_start":"337'1500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7006,"log_dups_size":0,"ondisk_log_size":7006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:31.767944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389021696,"num_objects":5936,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8506,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.082523+0000","last_change":"2024-09-13T15:01:17.859549+0000","last_active":"2024-09-13T15:02:17.082523+0000","last_peered":"2024-09-13T15:02:17.082523+0000","last_clean":"2024-09-13T15:02:17.082523+0000","last_became_active":"2024-09-13T15:01:17.858796+0000","last_became_peered":"2024-09-13T15:01:17.858796+0000","last_unstale":"2024-09-13T15:02:17.082523+0000","last_undegraded":"2024-09-13T15:02:17.082523+0000","last_fullsized":"2024-09-13T15:02:17.082523+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:32.898026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":320,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.091902+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:02:17.091902+0000","last_peered":"2024-09-13T15:02:17.091902+0000","last_clean":"2024-09-13T15:02:17.091902+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:02:17.091902+0000","last_undegraded":"2024-09-13T15:02:17.091902+0000","last_fullsized":"2024-09-13T15:02:17.091902+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"350'8488","reported_seq":8516,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:29.154681+0000","last_change":"2024-09-13T15:00:23.972864+0000","last_active":"2024-09-13T15:02:29.154681+0000","last_peered":"2024-09-13T15:02:29.154681+0000","last_clean":"2024-09-13T15:02:29.154681+0000","last_became_active":"2024-09-13T15:00:23.972539+0000","last_became_peered":"2024-09-13T15:00:23.972539+0000","last_unstale":"2024-09-13T15:02:29.154681+0000","last_undegraded":"2024-09-13T15:02:29.154681+0000","last_fullsized":"2024-09-13T15:02:29.154681+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:22.430857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389021696,"num_objects":5936,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8488,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":738,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.089543+0000","last_change":"2024-09-13T14:59:43.552438+0000","last_active":"2024-09-13T15:02:04.089543+0000","last_peered":"2024-09-13T15:02:04.089543+0000","last_clean":"2024-09-13T15:02:04.089543+0000","last_became_active":"2024-09-13T14:59:43.551943+0000","last_became_peered":"2024-09-13T14:59:43.551943+0000","last_unstale":"2024-09-13T15:02:04.089543+0000","last_undegraded":"2024-09-13T15:02:04.089543+0000","last_fullsized":"2024-09-13T15:02:04.089543+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:03:18.178351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":970,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078843+0000","last_change":"2024-09-13T15:00:15.945425+0000","last_active":"2024-09-13T15:02:04.078843+0000","last_peered":"2024-09-13T15:02:04.078843+0000","last_clean":"2024-09-13T15:02:04.078843+0000","last_became_active":"2024-09-13T15:00:15.944987+0000","last_became_peered":"2024-09-13T15:00:15.944987+0000","last_unstale":"2024-09-13T15:02:04.078843+0000","last_undegraded":"2024-09-13T15:02:04.078843+0000","last_fullsized":"2024-09-13T15:02:04.078843+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:58.158525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":528,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.083812+0000","last_change":"2024-09-13T14:59:04.214511+0000","last_active":"2024-09-13T15:02:17.083812+0000","last_peered":"2024-09-13T15:02:17.083812+0000","last_clean":"2024-09-13T15:02:17.083812+0000","last_became_active":"2024-09-13T14:59:04.214185+0000","last_became_peered":"2024-09-13T14:59:04.214185+0000","last_unstale":"2024-09-13T15:02:17.083812+0000","last_undegraded":"2024-09-13T15:02:17.083812+0000","last_fullsized":"2024-09-13T15:02:17.083812+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:33.504798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":173,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077765+0000","last_change":"2024-09-13T14:59:43.549931+0000","last_active":"2024-09-13T15:02:04.077765+0000","last_peered":"2024-09-13T15:02:04.077765+0000","last_clean":"2024-09-13T15:02:04.077765+0000","last_became_active":"2024-09-13T14:59:43.549564+0000","last_became_peered":"2024-09-13T14:59:43.549564+0000","last_unstale":"2024-09-13T15:02:04.077765+0000","last_undegraded":"2024-09-13T15:02:04.077765+0000","last_fullsized":"2024-09-13T15:02:04.077765+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:10.194065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"350'8562","reported_seq":8611,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.486157+0000","last_change":"2024-09-13T15:00:24.024002+0000","last_active":"2024-09-13T15:02:30.486157+0000","last_peered":"2024-09-13T15:02:30.486157+0000","last_clean":"2024-09-13T15:02:30.486157+0000","last_became_active":"2024-09-13T15:00:24.023708+0000","last_became_peered":"2024-09-13T15:00:24.023708+0000","last_unstale":"2024-09-13T15:02:30.486157+0000","last_undegraded":"2024-09-13T15:02:30.486157+0000","last_fullsized":"2024-09-13T15:02:30.486157+0000","mapping_epoch":331,"log_start":"337'1500","ondisk_log_start":"337'1500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7062,"log_dups_size":0,"ondisk_log_size":7062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:46.507079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385744896,"num_objects":5886,"num_object_clones":0,"num_object_copies":17658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8562,"num_write_kb":462336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":109,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.089729+0000","last_change":"2024-09-13T14:59:42.568409+0000","last_active":"2024-09-13T15:02:04.089729+0000","last_peered":"2024-09-13T15:02:04.089729+0000","last_clean":"2024-09-13T15:02:04.089729+0000","last_became_active":"2024-09-13T14:59:42.567972+0000","last_became_peered":"2024-09-13T14:59:42.567972+0000","last_unstale":"2024-09-13T15:02:04.089729+0000","last_undegraded":"2024-09-13T15:02:04.089729+0000","last_fullsized":"2024-09-13T15:02:04.089729+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:21.721692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":522,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.101830+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:02:17.101830+0000","last_peered":"2024-09-13T15:02:17.101830+0000","last_clean":"2024-09-13T15:02:17.101830+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:02:17.101830+0000","last_undegraded":"2024-09-13T15:02:17.101830+0000","last_fullsized":"2024-09-13T15:02:17.101830+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":517,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.102686+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:02:17.102686+0000","last_peered":"2024-09-13T15:02:17.102686+0000","last_clean":"2024-09-13T15:02:17.102686+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:02:17.102686+0000","last_undegraded":"2024-09-13T15:02:17.102686+0000","last_fullsized":"2024-09-13T15:02:17.102686+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"350'8591","reported_seq":8640,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.186066+0000","last_change":"2024-09-13T15:00:23.973296+0000","last_active":"2024-09-13T15:02:30.186066+0000","last_peered":"2024-09-13T15:02:30.186066+0000","last_clean":"2024-09-13T15:02:30.186066+0000","last_became_active":"2024-09-13T15:00:23.972661+0000","last_became_peered":"2024-09-13T15:00:23.972661+0000","last_unstale":"2024-09-13T15:02:30.186066+0000","last_undegraded":"2024-09-13T15:02:30.186066+0000","last_fullsized":"2024-09-13T15:02:30.186066+0000","mapping_epoch":331,"log_start":"337'1700","ondisk_log_start":"337'1700","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6891,"log_dups_size":0,"ondisk_log_size":6891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:58.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394330112,"num_objects":6017,"num_object_clones":0,"num_object_copies":18051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8591,"num_write_kb":467456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":957,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.091504+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:02:17.091504+0000","last_peered":"2024-09-13T15:02:17.091504+0000","last_clean":"2024-09-13T15:02:17.091504+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:02:17.091504+0000","last_undegraded":"2024-09-13T15:02:17.091504+0000","last_fullsized":"2024-09-13T15:02:17.091504+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":578,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.096525+0000","last_change":"2024-09-13T15:00:15.945376+0000","last_active":"2024-09-13T15:02:17.096525+0000","last_peered":"2024-09-13T15:02:17.096525+0000","last_clean":"2024-09-13T15:02:17.096525+0000","last_became_active":"2024-09-13T15:00:15.944813+0000","last_became_peered":"2024-09-13T15:00:15.944813+0000","last_unstale":"2024-09-13T15:02:17.096525+0000","last_undegraded":"2024-09-13T15:02:17.096525+0000","last_fullsized":"2024-09-13T15:02:17.096525+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:02.237663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":323,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.095208+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:02:17.095208+0000","last_peered":"2024-09-13T15:02:17.095208+0000","last_clean":"2024-09-13T15:02:17.095208+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:02:17.095208+0000","last_undegraded":"2024-09-13T15:02:17.095208+0000","last_fullsized":"2024-09-13T15:02:17.095208+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"350'8514","reported_seq":8542,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:29.534589+0000","last_change":"2024-09-13T15:00:24.022623+0000","last_active":"2024-09-13T15:02:29.534589+0000","last_peered":"2024-09-13T15:02:29.534589+0000","last_clean":"2024-09-13T15:02:29.534589+0000","last_became_active":"2024-09-13T15:00:24.022333+0000","last_became_peered":"2024-09-13T15:00:24.022333+0000","last_unstale":"2024-09-13T15:02:29.534589+0000","last_undegraded":"2024-09-13T15:02:29.534589+0000","last_fullsized":"2024-09-13T15:02:29.534589+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":8514,"log_dups_size":0,"ondisk_log_size":8514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:24.771552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388497408,"num_objects":5928,"num_object_clones":0,"num_object_copies":17784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":246,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078581+0000","last_change":"2024-09-13T14:59:43.578518+0000","last_active":"2024-09-13T15:02:04.078581+0000","last_peered":"2024-09-13T15:02:04.078581+0000","last_clean":"2024-09-13T15:02:04.078581+0000","last_became_active":"2024-09-13T14:59:43.552215+0000","last_became_peered":"2024-09-13T14:59:43.552215+0000","last_unstale":"2024-09-13T15:02:04.078581+0000","last_undegraded":"2024-09-13T15:02:04.078581+0000","last_fullsized":"2024-09-13T15:02:04.078581+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:29.168795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":365,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.080102+0000","last_change":"2024-09-13T14:59:43.550004+0000","last_active":"2024-09-13T15:02:04.080102+0000","last_peered":"2024-09-13T15:02:04.080102+0000","last_clean":"2024-09-13T15:02:04.080102+0000","last_became_active":"2024-09-13T14:59:43.549533+0000","last_became_peered":"2024-09-13T14:59:43.549533+0000","last_unstale":"2024-09-13T15:02:04.080102+0000","last_undegraded":"2024-09-13T15:02:04.080102+0000","last_fullsized":"2024-09-13T15:02:04.080102+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:00.462528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":133,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078598+0000","last_change":"2024-09-13T15:00:14.949848+0000","last_active":"2024-09-13T15:02:04.078598+0000","last_peered":"2024-09-13T15:02:04.078598+0000","last_clean":"2024-09-13T15:02:04.078598+0000","last_became_active":"2024-09-13T15:00:14.949644+0000","last_became_peered":"2024-09-13T15:00:14.949644+0000","last_unstale":"2024-09-13T15:02:04.078598+0000","last_undegraded":"2024-09-13T15:02:04.078598+0000","last_fullsized":"2024-09-13T15:02:04.078598+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:17.762542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"350'8698","reported_seq":8746,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.881615+0000","last_change":"2024-09-13T15:00:24.024125+0000","last_active":"2024-09-13T15:02:30.881615+0000","last_peered":"2024-09-13T15:02:30.881615+0000","last_clean":"2024-09-13T15:02:30.881615+0000","last_became_active":"2024-09-13T15:00:24.023801+0000","last_became_peered":"2024-09-13T15:00:24.023801+0000","last_unstale":"2024-09-13T15:02:30.881615+0000","last_undegraded":"2024-09-13T15:02:30.881615+0000","last_fullsized":"2024-09-13T15:02:30.881615+0000","mapping_epoch":331,"log_start":"337'1500","ondisk_log_start":"337'1500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7198,"log_dups_size":0,"ondisk_log_size":7198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.418853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392691712,"num_objects":5992,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8698,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":223,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078405+0000","last_change":"2024-09-13T15:00:14.948015+0000","last_active":"2024-09-13T15:02:04.078405+0000","last_peered":"2024-09-13T15:02:04.078405+0000","last_clean":"2024-09-13T15:02:04.078405+0000","last_became_active":"2024-09-13T15:00:14.947799+0000","last_became_peered":"2024-09-13T15:00:14.947799+0000","last_unstale":"2024-09-13T15:02:04.078405+0000","last_undegraded":"2024-09-13T15:02:04.078405+0000","last_fullsized":"2024-09-13T15:02:04.078405+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:37.576480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1165,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.095222+0000","last_change":"2024-09-13T14:59:42.570065+0000","last_active":"2024-09-13T15:02:17.095222+0000","last_peered":"2024-09-13T15:02:17.095222+0000","last_clean":"2024-09-13T15:02:17.095222+0000","last_became_active":"2024-09-13T14:59:42.569452+0000","last_became_peered":"2024-09-13T14:59:42.569452+0000","last_unstale":"2024-09-13T15:02:17.095222+0000","last_undegraded":"2024-09-13T15:02:17.095222+0000","last_fullsized":"2024-09-13T15:02:17.095222+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:05.305260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":167,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.106699+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:02:17.106699+0000","last_peered":"2024-09-13T15:02:17.106699+0000","last_clean":"2024-09-13T15:02:17.106699+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:02:17.106699+0000","last_undegraded":"2024-09-13T15:02:17.106699+0000","last_fullsized":"2024-09-13T15:02:17.106699+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"350'8456","reported_seq":8505,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.246279+0000","last_change":"2024-09-13T15:00:24.024834+0000","last_active":"2024-09-13T15:02:30.246279+0000","last_peered":"2024-09-13T15:02:30.246279+0000","last_clean":"2024-09-13T15:02:30.246279+0000","last_became_active":"2024-09-13T15:00:24.024491+0000","last_became_peered":"2024-09-13T15:00:24.024491+0000","last_unstale":"2024-09-13T15:02:30.246279+0000","last_undegraded":"2024-09-13T15:02:30.246279+0000","last_fullsized":"2024-09-13T15:02:30.246279+0000","mapping_epoch":331,"log_start":"337'1600","ondisk_log_start":"337'1600","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":6856,"log_dups_size":0,"ondisk_log_size":6856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:35.133306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385351680,"num_objects":5880,"num_object_clones":0,"num_object_copies":17640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":577,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.095137+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:02:17.095137+0000","last_peered":"2024-09-13T15:02:17.095137+0000","last_clean":"2024-09-13T15:02:17.095137+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:02:17.095137+0000","last_undegraded":"2024-09-13T15:02:17.095137+0000","last_fullsized":"2024-09-13T15:02:17.095137+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":109,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:09.140133+0000","last_change":"2024-09-13T14:59:42.576338+0000","last_active":"2024-09-13T15:02:09.140133+0000","last_peered":"2024-09-13T15:02:09.140133+0000","last_clean":"2024-09-13T15:02:09.140133+0000","last_became_active":"2024-09-13T14:59:42.575875+0000","last_became_peered":"2024-09-13T14:59:42.575875+0000","last_unstale":"2024-09-13T15:02:09.140133+0000","last_undegraded":"2024-09-13T15:02:09.140133+0000","last_fullsized":"2024-09-13T15:02:09.140133+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:51.037045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":512,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.101134+0000","last_change":"2024-09-13T14:59:42.572653+0000","last_active":"2024-09-13T15:02:17.101134+0000","last_peered":"2024-09-13T15:02:17.101134+0000","last_clean":"2024-09-13T15:02:17.101134+0000","last_became_active":"2024-09-13T14:59:42.572326+0000","last_became_peered":"2024-09-13T14:59:42.572326+0000","last_unstale":"2024-09-13T15:02:17.101134+0000","last_undegraded":"2024-09-13T15:02:17.101134+0000","last_fullsized":"2024-09-13T15:02:17.101134+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:02.284316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"350'8599","reported_seq":8646,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:30.886581+0000","last_change":"2024-09-13T15:00:24.023353+0000","last_active":"2024-09-13T15:02:30.886581+0000","last_peered":"2024-09-13T15:02:30.886581+0000","last_clean":"2024-09-13T15:02:30.886581+0000","last_became_active":"2024-09-13T15:00:24.023074+0000","last_became_peered":"2024-09-13T15:00:24.023074+0000","last_unstale":"2024-09-13T15:02:30.886581+0000","last_undegraded":"2024-09-13T15:02:30.886581+0000","last_fullsized":"2024-09-13T15:02:30.886581+0000","mapping_epoch":331,"log_start":"337'1400","ondisk_log_start":"337'1400","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7199,"log_dups_size":0,"ondisk_log_size":7199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:50.987256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384892928,"num_objects":5873,"num_object_clones":0,"num_object_copies":17619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":463104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":607,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.101427+0000","last_change":"2024-09-13T14:59:42.561566+0000","last_active":"2024-09-13T15:02:17.101427+0000","last_peered":"2024-09-13T15:02:17.101427+0000","last_clean":"2024-09-13T15:02:17.101427+0000","last_became_active":"2024-09-13T14:59:42.561224+0000","last_became_peered":"2024-09-13T14:59:42.561224+0000","last_unstale":"2024-09-13T15:02:17.101427+0000","last_undegraded":"2024-09-13T15:02:17.101427+0000","last_fullsized":"2024-09-13T15:02:17.101427+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.918624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1040,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.096078+0000","last_change":"2024-09-13T14:59:07.221776+0000","last_active":"2024-09-13T15:02:17.096078+0000","last_peered":"2024-09-13T15:02:17.096078+0000","last_clean":"2024-09-13T15:02:17.096078+0000","last_became_active":"2024-09-13T14:59:07.221231+0000","last_became_peered":"2024-09-13T14:59:07.221231+0000","last_unstale":"2024-09-13T15:02:17.096078+0000","last_undegraded":"2024-09-13T15:02:17.096078+0000","last_fullsized":"2024-09-13T15:02:17.096078+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:18.106031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":319,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.082930+0000","last_change":"2024-09-13T14:59:42.561395+0000","last_active":"2024-09-13T15:02:17.082930+0000","last_peered":"2024-09-13T15:02:17.082930+0000","last_clean":"2024-09-13T15:02:17.082930+0000","last_became_active":"2024-09-13T14:59:42.561100+0000","last_became_peered":"2024-09-13T14:59:42.561100+0000","last_unstale":"2024-09-13T15:02:17.082930+0000","last_undegraded":"2024-09-13T15:02:17.082930+0000","last_fullsized":"2024-09-13T15:02:17.082930+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:31.681550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"350'8440","reported_seq":8469,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:29.493904+0000","last_change":"2024-09-13T15:00:24.023000+0000","last_active":"2024-09-13T15:02:29.493904+0000","last_peered":"2024-09-13T15:02:29.493904+0000","last_clean":"2024-09-13T15:02:29.493904+0000","last_became_active":"2024-09-13T15:00:24.022727+0000","last_became_peered":"2024-09-13T15:00:24.022727+0000","last_unstale":"2024-09-13T15:02:29.493904+0000","last_undegraded":"2024-09-13T15:02:29.493904+0000","last_fullsized":"2024-09-13T15:02:29.493904+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:35.731087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391708696,"num_objects":5978,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8440,"num_write_kb":461313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":453,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.102962+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:02:17.102962+0000","last_peered":"2024-09-13T15:02:17.102962+0000","last_clean":"2024-09-13T15:02:17.102962+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:02:17.102962+0000","last_undegraded":"2024-09-13T15:02:17.102962+0000","last_fullsized":"2024-09-13T15:02:17.102962+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":931,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.087850+0000","last_change":"2024-09-13T15:00:15.946328+0000","last_active":"2024-09-13T15:02:04.087850+0000","last_peered":"2024-09-13T15:02:04.087850+0000","last_clean":"2024-09-13T15:02:04.087850+0000","last_became_active":"2024-09-13T15:00:15.944579+0000","last_became_peered":"2024-09-13T15:00:15.944579+0000","last_unstale":"2024-09-13T15:02:04.087850+0000","last_undegraded":"2024-09-13T15:02:04.087850+0000","last_fullsized":"2024-09-13T15:02:04.087850+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:29:03.065364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":361,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077385+0000","last_change":"2024-09-13T14:59:43.550071+0000","last_active":"2024-09-13T15:02:04.077385+0000","last_peered":"2024-09-13T15:02:04.077385+0000","last_clean":"2024-09-13T15:02:04.077385+0000","last_became_active":"2024-09-13T14:59:43.549357+0000","last_became_peered":"2024-09-13T14:59:43.549357+0000","last_unstale":"2024-09-13T15:02:04.077385+0000","last_undegraded":"2024-09-13T15:02:04.077385+0000","last_fullsized":"2024-09-13T15:02:04.077385+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:11.778618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":345,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.080057+0000","last_change":"2024-09-13T14:59:04.214207+0000","last_active":"2024-09-13T15:02:04.080057+0000","last_peered":"2024-09-13T15:02:04.080057+0000","last_clean":"2024-09-13T15:02:04.080057+0000","last_became_active":"2024-09-13T14:59:04.213866+0000","last_became_peered":"2024-09-13T14:59:04.213866+0000","last_unstale":"2024-09-13T15:02:04.080057+0000","last_undegraded":"2024-09-13T15:02:04.080057+0000","last_fullsized":"2024-09-13T15:02:04.080057+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:19.940621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":429,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077782+0000","last_change":"2024-09-13T15:00:15.945494+0000","last_active":"2024-09-13T15:02:04.077782+0000","last_peered":"2024-09-13T15:02:04.077782+0000","last_clean":"2024-09-13T15:02:04.077782+0000","last_became_active":"2024-09-13T15:00:15.945117+0000","last_became_peered":"2024-09-13T15:00:15.945117+0000","last_unstale":"2024-09-13T15:02:04.077782+0000","last_undegraded":"2024-09-13T15:02:04.077782+0000","last_fullsized":"2024-09-13T15:02:04.077782+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:29:20.401090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19024,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":582,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.095132+0000","last_change":"2024-09-13T15:00:15.945220+0000","last_active":"2024-09-13T15:02:17.095132+0000","last_peered":"2024-09-13T15:02:17.095132+0000","last_clean":"2024-09-13T15:02:17.095132+0000","last_became_active":"2024-09-13T15:00:15.944818+0000","last_became_peered":"2024-09-13T15:00:15.944818+0000","last_unstale":"2024-09-13T15:02:17.095132+0000","last_undegraded":"2024-09-13T15:02:17.095132+0000","last_fullsized":"2024-09-13T15:02:17.095132+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:24.846652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":81,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.089123+0000","last_change":"2024-09-13T14:57:56.271513+0000","last_active":"2024-09-13T15:02:04.089123+0000","last_peered":"2024-09-13T15:02:04.089123+0000","last_clean":"2024-09-13T15:02:04.089123+0000","last_became_active":"2024-09-13T14:57:56.258172+0000","last_became_peered":"2024-09-13T14:57:56.258172+0000","last_unstale":"2024-09-13T15:02:04.089123+0000","last_undegraded":"2024-09-13T15:02:04.089123+0000","last_fullsized":"2024-09-13T15:02:04.089123+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:41.410545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":225,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078605+0000","last_change":"2024-09-13T14:57:56.258522+0000","last_active":"2024-09-13T15:02:04.078605+0000","last_peered":"2024-09-13T15:02:04.078605+0000","last_clean":"2024-09-13T15:02:04.078605+0000","last_became_active":"2024-09-13T14:57:56.258243+0000","last_became_peered":"2024-09-13T14:57:56.258243+0000","last_unstale":"2024-09-13T15:02:04.078605+0000","last_undegraded":"2024-09-13T15:02:04.078605+0000","last_fullsized":"2024-09-13T15:02:04.078605+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:02.030552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":555,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.082244+0000","last_change":"2024-09-13T14:59:04.211006+0000","last_active":"2024-09-13T15:02:17.082244+0000","last_peered":"2024-09-13T15:02:17.082244+0000","last_clean":"2024-09-13T15:02:17.082244+0000","last_became_active":"2024-09-13T14:59:04.210670+0000","last_became_peered":"2024-09-13T14:59:04.210670+0000","last_unstale":"2024-09-13T15:02:17.082244+0000","last_undegraded":"2024-09-13T15:02:17.082244+0000","last_fullsized":"2024-09-13T15:02:17.082244+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:50.748684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":545,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.101508+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:02:17.101508+0000","last_peered":"2024-09-13T15:02:17.101508+0000","last_clean":"2024-09-13T15:02:17.101508+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:02:17.101508+0000","last_undegraded":"2024-09-13T15:02:17.101508+0000","last_fullsized":"2024-09-13T15:02:17.101508+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":599,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.083065+0000","last_change":"2024-09-13T14:59:04.214486+0000","last_active":"2024-09-13T15:02:17.083065+0000","last_peered":"2024-09-13T15:02:17.083065+0000","last_clean":"2024-09-13T15:02:17.083065+0000","last_became_active":"2024-09-13T14:59:04.214144+0000","last_became_peered":"2024-09-13T14:59:04.214144+0000","last_unstale":"2024-09-13T15:02:17.083065+0000","last_undegraded":"2024-09-13T15:02:17.083065+0000","last_fullsized":"2024-09-13T15:02:17.083065+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:26.351108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":143,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078535+0000","last_change":"2024-09-13T14:59:43.552494+0000","last_active":"2024-09-13T15:02:04.078535+0000","last_peered":"2024-09-13T15:02:04.078535+0000","last_clean":"2024-09-13T15:02:04.078535+0000","last_became_active":"2024-09-13T14:59:43.552209+0000","last_became_peered":"2024-09-13T14:59:43.552209+0000","last_unstale":"2024-09-13T15:02:04.078535+0000","last_undegraded":"2024-09-13T15:02:04.078535+0000","last_fullsized":"2024-09-13T15:02:04.078535+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:21.140470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":570,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.083143+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:02:17.083143+0000","last_peered":"2024-09-13T15:02:17.083143+0000","last_clean":"2024-09-13T15:02:17.083143+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:02:17.083143+0000","last_undegraded":"2024-09-13T15:02:17.083143+0000","last_fullsized":"2024-09-13T15:02:17.083143+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":527,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.102771+0000","last_change":"2024-09-13T15:01:17.859504+0000","last_active":"2024-09-13T15:02:17.102771+0000","last_peered":"2024-09-13T15:02:17.102771+0000","last_clean":"2024-09-13T15:02:17.102771+0000","last_became_active":"2024-09-13T15:01:17.849652+0000","last_became_peered":"2024-09-13T15:01:17.849652+0000","last_unstale":"2024-09-13T15:02:17.102771+0000","last_undegraded":"2024-09-13T15:02:17.102771+0000","last_fullsized":"2024-09-13T15:02:17.102771+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:04.098478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":544,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.094813+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:02:17.094813+0000","last_peered":"2024-09-13T15:02:17.094813+0000","last_clean":"2024-09-13T15:02:17.094813+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:02:17.094813+0000","last_undegraded":"2024-09-13T15:02:17.094813+0000","last_fullsized":"2024-09-13T15:02:17.094813+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":408,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.094736+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:02:17.094736+0000","last_peered":"2024-09-13T15:02:17.094736+0000","last_clean":"2024-09-13T15:02:17.094736+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:02:17.094736+0000","last_undegraded":"2024-09-13T15:02:17.094736+0000","last_fullsized":"2024-09-13T15:02:17.094736+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":349,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077701+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:02:04.077701+0000","last_peered":"2024-09-13T15:02:04.077701+0000","last_clean":"2024-09-13T15:02:04.077701+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:02:04.077701+0000","last_undegraded":"2024-09-13T15:02:04.077701+0000","last_fullsized":"2024-09-13T15:02:04.077701+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":159,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.087730+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:02:17.087730+0000","last_peered":"2024-09-13T15:02:17.087730+0000","last_clean":"2024-09-13T15:02:17.087730+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:02:17.087730+0000","last_undegraded":"2024-09-13T15:02:17.087730+0000","last_fullsized":"2024-09-13T15:02:17.087730+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":516,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.101230+0000","last_change":"2024-09-13T14:59:42.573105+0000","last_active":"2024-09-13T15:02:17.101230+0000","last_peered":"2024-09-13T15:02:17.101230+0000","last_clean":"2024-09-13T15:02:17.101230+0000","last_became_active":"2024-09-13T14:59:42.572543+0000","last_became_peered":"2024-09-13T14:59:42.572543+0000","last_unstale":"2024-09-13T15:02:17.101230+0000","last_undegraded":"2024-09-13T15:02:17.101230+0000","last_fullsized":"2024-09-13T15:02:17.101230+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:00.772607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":359,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.080051+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:02:04.080051+0000","last_peered":"2024-09-13T15:02:04.080051+0000","last_clean":"2024-09-13T15:02:04.080051+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:02:04.080051+0000","last_undegraded":"2024-09-13T15:02:04.080051+0000","last_fullsized":"2024-09-13T15:02:04.080051+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":191,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078440+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T15:02:04.078440+0000","last_peered":"2024-09-13T15:02:04.078440+0000","last_clean":"2024-09-13T15:02:04.078440+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T15:02:04.078440+0000","last_undegraded":"2024-09-13T15:02:04.078440+0000","last_fullsized":"2024-09-13T15:02:04.078440+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":347,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.077583+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:02:04.077583+0000","last_peered":"2024-09-13T15:02:04.077583+0000","last_clean":"2024-09-13T15:02:04.077583+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:02:04.077583+0000","last_undegraded":"2024-09-13T15:02:04.077583+0000","last_fullsized":"2024-09-13T15:02:04.077583+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":206,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:04.078566+0000","last_change":"2024-09-13T14:59:04.223059+0000","last_active":"2024-09-13T15:02:04.078566+0000","last_peered":"2024-09-13T15:02:04.078566+0000","last_clean":"2024-09-13T15:02:04.078566+0000","last_became_active":"2024-09-13T14:59:04.218050+0000","last_became_peered":"2024-09-13T14:59:04.218050+0000","last_unstale":"2024-09-13T15:02:04.078566+0000","last_undegraded":"2024-09-13T15:02:04.078566+0000","last_fullsized":"2024-09-13T15:02:04.078566+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:37.540453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":580,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.085078+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:02:17.085078+0000","last_peered":"2024-09-13T15:02:17.085078+0000","last_clean":"2024-09-13T15:02:17.085078+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:02:17.085078+0000","last_undegraded":"2024-09-13T15:02:17.085078+0000","last_fullsized":"2024-09-13T15:02:17.085078+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":595,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.094666+0000","last_change":"2024-09-13T15:00:15.945414+0000","last_active":"2024-09-13T15:02:17.094666+0000","last_peered":"2024-09-13T15:02:17.094666+0000","last_clean":"2024-09-13T15:02:17.094666+0000","last_became_active":"2024-09-13T15:00:15.945043+0000","last_became_peered":"2024-09-13T15:00:15.945043+0000","last_unstale":"2024-09-13T15:02:17.094666+0000","last_undegraded":"2024-09-13T15:02:17.094666+0000","last_fullsized":"2024-09-13T15:02:17.094666+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:23.243083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":372,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-13T15:02:09.398122+0000","last_change":"2024-09-13T14:59:04.212433+0000","last_active":"2024-09-13T15:02:09.398122+0000","last_peered":"2024-09-13T15:02:09.398122+0000","last_clean":"2024-09-13T15:02:09.398122+0000","last_became_active":"2024-09-13T14:59:04.212152+0000","last_became_peered":"2024-09-13T14:59:04.212152+0000","last_unstale":"2024-09-13T15:02:09.398122+0000","last_undegraded":"2024-09-13T15:02:09.398122+0000","last_fullsized":"2024-09-13T15:02:09.398122+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:08:04.818877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":569,"reported_epoch":350,"state":"active+clean","last_fresh":"2024-09-13T15:02:17.100878+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:02:17.100878+0000","last_peered":"2024-09-13T15:02:17.100878+0000","last_clean":"2024-09-13T15:02:17.100878+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:02:17.100878+0000","last_undegraded":"2024-09-13T15:02:17.100878+0000","last_fullsized":"2024-09-13T15:02:17.100878+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6234570776,"num_objects":95133,"num_object_clones":0,"num_object_copies":285399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95133,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136941,"num_write_kb":7426305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18753400832,"data_stored":18753388616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119241,"ondisk_log_size":119241,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":22254,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":21452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2231056,"kb_used_data":1915272,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91514096,"statfs":{"total":95995035648,"available":93710434304,"internally_reserved":0,"allocated":1961238528,"data_stored":1959333126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":323353035},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":197568496057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757692,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3507324,"kb_used_data":3057012,"kb_used_omap":1,"kb_used_meta":450302,"kb_avail":90237828,"statfs":{"total":95995035648,"available":92403535872,"internally_reserved":0,"allocated":3130380288,"data_stored":3128490579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":461109703},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019332,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2421316,"kb_used_data":1925052,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91323836,"statfs":{"total":95995035648,"available":93515608064,"internally_reserved":0,"allocated":1971253248,"data_stored":1969359805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468217,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3267128,"kb_used_data":2676016,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":90478024,"statfs":{"total":95995035648,"available":92649496576,"internally_reserved":0,"allocated":2740240384,"data_stored":2738354521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":605285540},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273116,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3654028,"kb_used_data":3507716,"kb_used_omap":12,"kb_used_meta":146227,"kb_avail":90091124,"statfs":{"total":95995035648,"available":92253310976,"internally_reserved":0,"allocated":3591901184,"data_stored":3590000099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":149737344},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795269,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3204408,"kb_used_data":2663216,"kb_used_omap":7,"kb_used_meta":541176,"kb_avail":90540744,"statfs":{"total":95995035648,"available":92713721856,"internally_reserved":0,"allocated":2727133184,"data_stored":2725247724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":554164850},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502051,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2792896,"kb_used_data":2661368,"kb_used_omap":10,"kb_used_meta":131509,"kb_avail":90952256,"statfs":{"total":95995035648,"available":93135110144,"internally_reserved":0,"allocated":2725240832,"data_stored":2723347784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":134665919},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2715746304,"data_stored":2715746304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2721185792,"data_stored":2721185792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3553562624,"data_stored":3553558552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2720792576,"data_stored":2720792576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1966669824,"data_stored":1966669824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3123515392,"data_stored":3123511320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1951928320,"data_stored":1951924248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:02:32.389 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:02:32.392 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:02:32.392 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T15:02:32.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T15:02:32.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.478 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:32.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:32 smithi172 ceph-mon[100696]: pgmap v1116: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-13T15:02:32.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:32.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:32.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:32.730 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:32.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:32.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:32.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:32.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:32.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:33.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:33.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:33.142 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:33.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.279 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:33.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:33.379 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:33 smithi154 ceph-mon[107038]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:33.379 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:33 smithi154 ceph-mon[102314]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:33.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:33.405 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:33 smithi172 ceph-mon[100696]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:02:33.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:33.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:33.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:33.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:33.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:02:33.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:33.820 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:33.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:33.952 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:02:33.952 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:02:33.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:33.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:34.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:34.112 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:34.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:34.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:34 smithi154 ceph-mon[102314]: pgmap v1117: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 655 op/s 2024-09-13T15:02:34.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:34 smithi154 ceph-mon[107038]: pgmap v1117: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 655 op/s 2024-09-13T15:02:34.237 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:34.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:34.363 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:34.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:34 smithi172 ceph-mon[100696]: pgmap v1117: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 655 op/s 2024-09-13T15:02:34.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:34.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:34.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:34.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:34.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:34.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:34.936 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:34.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:34.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:02:35.020 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:35.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:35.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:35.099 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:02:35.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:35.201 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3879977117' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:35.202 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:35 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:35.202 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3879977117' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:35.202 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:35 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:35.248 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:35.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:35.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3879977117' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:35 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:02:35.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:35.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:35.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:35.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:35.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:35.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:35.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:35.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:35.913 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:35.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:36.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:36.041 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:36.087 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[102314]: pgmap v1118: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 766 op/s 2024-09-13T15:02:36.088 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:36.088 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:36.088 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[102314]: osdmap e351: 8 total, 8 up, 7 in 2024-09-13T15:02:36.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:36.170 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:36.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:36.297 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:36.400 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:36.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:36.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[107038]: pgmap v1118: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 766 op/s 2024-09-13T15:02:36.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:36.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:36.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:36 smithi154 ceph-mon[107038]: osdmap e351: 8 total, 8 up, 7 in 2024-09-13T15:02:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:36 smithi172 ceph-mon[100696]: pgmap v1118: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 766 op/s 2024-09-13T15:02:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:36 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:02:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:36 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:02:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:36 smithi172 ceph-mon[100696]: osdmap e351: 8 total, 8 up, 7 in 2024-09-13T15:02:36.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:36.554 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:36.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:36.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:36.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:36.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:36.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:36.940 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:36.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:36.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:37.016 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:37.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.138 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:37.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:37.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:37.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:37.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:37.385 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:37 smithi154 ceph-mon[102314]: pgmap v1120: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T15:02:37.385 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:37 smithi154 ceph-mon[107038]: pgmap v1120: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T15:02:37.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:37 smithi172 ceph-mon[100696]: pgmap v1120: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T15:02:37.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:37.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:37.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:37.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:37.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:37.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:37.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:37.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:37.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:38.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:02:38.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:38.213 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:38.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:38.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:38.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:38.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:38.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:38.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:38.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:38.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:38.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:38.918 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:38.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:39.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:39.046 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:39.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:39.171 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:39.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:39.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:39.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:39.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:39.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:39.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:39.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:39.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:39.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:39.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:39.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:39.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:40.044 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:40.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:40.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:40.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:40.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:40 smithi154 ceph-mon[107038]: pgmap v1121: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T15:02:40.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:40 smithi154 ceph-mon[102314]: pgmap v1121: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T15:02:40.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:40.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:40.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:40.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:40.462 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:40 smithi172 ceph-mon[100696]: pgmap v1121: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-13T15:02:40.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:40.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:40.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:40.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:40.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:40.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:40.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:40.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:41.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:41.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:41.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:41.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:41.291 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:41.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:41.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:41.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:41.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:41.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:41.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:41.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:41.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:41.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:41.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:41.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:41.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:41.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:42.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:42.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:42.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:42.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:42 smithi154 ceph-mon[102314]: pgmap v1122: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:02:42.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:42 smithi154 ceph-mon[107038]: pgmap v1122: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:02:42.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:02:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:02:41] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-09-13T15:02:42.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:42.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:42.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:42.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:42 smithi172 ceph-mon[100696]: pgmap v1122: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:02:42.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.599 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:42.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:42.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:42.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:42.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:42.829 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:02:42.830 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:42.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:42.869 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:02:42.870 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:42.909 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:02:42.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:42.940 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:02:42.940 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:42.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:42.972 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:02:42.972 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:42.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:42.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:43.006 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:02:43.007 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:43.042 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:02:43.042 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:43.072 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:02:43.072 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:43.094 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/678387143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:43.094 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:43.095 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/678387143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:43.095 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:43.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:43.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:43.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:43.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:43.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:43.404 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/678387143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:43.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:43.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:43.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:43.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:43.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:43.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:43.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:43.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:43.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:44.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:44.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:44.158 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:44 smithi154 ceph-mon[102314]: pgmap v1123: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:02:44.158 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:44 smithi154 ceph-mon[107038]: pgmap v1123: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:02:44.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:44.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:44.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:44.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:44 smithi172 ceph-mon[100696]: pgmap v1123: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:02:44.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:44.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:44.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-13T15:02:44.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:44.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:44.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:44.733 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:44.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:44.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:44.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:44.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:44.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:45.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:45.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:45.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:45.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:45.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:45.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:45.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:45.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:45.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:45.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:45.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:45.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:45.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:45.984 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:45.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:46.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:46.105 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:02:46.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:46.147 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:02:46.147 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:02:46.147 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T15:02:46.148 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:02:46.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:46.257 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:46.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:46.384 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:46.395 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:46 smithi154 ceph-mon[102314]: pgmap v1124: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 828 op/s 2024-09-13T15:02:46.395 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:46 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/877935096' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:46.395 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:46 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:46.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:46 smithi154 ceph-mon[107038]: pgmap v1124: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 828 op/s 2024-09-13T15:02:46.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:46 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/877935096' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:46.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:46 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:46.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:46 smithi172 ceph-mon[100696]: pgmap v1124: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 828 op/s 2024-09-13T15:02:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:46 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/877935096' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:46 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:02:46.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:46.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:46.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:46.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.790 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:46.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:46.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:46.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:46.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:47.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:47.057 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:47.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:47.187 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:47.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:47.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:47.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:47.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:02:47.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:47.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[102314]: osdmap e352: 8 total, 8 up, 7 in 2024-09-13T15:02:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:02:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:47 smithi154 ceph-mon[107038]: osdmap e352: 8 total, 8 up, 7 in 2024-09-13T15:02:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:47 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:02:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:47 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:02:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:47 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:02:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:47 smithi172 ceph-mon[100696]: osdmap e352: 8 total, 8 up, 7 in 2024-09-13T15:02:47.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:47.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:47.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:47.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:47.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:47.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:02:47.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:47.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:47.962 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:48.011 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:48.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:48.138 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:48.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:48.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:48.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:48.404 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:48.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:48 smithi154 ceph-mon[102314]: pgmap v1126: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:02:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:48 smithi154 ceph-mon[107038]: pgmap v1126: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:02:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:48 smithi172 ceph-mon[100696]: pgmap v1126: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:02:48.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:48.547 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:48.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:48.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:48.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:48.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:48.822 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:48.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:48.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:48.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:49.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:49.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:49.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:49.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:49.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:49.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:02:49.356 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:49 smithi154 ceph-mon[102314]: pgmap v1127: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:02:49.357 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:49 smithi154 ceph-mon[107038]: pgmap v1127: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:02:49.363 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:49.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:49 smithi172 ceph-mon[100696]: pgmap v1127: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:02:49.491 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:49.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:49.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:49.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:49.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:49.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:49.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:50.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:50.184 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:50.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:50.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:50.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:50.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:50.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:50.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.716 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:50.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:50.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:50.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:02:50.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:50.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:50.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:50.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:51.053 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:51.054 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:51.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:51.109 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:51.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:51.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:51.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:51.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:51.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:51.547 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:51.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:51.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:51.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:51.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:51.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:51.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:51.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:52.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:52.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:52.109 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:52 smithi154 ceph-mon[102314]: pgmap v1128: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-13T15:02:52.109 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:52 smithi154 ceph-mon[107038]: pgmap v1128: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-13T15:02:52.110 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:02:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:02:51] "GET /metrics HTTP/1.1" 200 36608 "" "Prometheus/2.51.0" 2024-09-13T15:02:52.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:52.236 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:52.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:52.366 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:52.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:52 smithi172 ceph-mon[100696]: pgmap v1128: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-13T15:02:52.491 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:52.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:52.622 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:52.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:52.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:52.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:52.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:52.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:53.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:53.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:53.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:53.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:53.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:53.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:53.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:53.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:53.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:53.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:53.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:53.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:53.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:53.837 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:53.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:53.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:53.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:54.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:54.088 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:54.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:54.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:54.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:54.339 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:54.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:54.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:54 smithi154 ceph-mon[102314]: pgmap v1129: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-13T15:02:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:54 smithi154 ceph-mon[107038]: pgmap v1129: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-13T15:02:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:54 smithi172 ceph-mon[100696]: pgmap v1129: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 821 op/s 2024-09-13T15:02:54.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:54.627 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:54.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:54.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:54.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:54.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:54.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:55.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:55.037 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:55.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:55.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:55.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:55.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:55.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:55.437 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:55.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:55.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:55.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:55.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:55.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:55.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:55.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:55.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:55.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:56.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:56.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:56.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:56.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:56.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:56.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:56.402 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:56 smithi154 ceph-mon[102314]: pgmap v1130: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:02:56.402 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:56 smithi154 ceph-mon[107038]: pgmap v1130: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:02:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:56 smithi172 ceph-mon[100696]: pgmap v1130: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:02:56.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:56.523 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:56.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:56.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:56.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:56.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:56.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:56.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:56.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:57.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-13T15:02:57.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:57.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:57.230 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:57.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:57.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:57.379 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:57.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:02:57.530 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:02:57.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:57.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:57.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:02:57.840 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:02:57.862 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:02:57.862 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:02:57.862 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:02:57.862 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:02:57.863 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:02:57.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:57.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:57.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:58.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:58.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:58.130 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[102314]: pgmap v1131: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 767 op/s 2024-09-13T15:02:58.130 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:02:58.130 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:02:58.131 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:58.131 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3769923799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[107038]: pgmap v1131: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 767 op/s 2024-09-13T15:02:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:02:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:02:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:58.131 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:02:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3769923799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:58.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:58.250 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:58.330 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:58 smithi172 ceph-mon[100696]: pgmap v1131: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 767 op/s 2024-09-13T15:02:58.330 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:02:58.330 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:02:58.330 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:02:58.330 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:02:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3769923799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:02:58.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:02:58.378 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:58.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:02:58.383 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:02:58.383 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:02:58.384 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:02:58.384 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:02:58.384 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:02:58.384 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:02:58.384 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:02:58.384 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:02:58.384 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:02:58.385 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:02:58.385 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:02:58.385 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:02:58.385 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:02:58.385 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:02:58.385 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:02:58.385 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:02:58.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:58.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:58.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.661 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:58.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:58.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:58.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.789 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:58.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:58.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:58.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:58.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:58.928 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:02:59.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:59.050 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:59.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:02:59.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:02:59.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:59.325 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:59.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:02:59.453 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:02:59.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:59.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:02:59.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:02:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:02:59.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:02:59.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:02:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:59.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:02:59.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:02:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:02:59.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:02:59.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:00.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:00.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:00.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.251 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:00.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:00.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:03:00.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:00 smithi154 ceph-mon[102314]: pgmap v1132: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 697 op/s 2024-09-13T15:03:00.352 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:00 smithi154 ceph-mon[107038]: pgmap v1132: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 697 op/s 2024-09-13T15:03:00.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:00.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:00 smithi172 ceph-mon[100696]: pgmap v1132: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 697 op/s 2024-09-13T15:03:00.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:00.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:00.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:00.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:00.755 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:00.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:00.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:00.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:00.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:00.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:01.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:01.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:01.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.239 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:01.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:01.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:01.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:01.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:01.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:01.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:01.614 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:01.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:01.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:01.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:01.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:01.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:01.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:02.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:02.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:02.080 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:03:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:03:01] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-13T15:03:02.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:02.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:02.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:02.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:02.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:02.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:02 smithi154 ceph-mon[102314]: pgmap v1133: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T15:03:02.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:02 smithi154 ceph-mon[107038]: pgmap v1133: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T15:03:02.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:02 smithi172 ceph-mon[100696]: pgmap v1133: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-13T15:03:02.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:02.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:02.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:02.722 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:02.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:02.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:02.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:02.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:02.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:02.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:03.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:03.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:03.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:03.261 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:03:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:03.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:03.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:03.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/814819723' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:03.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:03 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/814819723' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:03.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:03 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:03.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/814819723' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:03 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:03.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:03.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:03.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:03.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:03.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:03.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:03.893 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:03.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:03.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:04.020 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:04.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:04.151 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:04.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:04.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:04.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:04.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:04.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[107038]: pgmap v1134: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 681 op/s 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[107038]: osdmap e353: 8 total, 8 up, 7 in 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[102314]: pgmap v1134: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 681 op/s 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:04.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[102314]: osdmap e353: 8 total, 8 up, 7 in 2024-09-13T15:03:04.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:04.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:04 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:04 smithi172 ceph-mon[100696]: pgmap v1134: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 681 op/s 2024-09-13T15:03:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:04 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:04 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:04 smithi172 ceph-mon[100696]: osdmap e353: 8 total, 8 up, 7 in 2024-09-13T15:03:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:04 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:04.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:04.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:04.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:04.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:04.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:04.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:04.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:04.942 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:04.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:05.036 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:05.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:05.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:05.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:05.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:05.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:05.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:05.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:05.474 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:05.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:05.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:05.708 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:05.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:05.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:05.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:05.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:05.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:05.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:06.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:06.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:06.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:06.269 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:05 smithi154 ceph-mon[102314]: pgmap v1136: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-09-13T15:03:06.270 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:05 smithi154 ceph-mon[107038]: pgmap v1136: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-09-13T15:03:06.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:06.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:06.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:06.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:06.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:05 smithi172 ceph-mon[100696]: pgmap v1136: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-09-13T15:03:06.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:06.594 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:06.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:06.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:06.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:06.865 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:06.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:06.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:07.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:07.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:07.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:07.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:07.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:07.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:07.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:07.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:07.554 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:07.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:07.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:07.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:03:07.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T15:03:07.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:07.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:07.851 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:07.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:07.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:07.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:08.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:08.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:08.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:08.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:08 smithi154 ceph-mon[102314]: pgmap v1137: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-13T15:03:08.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:08 smithi154 ceph-mon[107038]: pgmap v1137: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-13T15:03:08.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:08.325 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:08.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:08.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:08 smithi172 ceph-mon[100696]: pgmap v1137: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-13T15:03:08.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:08.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:08.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:08.754 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:08.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:08.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:08.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:08.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:08.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:09.021 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:09.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:09.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:09.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:09.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:09.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:09.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:09.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:09.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:09.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:09.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:09.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:09.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:09.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:09.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:09.999 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:10.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:10.159 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:10 smithi154 ceph-mon[102314]: pgmap v1138: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-13T15:03:10.159 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:10 smithi154 ceph-mon[107038]: pgmap v1138: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-13T15:03:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:10.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:10.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:10.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:10.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:10.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:10 smithi172 ceph-mon[100696]: pgmap v1138: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 795 op/s 2024-09-13T15:03:10.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:10.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:10.654 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:10.677 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:10.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:10.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:10.849 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:03:10.850 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:10.857 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:10.857 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:10.898 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:03:10.898 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:10.924 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:10.930 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:03:10.930 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:10.949 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:10.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:10.972 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:03:10.973 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:11.005 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:03:11.006 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:11.035 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:03:11.036 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:11.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:11.070 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:03:11.070 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:11.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:11.100 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:11.103 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:11.109 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:03:11.110 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:11.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:11.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:11.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:11.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:11.357 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:11.432 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:11 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4019014126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:11.433 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:11 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4019014126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:11.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:11.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:11 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4019014126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:11.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:11.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:11.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:11.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:11.776 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:11.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:11.901 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:11.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:11.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:03:12.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:12.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:12.129 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:12 smithi154 ceph-mon[102314]: pgmap v1139: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-13T15:03:12.129 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:12 smithi154 ceph-mon[107038]: pgmap v1139: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-13T15:03:12.129 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:03:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:03:11] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-13T15:03:12.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:12.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:12.212 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:12 smithi172 ceph-mon[100696]: pgmap v1139: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-13T15:03:12.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:12.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:12.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:12.487 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:12.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:12.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:12.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:12.751 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:12.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:12.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:12.881 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:12.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:12.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:13.009 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:13.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:13.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:13.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:13.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:13.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:13.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:13.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:13.545 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:13.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:13.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:13.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:13.826 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:13.928 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:13.954 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:13.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:14.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:14.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:14.135 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:03:14.135 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:03:14.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.135 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:172700 8.5 8:ae445411:::benchmark_data_smithi154_261841_object56661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:03:14.131892+0000", 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.002108272, 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021412929999999998, 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 172700 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.131892+0000", 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.131892+0000", 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.131891+0000", 2024-09-13T15:03:14.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.131897+0000", 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7909999999999999e-06 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.131911+0000", 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3309000000000001e-05 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.131917+0000", 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7109999999999998e-06 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.131977+0000", 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9058999999999998e-05 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132114+0000", 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013726600000000001 2024-09-13T15:03:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132235+0000", 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121148 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.35787.0:172696 8.b e353/331 8:d27f3e83:::benchmark_data_smithi154_261841_object56657:head v 353'10824, mlcod=353'10822)", 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:03:14.132006+0000", 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0019945560000000002, 2024-09-13T15:03:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020841240000000001, 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132006+0000", 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132006+0000", 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132005+0000", 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132010+0000", 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2020000000000003e-06 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132019+0000", 2024-09-13T15:03:14.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5590000000000006e-06 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132024+0000", 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.639e-06 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132064+0000", 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0568999999999997e-05 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:14.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:14.132077+0000", 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3209e-05 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 2 2024-09-13T15:03:14.144 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:03:14.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:14 smithi154 ceph-mon[102314]: pgmap v1140: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-13T15:03:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:14 smithi154 ceph-mon[107038]: pgmap v1140: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-13T15:03:14.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:14.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:14.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:14.390 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:14.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:14.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:14 smithi172 ceph-mon[100696]: pgmap v1140: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 798 op/s 2024-09-13T15:03:14.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:14.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:14.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:14.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:14.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:14.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:14.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:14.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:14.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:14.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:15.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:15.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:03:15.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:15.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:15.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:15.292 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:15.364 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:15.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:03:15.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:15.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:15.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:15.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:15.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:15.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:15.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:15.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:15.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:15.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:15.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:16.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:16.076 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:16.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:16.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:16.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:16.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:16.469 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:16 smithi154 ceph-mon[102314]: pgmap v1141: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 803 op/s 2024-09-13T15:03:16.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:16 smithi154 ceph-mon[107038]: pgmap v1141: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 803 op/s 2024-09-13T15:03:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:16 smithi172 ceph-mon[100696]: pgmap v1141: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 803 op/s 2024-09-13T15:03:16.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:16.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:16.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:16.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:16.766 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:16.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:16.891 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:16.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:16.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:16.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:17.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:17.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:17.165 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:17.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:17.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:17.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:17.431 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:17.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:17.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:17.616 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:03:17.616 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:03:17.617 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:03:17.617 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:03:17.617 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:03:17.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:17.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:17.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:17.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:17.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:17.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:17.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:18.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:18.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:18.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:18.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:18 smithi172 ceph-mon[100696]: pgmap v1142: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-13T15:03:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1789784496' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:18.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:03:18.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.387 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:18.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:18.403 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:03:18.403 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:18 smithi154 ceph-mon[102314]: pgmap v1142: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-13T15:03:18.404 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1789784496' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:18.404 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:18 smithi154 ceph-mon[107038]: pgmap v1142: 89 pgs: 89 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-13T15:03:18.404 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1789784496' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:18.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:18.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:18.582 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:18.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:18.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:18.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:18.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:18.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:18.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:18.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:18.964 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:03:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=cleanup t=2024-09-13T15:03:18.842185098Z level=info msg="Completed cleanup jobs" duration=32.099507ms 2024-09-13T15:03:18.964 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:03:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=plugins.update.checker t=2024-09-13T15:03:18.951059916Z level=info msg="Update check succeeded" duration=65.237765ms 2024-09-13T15:03:18.964 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:03:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana.update.checker t=2024-09-13T15:03:18.963542281Z level=info msg="Update check succeeded" duration=67.681617ms 2024-09-13T15:03:19.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:19.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:19.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:19 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1789784496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:19 smithi172 ceph-mon[100696]: osdmap e354: 8 total, 8 up, 7 in 2024-09-13T15:03:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:19 smithi172 ceph-mon[100696]: pgmap v1144: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 835 op/s 2024-09-13T15:03:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.242 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:19.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:19.291 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:19.291 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1789784496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:19.291 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[102314]: osdmap e354: 8 total, 8 up, 7 in 2024-09-13T15:03:19.291 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[102314]: pgmap v1144: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 835 op/s 2024-09-13T15:03:19.293 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:19.293 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1789784496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:19.293 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[107038]: osdmap e354: 8 total, 8 up, 7 in 2024-09-13T15:03:19.293 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:19 smithi154 ceph-mon[107038]: pgmap v1144: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 835 op/s 2024-09-13T15:03:19.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:19.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:19.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:19.497 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:19.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:19.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:19.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:19.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:19.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:19.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:19.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:20.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:20.047 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:20.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:20.271 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:20.357 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:03:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:03:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:03:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:03:20.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150521 7.8 7:11a94199:::benchmark_data_smithi154_225924_object40257:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.290904+0000", 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 349.06246412199999, 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.023286141, 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150521 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290904+0000", 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290904+0000", 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290902+0000", 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290908+0000", 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9279999999999998e-06 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290919+0000", 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.132e-05 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290925+0000", 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2220000000000001e-06 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290978+0000", 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3409000000000003e-05 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290986+0000", 2024-09-13T15:03:20.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3659999999999992e-06 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310964+0000", 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0199773940000001 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314190+0000", 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032259120000000001 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150526 7.8 7:16bba6af:::benchmark_data_smithi154_225924_object40262:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.290939+0000", 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 349.06242904599998, 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0235627060000001, 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150526 2024-09-13T15:03:20.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290939+0000", 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290939+0000", 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290938+0000", 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290940+0000", 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0020000000000001e-06 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290945+0000", 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7840000000000003e-06 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290949+0000", 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.518e-06 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.291003+0000", 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4715e-05 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.291009+0000", 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1359999999999996e-06 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314426+0000", 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0234177849999999 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.314501+0000", 2024-09-13T15:03:20.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5047000000000002e-05 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210203 7.1 7:855d5ed3:::benchmark_data_smithi154_225924_object99939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.562662+0000", 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 212.790705798, 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0491108170000001, 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.368 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210203 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562662+0000", 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562662+0000", 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562661+0000", 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562666+0000", 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.023e-06 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562694+0000", 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8348e-05 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.562702+0000", 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3300000000000001e-06 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.570904+0000", 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082024420000000008 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.571040+0000", 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000135506 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1,3", 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.571167+0000", 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127704 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.574660+0000", 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003492878 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.576360+0000", 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017001539999999999 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.581800+0000", 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054399729999999999 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.581814+0000", 2024-09-13T15:03:20.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3392e-05 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.611724+0000", 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.029910165 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.611773+0000", 2024-09-13T15:03:20.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8810999999999997e-05 2024-09-13T15:03:20.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.375 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.376 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.376 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210228 7.1 7:8f77d107:::benchmark_data_smithi154_225924_object99964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:03:20.376 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.597289+0000", 2024-09-13T15:03:20.376 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 212.75607822399999, 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0257111800000001, 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210228 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597289+0000", 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597289+0000", 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597288+0000", 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597296+0000", 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7209999999999994e-06 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597311+0000", 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5577e-05 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597319+0000", 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0369999999999995e-06 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597361+0000", 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2259999999999999e-05 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597371+0000", 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3510000000000008e-06 2024-09-13T15:03:20.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.612091+0000", 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0147197939999999 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.612177+0000", 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6237999999999997e-05 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:20.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.612273+0000", 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5977999999999995e-05 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617796+0000", 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055232850000000002 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617836+0000", 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9480999999999997e-05 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.622948+0000", 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051121159999999999 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.622978+0000", 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0528999999999999e-05 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623001+0000", 2024-09-13T15:03:20.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2422e-05 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210229 7.1 7:8b22e94d:::benchmark_data_smithi154_225924_object99965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.597333+0000", 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 212.75603509499999, 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.028199431, 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.384 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210229 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597333+0000", 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597333+0000", 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597332+0000", 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597335+0000", 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2430000000000001e-06 2024-09-13T15:03:20.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597343+0000", 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5750000000000003e-06 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597350+0000", 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8739999999999997e-06 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597390+0000", 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9935999999999998e-05 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.597395+0000", 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0420000000000002e-06 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.615283+0000", 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0178883190000001 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:20.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.615370+0000", 2024-09-13T15:03:20.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6768999999999995e-05 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.615445+0000", 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5568000000000006e-05 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.619948+0000", 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045030319999999997 2024-09-13T15:03:20.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.619965+0000", 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6731999999999998e-05 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625482+0000", 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055164360000000004 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625510+0000", 2024-09-13T15:03:20.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8212999999999999e-05 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625532+0000", 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2183000000000001e-05 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210246 7.1 7:85e1b4c7:::benchmark_data_smithi154_225924_object99982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.613962+0000", 2024-09-13T15:03:20.393 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 212.73940555600001, 2024-09-13T15:03:20.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.011727381, 2024-09-13T15:03:20.394 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.394 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.394 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.394 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.394 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.394 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210246 2024-09-13T15:03:20.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.395 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613962+0000", 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613962+0000", 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613961+0000", 2024-09-13T15:03:20.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613967+0000", 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6039999999999998e-06 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613981+0000", 2024-09-13T15:03:20.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3553000000000001e-05 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.613987+0000", 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9050000000000002e-06 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.614024+0000", 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7338000000000001e-05 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.614031+0000", 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2400000000000004e-06 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617851+0000", 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.003820465 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617918+0000", 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6568000000000004e-05 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.617981+0000", 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3125000000000003e-05 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623045+0000", 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050646149999999997 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625537+0000", 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024919159999999998 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625618+0000", 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0315999999999995e-05 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625648+0000", 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0035e-05 2024-09-13T15:03:20.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625690+0000", 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1974000000000002e-05 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210303 7.1 7:8502be6f:::benchmark_data_smithi154_225924_object100039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.669545+0000", 2024-09-13T15:03:20.403 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 212.68382289499999, 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95965307700000002, 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210303 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669545+0000", 2024-09-13T15:03:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669545+0000", 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669543+0000", 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669551+0000", 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9289999999999998e-06 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669567+0000", 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5787e-05 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669576+0000", 2024-09-13T15:03:20.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6289999999999999e-06 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669619+0000", 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3779000000000002e-05 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.669627+0000", 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2660000000000004e-06 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.619977+0000", 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95034996500000002 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.620055+0000", 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7961999999999997e-05 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.620156+0000", 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101539 2024-09-13T15:03:20.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625574+0000", 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054177369999999997 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625698+0000", 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000123786 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.629150+0000", 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003452685 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.629176+0000", 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5774000000000001e-05 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.629198+0000", 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1840000000000001e-05 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.410 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210385 7.1 7:886f9cba:::benchmark_data_smithi154_225924_object100121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.754596+0000", 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 212.59877200400001, 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.89118368199999998, 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210385 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754596+0000", 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754596+0000", 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754594+0000", 2024-09-13T15:03:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754603+0000", 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8880000000000003e-06 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754621+0000", 2024-09-13T15:03:20.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7768e-05 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754630+0000", 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.312e-06 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754681+0000", 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1175999999999999e-05 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.754689+0000", 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3839999999999997e-06 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623069+0000", 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.86837958500000001 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623165+0000", 2024-09-13T15:03:20.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6273000000000005e-05 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.623278+0000", 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112574 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.631655+0000", 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0083774060000000004 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.633250+0000", 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001594449 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645708+0000", 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012458545 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645740+0000", 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1436999999999997e-05 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645779+0000", 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9666000000000003e-05 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:210391 7.1 7:89b8523a:::benchmark_data_smithi154_225924_object100127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:47.760314+0000", 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 212.593053849, 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.88564320399999996, 2024-09-13T15:03:20.418 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 210391 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760314+0000", 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760314+0000", 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760312+0000", 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760320+0000", 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0709999999999995e-06 2024-09-13T15:03:20.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760336+0000", 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5668e-05 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760344+0000", 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.439e-06 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760412+0000", 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7853000000000002e-05 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:47.760429+0000", 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7289999999999999e-05 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625724+0000", 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.865294118 2024-09-13T15:03:20.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625795+0000", 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1068000000000005e-05 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.625864+0000", 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9266999999999999e-05 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.633215+0000", 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0073508699999999998 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645821+0000", 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012605893999999999 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645900+0000", 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299000000000001e-05 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645924+0000", 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4477999999999998e-05 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:48.645957+0000", 2024-09-13T15:03:20.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2577999999999999e-05 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:58930 8.2 8:416c7ae0:::benchmark_data_smithi154_261841_object58929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:01:06.071362+0000", 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.28200616500001, 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.13204650800000001, 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58930 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071362+0000", 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071362+0000", 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071360+0000", 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071404+0000", 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4690000000000001e-05 2024-09-13T15:03:20.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071416+0000", 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1691e-05 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071423+0000", 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6839999999999999e-06 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071464+0000", 2024-09-13T15:03:20.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0969e-05 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071591+0000", 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012762 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071733+0000", 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014187499999999999 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.075396+0000", 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003662851 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.075422+0000", 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5408000000000001e-05 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.203307+0000", 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12788508100000001 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.203365+0000", 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8641999999999999e-05 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.203408+0000", 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2796999999999997e-05 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:58931 8.2 8:4c64a6d1:::benchmark_data_smithi154_261841_object58930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:01:06.071949+0000", 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.28141869500001, 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.13691394600000001, 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 58931 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071949+0000", 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071949+0000", 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071948+0000", 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.071999+0000", 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0615000000000002e-05 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.072010+0000", 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1326e-05 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.072017+0000", 2024-09-13T15:03:20.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1670000000000002e-06 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.073445+0000", 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001428108 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.073560+0000", 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011495899999999999 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:03:20.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.073687+0000", 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012675699999999999 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.075506+0000", 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001819216 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.075568+0000", 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1878999999999995e-05 2024-09-13T15:03:20.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.208776+0000", 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.13320823200000001 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.208825+0000", 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8967000000000001e-05 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.208863+0000", 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7623000000000002e-05 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.35787.0:58945 8.4 e339/331 8:263eeb89:::benchmark_data_smithi154_261841_object58944:head v 339'3659, mlcod=339'3657)", 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:01:06.082160+0000", 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.27120800899999, 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.128228487, 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.082160+0000", 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.082160+0000", 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.082158+0000", 2024-09-13T15:03:20.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.082184+0000", 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5626000000000002e-05 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.203267+0000", 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.121083383 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.203281+0000", 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3624e-05 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.203334+0000", 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2933999999999998e-05 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.203360+0000", 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6239999999999999e-05 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.210354+0000", 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0069935079999999998 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:06.210388+0000", 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4479e-05 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:75651 8.5 8:a0e744cc:::benchmark_data_smithi154_261841_object75650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:01:20.779258+0000", 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 119.57410964499999, 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.134365237, 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 75651 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779258+0000", 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779258+0000", 2024-09-13T15:03:20.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779257+0000", 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779303+0000", 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5593000000000002e-05 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779313+0000", 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8980000000000008e-06 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779319+0000", 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8290000000000002e-06 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779361+0000", 2024-09-13T15:03:20.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1396000000000001e-05 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779480+0000", 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119033 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.779585+0000", 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105128 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.781620+0000", 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002034784 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.781788+0000", 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168178 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.793310+0000", 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011522179 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913557+0000", 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.120246461 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913623+0000", 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6692000000000005e-05 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.450 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:75656 8.5 8:a58628cd:::benchmark_data_smithi154_261841_object75655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:01:20.786647+0000", 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 119.566720373, 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12709884199999999, 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 75656 2024-09-13T15:03:20.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786647+0000", 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786647+0000", 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786645+0000", 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999971 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786754+0000", 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000109535 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786783+0000", 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8572e-05 2024-09-13T15:03:20.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786811+0000", 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8096999999999999e-05 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786851+0000", 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0080999999999998e-05 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.786940+0000", 2024-09-13T15:03:20.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9081999999999999e-05 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.787028+0000", 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8361000000000003e-05 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.791424+0000", 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043954670000000001 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.791448+0000", 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4314000000000001e-05 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913699+0000", 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12225119 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913726+0000", 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6961000000000002e-05 2024-09-13T15:03:20.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913746+0000", 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0001999999999999e-05 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:75662 8.5 8:aaaacb7c:::benchmark_data_smithi154_261841_object75661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:01:20.787755+0000", 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 119.565612839, 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12606756, 2024-09-13T15:03:20.457 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 75662 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.787755+0000", 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.787755+0000", 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.787754+0000", 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.787824+0000", 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9554000000000005e-05 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.787835+0000", 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1392999999999999e-05 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.787842+0000", 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.708e-06 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.789027+0000", 2024-09-13T15:03:20.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011856880000000001 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.789168+0000", 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014023099999999999 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.789271+0000", 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103173 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.791549+0000", 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022780999999999999 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.792009+0000", 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046065899999999998 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913783+0000", 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12177371400000001 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913805+0000", 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1393e-05 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913822+0000", 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7833999999999999e-05 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.463 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:75670 8.5 8:a85c5ec7:::benchmark_data_smithi154_261841_object75669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:01:20.796191+0000", 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 119.55717673700001, 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.12898749900000001, 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 75670 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.796191+0000", 2024-09-13T15:03:20.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.796191+0000", 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.796190+0000", 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.796240+0000", 2024-09-13T15:03:20.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0658000000000002e-05 2024-09-13T15:03:20.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913483+0000", 2024-09-13T15:03:20.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.11724281 2024-09-13T15:03:20.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913518+0000", 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4261999999999998e-05 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913852+0000", 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033403099999999999 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.913951+0000", 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9233000000000003e-05 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.914047+0000", 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6261999999999994e-05 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.922257+0000", 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082101489999999999 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.922291+0000", 2024-09-13T15:03:20.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3441000000000001e-05 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.924992+0000", 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002701565 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.925136+0000", 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014332 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:01:20.925178+0000", 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2889999999999998e-05 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:132758 8.5 8:ac698dd2:::benchmark_data_smithi154_261841_object16719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:02:24.556406+0000", 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 55.796961257, 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083582114999999998, 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.471 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 132758 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556406+0000", 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556406+0000", 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556405+0000", 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556412+0000", 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3649999999999998e-06 2024-09-13T15:03:20.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556431+0000", 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8229000000000001e-05 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556439+0000", 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3270000000000001e-06 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556484+0000", 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5426999999999997e-05 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556588+0000", 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104141 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.556699+0000", 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110433 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.560258+0000", 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035586480000000002 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.560370+0000", 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112532 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.639900+0000", 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.079529919000000004 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.639951+0000", 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1081000000000001e-05 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:24.639989+0000", 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7475000000000003e-05 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:157133 8.2 8:45cec752:::benchmark_data_smithi154_261841_object41094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:02:54.923886+0000", 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.429481253999999, 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.101625153, 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 157133 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.478 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.923886+0000", 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.923886+0000", 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.923885+0000", 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.923892+0000", 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9949999999999999e-06 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.923905+0000", 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3570000000000001e-05 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.923913+0000", 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3209999999999999e-06 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.923953+0000", 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0368999999999999e-05 2024-09-13T15:03:20.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.924030+0000", 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6635000000000001e-05 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:03:20.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.924102+0000", 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1932000000000002e-05 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.927199+0000", 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003097097 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.927989+0000", 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079007700000000003 2024-09-13T15:03:20.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:55.025413+0000", 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.097424206999999999 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:55.025469+0000", 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6292000000000002e-05 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:55.025512+0000", 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2215000000000002e-05 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:157134 8.2 8:42692bec:::benchmark_data_smithi154_261841_object41095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:02:54.928905+0000", 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.424463095, 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.096778446000000004, 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.487 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 157134 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.928905+0000", 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.928905+0000", 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.928904+0000", 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.928909+0000", 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0900000000000004e-06 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.928919+0000", 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0546e-05 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.928924+0000", 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7369999999999997e-06 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.928957+0000", 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2876999999999997e-05 2024-09-13T15:03:20.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.929039+0000", 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2399999999999997e-05 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.929111+0000", 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1346000000000004e-05 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.932562+0000", 2024-09-13T15:03:20.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003451225 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:54.932727+0000", 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000165434 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:55.025570+0000", 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.092842456000000004 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:55.025625+0000", 2024-09-13T15:03:20.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917999999999998e-05 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:02:55.025683+0000", 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8452999999999998e-05 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:171807 8.2 8:47d4e821:::benchmark_data_smithi154_261841_object55768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-13T15:03:20.493 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:03:13.092346+0000", 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 7.261021758, 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.066860625000000007, 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 171807 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:20.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.092346+0000", 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.092346+0000", 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.092344+0000", 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.092360+0000", 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5225000000000001e-05 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.092375+0000", 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4805e-05 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.092382+0000", 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6299999999999998e-06 2024-09-13T15:03:20.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.104856+0000", 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012473496000000001 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.104972+0000", 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116298 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,5", 2024-09-13T15:03:20.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.105059+0000", 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7301999999999999e-05 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.108303+0000", 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032436169999999999 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.116370+0000", 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008067612 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.159131+0000", 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042760744000000003 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.159173+0000", 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1990999999999998e-05 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:13.159207+0000", 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3349000000000002e-05 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.499 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.500 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:20.500 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:20.500 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:03:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:20.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:20.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:20.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:20.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:20.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:20.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:20.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:20.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:21.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:03:21.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:21.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:21.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:03:21.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-13T15:03:21.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:21.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:21.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:21 smithi172 ceph-mon[100696]: pgmap v1145: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-13T15:03:21.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:21.519 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:21.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:21.641 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:21 smithi154 ceph-mon[102314]: pgmap v1145: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-13T15:03:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:21 smithi154 ceph-mon[107038]: pgmap v1145: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-13T15:03:21.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:21.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:21.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:21.901 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:21.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:22.004 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.047 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:22.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:22.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:22.174 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:03:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:03:21] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-13T15:03:22.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:22.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:22.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:22.413 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:22.428 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:22.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:22.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:22.558 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:22.558 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:22.559 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:22.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:22.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:22.563 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:22.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:22.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:22.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:03:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:03:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:22.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:22.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:22.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:22.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:22.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:23.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:23.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:23.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:23.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:23.322 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:23.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:23.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:23.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:23.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:23.597 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:23.639 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:23 smithi172 ceph-mon[100696]: pgmap v1146: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-13T15:03:23.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:23.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:23 smithi154 ceph-mon[102314]: pgmap v1146: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-13T15:03:23.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:23 smithi154 ceph-mon[107038]: pgmap v1146: 89 pgs: 89 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 830 op/s 2024-09-13T15:03:23.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:23.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:23.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:23.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:23.998 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:23.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:24.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:24.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:24.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:24.237 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:03:24.237 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:03:24.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.237 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:180926 8.9 8:9ded7e42:::benchmark_data_smithi154_261841_object64887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-13T15:03:24.237 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:03:24.220898+0000", 2024-09-13T15:03:24.237 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.014241258, 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014263125, 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 180926 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.220898+0000", 2024-09-13T15:03:24.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.220898+0000", 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.220897+0000", 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.220902+0000", 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5979999999999996e-06 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.220914+0000", 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1279e-05 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:24.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.220920+0000", 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5250000000000002e-06 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.220968+0000", 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7333999999999998e-05 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.221047+0000", 2024-09-13T15:03:24.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9680999999999998e-05 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.221112+0000", 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4712e-05 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.224941+0000", 2024-09-13T15:03:24.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038291050000000002 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:24.229202+0000", 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004261305 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-13T15:03:24.243 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:03:24.265 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:24.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:24.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3935018561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:24.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:24.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:24.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:24.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:24.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3935018561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:24.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3935018561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:24.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:24.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:24.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:24.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:24.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:24.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:03:25.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:25.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:25.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:25.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:25.227 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:25.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:25.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:25.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:25.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:25.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:25.614 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:25.646 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:03:25.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:03:25.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:03:25.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:03:25.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:03:25.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:03:25.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:03:25.647 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:25.648 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:25.685 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:25 smithi172 ceph-mon[100696]: pgmap v1147: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:03:25.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:25.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:25.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:25.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:25.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:25.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:25 smithi154 ceph-mon[102314]: pgmap v1147: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:03:25.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:25 smithi154 ceph-mon[107038]: pgmap v1147: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:03:25.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:26.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:26.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.134 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:26.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:26.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:26.263 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:26.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:26.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:26.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:26.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:26.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:26.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:26.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:26.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:26.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:26.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:26.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:26.951 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:26.951 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:27.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:27.077 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:27.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:27.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:27.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:27.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:27.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:27.485 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:27.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:27.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:27.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:03:27.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:27.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:27.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:27.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:27.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:27.922 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:27.996 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:28.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:28.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:28.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:28.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:28 smithi154 ceph-mon[107038]: pgmap v1148: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 806 op/s 2024-09-13T15:03:28.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:28.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:28 smithi154 ceph-mon[102314]: pgmap v1148: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 806 op/s 2024-09-13T15:03:28.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:28.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:28.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:28.380 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:28 smithi172 ceph-mon[100696]: pgmap v1148: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 806 op/s 2024-09-13T15:03:28.380 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:28.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:28.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:28.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:28.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:28.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:28.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:28.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:28.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:28.861 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:28.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:28.989 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:28.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:29.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:29.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:29.268 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:29.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:29.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:29.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:29.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:29.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:29.681 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:29.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:29.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:29.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:29.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:29.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:29.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:30.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:30.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:30.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:30.190 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:30.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:30.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:30.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:30.368 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:30 smithi172 ceph-mon[100696]: pgmap v1149: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T15:03:30.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:30.441 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:30 smithi154 ceph-mon[102314]: pgmap v1149: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T15:03:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:30 smithi154 ceph-mon[107038]: pgmap v1149: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 738 op/s 2024-09-13T15:03:30.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:30.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:30.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:30.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:30.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:30.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:30.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:30.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:30.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:30.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:31.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:31.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:31.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:31.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:31.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:31.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:31.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:31.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:31.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:31.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:31.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:31.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:31.858 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:31.858 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:31.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:31.921 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:31.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:32.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:32.048 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:32.090 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:03:32.091 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:03:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:03:31] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-13T15:03:32.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:32.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:32.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:32.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:32.389 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:32 smithi154 ceph-mon[102314]: pgmap v1150: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T15:03:32.389 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3296374258' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:32.389 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:32 smithi154 ceph-mon[107038]: pgmap v1150: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T15:03:32.389 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3296374258' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:32.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:32.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-13T15:03:32.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:32 smithi172 ceph-mon[100696]: pgmap v1150: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-13T15:03:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3296374258' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:03:32.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:03:32.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:32.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:32.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:32.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:32.782 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:32.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:32.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:32.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:33.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:33.037 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:33.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:33.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:33.224 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:33 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:33.224 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3296374258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:03:33.224 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:33 smithi172 ceph-mon[100696]: osdmap e355: 8 total, 8 up, 7 in 2024-09-13T15:03:33.224 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:33 smithi172 ceph-mon[100696]: pgmap v1152: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 818 op/s 2024-09-13T15:03:33.231 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:33.244 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:33.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:33.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:33.384 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:33.384 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3296374258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:03:33.384 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[102314]: osdmap e355: 8 total, 8 up, 7 in 2024-09-13T15:03:33.384 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[102314]: pgmap v1152: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 818 op/s 2024-09-13T15:03:33.385 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:33.385 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3296374258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:03:33.385 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[107038]: osdmap e355: 8 total, 8 up, 7 in 2024-09-13T15:03:33.385 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:33 smithi154 ceph-mon[107038]: pgmap v1152: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 818 op/s 2024-09-13T15:03:33.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:33.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:33.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:33.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:33.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:33.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:33.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:33.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:33.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:33.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:33.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:33.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:34.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:34.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:34.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:34.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:34.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:34.388 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:34.490 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:34.521 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:34.586 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:34.586 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:34.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:34.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:34.752 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:34.778 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:34.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:34.904 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:34.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:35.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:35.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:35.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:35.168 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:35.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:03:35.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:35.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:35.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:35.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:35.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:03:35.520 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.564 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:35.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:35.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:35.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:35.776 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:35.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.840 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:35.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:35.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:35.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:35.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:36.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:36.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:36.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:36 smithi154 ceph-mon[102314]: pgmap v1153: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 817 op/s 2024-09-13T15:03:36.219 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:36 smithi154 ceph-mon[107038]: pgmap v1153: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 817 op/s 2024-09-13T15:03:36.219 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:36 smithi172 ceph-mon[100696]: pgmap v1153: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 817 op/s 2024-09-13T15:03:36.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:36.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:36.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:36.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:36.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:36.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:36.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:36.732 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:36.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:36.857 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:36.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:36.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:36.960 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:03:36.961 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:36.997 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:03:36.997 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:37.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:37.027 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:03:37.027 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:37.071 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:03:37.071 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:37.102 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:03:37.102 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:37.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:37.138 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:03:37.139 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:37.139 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:37 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2009611199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:37.139 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:37 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2009611199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:37.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:37.173 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:03:37.174 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:37.213 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:03:37.214 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:37.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:37.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:37.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:37.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:37.468 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:37 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2009611199' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:37.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:37.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:37.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:37.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:37.767 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:03:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:03:37.767 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:03:37.768 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:03:37.768 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:03:37.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:37.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:37.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:37.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:37.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:38.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:38.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:38.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:38.269 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:38.338 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:38.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:38.398 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:38 smithi154 ceph-mon[102314]: pgmap v1154: 89 pgs: 89 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:03:38.399 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:38 smithi154 ceph-mon[107038]: pgmap v1154: 89 pgs: 89 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:03:38.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:38.484 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:38 smithi172 ceph-mon[100696]: pgmap v1154: 89 pgs: 89 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:03:38.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:38.548 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:38.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:38.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:38.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:38.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:38.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:03:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:38.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:38.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-13T15:03:38.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:38.989 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:38.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:39.067 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:39.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:39.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:39.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:39.296 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:39.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:39.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:39.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:39 smithi154 ceph-mon[102314]: pgmap v1155: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:03:39.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:39 smithi154 ceph-mon[107038]: pgmap v1155: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:03:39.477 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:39 smithi172 ceph-mon[100696]: pgmap v1155: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:03:39.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:39.567 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:39.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:39.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:39.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:39.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:39.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:39.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:39.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:40.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.135 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:40.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:40.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:03:40.149 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:40.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:40.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:40.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:40.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:40.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:40.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:40.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:40.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:40.876 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:03:40.876 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:03:40.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150525 7.c 7:3a1defd9:::benchmark_data_smithi154_225924_object40261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.271468+0000", 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 369.601619814, 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.038483329, 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150525 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271468+0000", 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271468+0000", 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271467+0000", 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271474+0000", 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.374e-06 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271489+0000", 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4942e-05 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271497+0000", 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4499999999999998e-06 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271547+0000", 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9669000000000002e-05 2024-09-13T15:03:40.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271709+0000", 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016256800000000001 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271846+0000", 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013647600000000001 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.276339+0000", 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044934969999999999 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.278783+0000", 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024441889999999998 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290520+0000", 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011737081 2024-09-13T15:03:40.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290532+0000", 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1182999999999999e-05 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.309898+0000", 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.019366231 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.309952+0000", 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4027999999999998e-05 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150538 7.c 7:32d10010:::benchmark_data_smithi154_225924_object40274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.311705+0000", 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 369.56138358499999, 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0082904450000001, 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.884 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150538 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311705+0000", 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311705+0000", 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311703+0000", 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311710+0000", 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7000000000000002e-06 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311721+0000", 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1728e-05 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311727+0000", 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4070000000000001e-06 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311778+0000", 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1570999999999998e-05 2024-09-13T15:03:40.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311789+0000", 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0328e-05 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310250+0000", 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99846129900000002 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310383+0000", 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013281299999999999 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310500+0000", 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117446 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.318391+0000", 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078911080000000008 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319857+0000", 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014651569999999999 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319909+0000", 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2748999999999999e-05 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319995+0000", 2024-09-13T15:03:40.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5818999999999999e-05 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198815 7.d 7:bb1741ae:::benchmark_data_smithi154_225924_object88551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.44833281199999, 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080673513000000002, 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198815 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424758+0000", 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8930000000000001e-06 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424765+0000", 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8210000000000004e-06 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424769+0000", 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7369999999999997e-06 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428701+0000", 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039317220000000003 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428826+0000", 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012472699999999999 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428948+0000", 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012246300000000001 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.432869+0000", 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003920626 2024-09-13T15:03:40.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.466811+0000", 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033941885999999998 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505293+0000", 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038482159000000002 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505389+0000", 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6051000000000006e-05 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505429+0000", 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198852 7.7 7:e4156631:::benchmark_data_smithi154_225924_object88588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.897 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.510600+0000", 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.362488681, 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.098632827000000006, 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198852 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510600+0000", 2024-09-13T15:03:40.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510600+0000", 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510599+0000", 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510604+0000", 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839999999999997e-06 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510615+0000", 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1566000000000001e-05 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510622+0000", 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510665+0000", 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2363000000000001e-05 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510782+0000", 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117868 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510914+0000", 2024-09-13T15:03:40.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013160799999999999 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.515165+0000", 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042514379999999997 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.518048+0000", 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002882201 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609141+0000", 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.091092993999999997 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609191+0000", 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0241999999999998e-05 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609232+0000", 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1523999999999997e-05 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.903 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198899 7.7 7:e8acc7c2:::benchmark_data_smithi154_225924_object88635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.610904+0000", 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.262184424, 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.078514312000000003, 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.904 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198899 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610904+0000", 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610904+0000", 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610897+0000", 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999928 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610911+0000", 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4121000000000001e-05 2024-09-13T15:03:40.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610925+0000", 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4239e-05 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610932+0000", 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2400000000000001e-06 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610979+0000", 2024-09-13T15:03:40.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7040999999999999e-05 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.611109+0000", 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013000300000000001 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.611226+0000", 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011720800000000001 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.615610+0000", 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043831440000000003 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.617396+0000", 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017859589999999999 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689331+0000", 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.071935456999999994 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689380+0000", 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8606999999999997e-05 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689418+0000", 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8600999999999999e-05 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.911 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198901 7.c 7:30e20b28:::benchmark_data_smithi154_225924_object88637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.614143+0000", 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.25894548399998, 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.088790589000000003, 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198901 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614143+0000", 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614143+0000", 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614141+0000", 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614147+0000", 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9830000000000001e-06 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614159+0000", 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.168e-05 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614166+0000", 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6159999999999998e-06 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614211+0000", 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5210000000000003e-05 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614339+0000", 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012781500000000001 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614459+0000", 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120282 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.618850+0000", 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043914069999999999 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.619032+0000", 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018138900000000001 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702829+0000", 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083797778000000003 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702886+0000", 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6388999999999998e-05 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702933+0000", 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7550999999999999e-05 2024-09-13T15:03:40.917 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198956 7.c 7:37128aaa:::benchmark_data_smithi154_225924_object88692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.743414+0000", 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.12967379000003, 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.105403712, 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.918 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198956 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743414+0000", 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743414+0000", 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743413+0000", 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743419+0000", 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1510000000000001e-06 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743432+0000", 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2309e-05 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743440+0000", 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5660000000000007e-06 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750304+0000", 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068643200000000001 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750419+0000", 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114232 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750550+0000", 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130949 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.754768+0000", 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042181919999999999 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.754801+0000", 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3374999999999998e-05 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848517+0000", 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.093715883999999999 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848778+0000", 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026113500000000001 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848818+0000", 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9990000000000002e-05 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198960 7.d 7:b59b0bff:::benchmark_data_smithi154_225924_object88696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.748323+0000", 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.12476532400001, 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084968948000000002, 2024-09-13T15:03:40.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198960 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748323+0000", 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748323+0000", 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748322+0000", 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748326+0000", 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7859999999999996e-06 2024-09-13T15:03:40.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748334+0000", 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9619999999999993e-06 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748339+0000", 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9570000000000001e-06 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748557+0000", 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000217216 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748697+0000", 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014085599999999999 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748819+0000", 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012206600000000001 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758177+0000", 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093571000000000001 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.765093+0000", 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006916447 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833219+0000", 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.068125854999999999 2024-09-13T15:03:40.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833252+0000", 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2867999999999998e-05 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833292+0000", 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0201999999999999e-05 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198961 7.d 7:be6e4db7:::benchmark_data_smithi154_225924_object88697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.750495+0000", 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.12259330900002, 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082969548000000004, 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198961 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.931 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750495+0000", 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750495+0000", 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750494+0000", 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750500+0000", 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2380000000000002e-06 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750513+0000", 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2578999999999999e-05 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750520+0000", 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9500000000000004e-06 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753665+0000", 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031451449999999998 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753794+0000", 2024-09-13T15:03:40.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012921499999999999 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753911+0000", 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117296 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758467+0000", 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045562709999999998 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.765159+0000", 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066919529999999996 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833387+0000", 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.068227963000000003 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833429+0000", 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1740999999999999e-05 2024-09-13T15:03:40.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833465+0000", 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5429999999999998e-05 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198966 7.7 7:e3c5df65:::benchmark_data_smithi154_225924_object88702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.758920+0000", 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 310.11416793799998, 2024-09-13T15:03:40.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.093099978999999999, 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198966 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758920+0000", 2024-09-13T15:03:40.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758920+0000", 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758919+0000", 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758927+0000", 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2209999999999997e-06 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758939+0000", 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1528999999999999e-05 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758947+0000", 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6519999999999995e-06 2024-09-13T15:03:40.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758999+0000", 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1884e-05 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.759133+0000", 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000134346 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.759259+0000", 2024-09-13T15:03:40.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000125463 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.762880+0000", 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036212050000000002 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.836898+0000", 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.074017828999999993 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.851938+0000", 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015040359 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.851994+0000", 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5305e-05 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.852020+0000", 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6689000000000001e-05 2024-09-13T15:03:40.943 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:206504 7.b e322/307 7:d19c4c5a:::benchmark_data_smithi154_225924_object96240:head v 322'12685, mlcod=0'0)", 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:42.690287+0000", 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 238.182801162, 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099866999999999994, 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690287+0000", 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690287+0000", 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690286+0000", 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690294+0000", 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8180000000000004e-06 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690309+0000", 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4773999999999999e-05 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690316+0000", 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1080000000000004e-06 2024-09-13T15:03:40.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690358+0000", 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2042000000000002e-05 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.690375+0000", 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7459e-05 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.700233+0000", 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009857345 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:42.700274+0000", 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1131999999999999e-05 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:207030 7.b e323/307 7:dd8ba281:::benchmark_data_smithi154_225924_object96766:head v 323'12720, mlcod=323'12716)", 2024-09-13T15:03:40.948 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:44.200333+0000", 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 236.672755118, 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.009194331, 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200333+0000", 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200333+0000", 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200332+0000", 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200339+0000", 2024-09-13T15:03:40.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5130000000000002e-06 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200354+0000", 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.466e-05 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200360+0000", 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.879e-06 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200398+0000", 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8108000000000002e-05 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.200421+0000", 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3777999999999998e-05 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.209496+0000", 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090747469999999993 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.209527+0000", 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1352000000000001e-05 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:207136 7.c e323/307 7:39ec0b5b:::benchmark_data_smithi154_225924_object96872:head v 323'12703, mlcod=323'12700)", 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:44.315206+0000", 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 236.557882694, 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0087497860000000007, 2024-09-13T15:03:40.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315206+0000", 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315206+0000", 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315205+0000", 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315212+0000", 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9519999999999997e-06 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315224+0000", 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2727e-05 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315232+0000", 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4930000000000001e-06 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315303+0000", 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1489999999999995e-05 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.315324+0000", 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0948e-05 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.323918+0000", 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085939659999999998 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.323955+0000", 2024-09-13T15:03:40.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7120999999999999e-05 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:207164 7.b e323/307 7:d12ce714:::benchmark_data_smithi154_225924_object96900:head v 323'12730, mlcod=323'12727)", 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:44.342079+0000", 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 236.531009047, 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0095232210000000001, 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.958 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342079+0000", 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342079+0000", 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342078+0000", 2024-09-13T15:03:40.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342086+0000", 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8650000000000001e-06 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342096+0000", 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0039e-05 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342103+0000", 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5060000000000001e-06 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342157+0000", 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4002000000000002e-05 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.342179+0000", 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1786000000000001e-05 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.351567+0000", 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093879240000000006 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.351602+0000", 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.591e-05 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:207211 7.c e323/307 7:39464d69:::benchmark_data_smithi154_225924_object96947:head v 323'12711, mlcod=323'12708)", 2024-09-13T15:03:40.962 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:44.382153+0000", 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 236.49093527900001, 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020006836, 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382153+0000", 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382153+0000", 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382152+0000", 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382163+0000", 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0951999999999999e-05 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382182+0000", 2024-09-13T15:03:40.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9122000000000001e-05 2024-09-13T15:03:40.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382189+0000", 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5240000000000003e-06 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382256+0000", 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6303999999999994e-05 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382287+0000", 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1106999999999999e-05 2024-09-13T15:03:40.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.402061+0000", 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019774040999999999 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.402160+0000", 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8954000000000002e-05 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:207221 7.c e323/307 7:32b36228:::benchmark_data_smithi154_225924_object96957:head v 323'12712, mlcod=323'12709)", 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:44.391404+0000", 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 236.48168417400001, 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010805639000000001, 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391404+0000", 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391404+0000", 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391403+0000", 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391412+0000", 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5950000000000005e-06 2024-09-13T15:03:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391441+0000", 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8909000000000001e-05 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391449+0000", 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8499999999999994e-06 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391499+0000", 2024-09-13T15:03:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0142999999999997e-05 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.391535+0000", 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6124000000000002e-05 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.402163+0000", 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010627088999999999 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.402210+0000", 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7170999999999999e-05 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:211748 7.c e325/307 7:334318d0:::benchmark_data_smithi154_225924_object101484:head v 325'12992, mlcod=325'12990)", 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:49.746819+0000", 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 231.12626881099999, 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091030580000000007, 2024-09-13T15:03:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746819+0000", 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746819+0000", 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746819+0000", 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746827+0000", 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1829999999999994e-06 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746841+0000", 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4049000000000001e-05 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746848+0000", 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4290000000000004e-06 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746929+0000", 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1101999999999997e-05 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.746956+0000", 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6707999999999999e-05 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.755821+0000", 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088649269999999999 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:49.755923+0000", 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010156499999999999 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:212080 7.b e325/307 7:d9f119b8:::benchmark_data_smithi154_225924_object101816:head v 325'13046, mlcod=325'13043)", 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:50.067041+0000", 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 230.80604680299999, 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016940786999999999, 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067041+0000", 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067041+0000", 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067040+0000", 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067048+0000", 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.571e-06 2024-09-13T15:03:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067063+0000", 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5033e-05 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067070+0000", 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4719999999999998e-06 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067132+0000", 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1681999999999997e-05 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067166+0000", 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4097000000000002e-05 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.083935+0000", 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016768314999999999 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.083982+0000", 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7728000000000001e-05 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:212268 7.c e325/307 7:37c48763:::benchmark_data_smithi154_225924_object102004:head v 325'13030, mlcod=325'13027)", 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:50.260079+0000", 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 230.61300927400001, 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0090336659999999992, 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260079+0000", 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260079+0000", 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260078+0000", 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260085+0000", 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6930000000000001e-06 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260101+0000", 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5875e-05 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260107+0000", 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5339999999999996e-06 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260180+0000", 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3030999999999999e-05 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.260203+0000", 2024-09-13T15:03:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2481999999999999e-05 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.269060+0000", 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088568999999999991 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.269113+0000", 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3075999999999998e-05 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:213785 7.b e325/307 7:de3b20b3:::benchmark_data_smithi154_225924_object103521:head v 325'13149, mlcod=325'13147)", 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:52.012003+0000", 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 228.861084901, 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011239575, 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012003+0000", 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012003+0000", 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012002+0000", 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012011+0000", 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0909999999999999e-06 2024-09-13T15:03:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012028+0000", 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6952e-05 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012037+0000", 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4719999999999995e-06 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012092+0000", 2024-09-13T15:03:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5364000000000001e-05 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.012115+0000", 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3257000000000001e-05 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.023210+0000", 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011094387000000001 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:52.023243+0000", 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3438000000000001e-05 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:40.990 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:03:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:40.992 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:40.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:41.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:41.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:41.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:41.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:41.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:41.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:41.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:41.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:41.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:41.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:41.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:41.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:41.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:41.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:41.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:03:41.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:41.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:42.057 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:42.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:42.101 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:03:42.113 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:42.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:42 smithi154 ceph-mon[102314]: pgmap v1156: 89 pgs: 89 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:42.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4047846212' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:42.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:42 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:42.169 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:42 smithi154 ceph-mon[107038]: pgmap v1156: 89 pgs: 89 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:42.169 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4047846212' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:42.169 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:42 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:42.170 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:03:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:03:41] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-13T15:03:42.213 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:42.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:42 smithi172 ceph-mon[100696]: pgmap v1156: 89 pgs: 89 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4047846212' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:42 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:03:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:42.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:42.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:42.396 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:42.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:42.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:42.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:42.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:42.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:42.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:42.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:42.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:42.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:43.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:43.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:43.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:43.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:43.274 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:43.274 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:03:43.274 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[102314]: osdmap e356: 8 total, 8 up, 7 in 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[102314]: pgmap v1158: 89 pgs: 89 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[107038]: osdmap e356: 8 total, 8 up, 7 in 2024-09-13T15:03:43.275 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:43.276 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:43.276 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:43 smithi154 ceph-mon[107038]: pgmap v1158: 89 pgs: 89 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:43.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:43.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:43.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:43 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:03:43.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:43 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:03:43.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:43 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:03:43.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:43 smithi172 ceph-mon[100696]: osdmap e356: 8 total, 8 up, 7 in 2024-09-13T15:03:43.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:43.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:43.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:43 smithi172 ceph-mon[100696]: pgmap v1158: 89 pgs: 89 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:43.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:43.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:43.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:43.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:43.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:43.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:43.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:43.922 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:43.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:44.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.068 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:44.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:44.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:44.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:44.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:44.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:44.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:44.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:44.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:44.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:44.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:44.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:44.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:44.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:44.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.35787.0:197879 8.d e356/331 8:b83f9411:::benchmark_data_smithi154_261841_object81840:head v 356'12264, mlcod=356'12262)", 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:03:44.858604+0000", 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0035935450000000001, 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036251759999999999, 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T15:03:44.864 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858604+0000", 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858604+0000", 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:03:44.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858603+0000", 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858612+0000", 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3699999999999995e-06 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858624+0000", 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2451e-05 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858632+0000", 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3100000000000003e-06 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858736+0000", 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104193 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:03:44.858766+0000", 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0258999999999999e-05 2024-09-13T15:03:44.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:44.868 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:03:44.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:44.868 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:03:44.868 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T15:03:44.868 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-13T15:03:44.868 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:03:44.873 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:44.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:44.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:45.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:45.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:45.142 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:45.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:45.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:45.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:03:45.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:45.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:45.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:45.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:45.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:45.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:45.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:03:45.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:45.798 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:45.851 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:45.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:45.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:45.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:46.040 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:46.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:46.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:46.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:46 smithi154 ceph-mon[102314]: pgmap v1159: 89 pgs: 89 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-13T15:03:46.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:46 smithi154 ceph-mon[107038]: pgmap v1159: 89 pgs: 89 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-13T15:03:46.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:46.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:46.357 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:46 smithi172 ceph-mon[100696]: pgmap v1159: 89 pgs: 89 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 823 op/s 2024-09-13T15:03:46.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:46.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:46.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:46.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:46.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:46.665 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:03:46.666 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:03:46.675 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:46.702 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:03:46.702 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:03:46.725 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:03:46.725 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:03:46.757 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:03:46.757 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:03:46.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:46.791 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:03:46.792 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:03:46.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:46.821 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:03:46.821 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:03:46.853 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:03:46.853 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:03:46.897 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:03:46.900 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:03:46.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:46.947 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:46.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:47.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:47.076 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:47.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:47.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:47.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:47.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:47.409 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:47 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/694955272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:47.409 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:47 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/694955272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:47.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:47.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:47 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/694955272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:03:47.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:47.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:47.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:47.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:47.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:47.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:47.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:48.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:48.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:48.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:48.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:48.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:48.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:48.390 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:48 smithi154 ceph-mon[102314]: pgmap v1160: 89 pgs: 89 active+clean; 2.1 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 826 op/s 2024-09-13T15:03:48.390 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:48 smithi154 ceph-mon[107038]: pgmap v1160: 89 pgs: 89 active+clean; 2.1 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 826 op/s 2024-09-13T15:03:48.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:48.453 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:48 smithi172 ceph-mon[100696]: pgmap v1160: 89 pgs: 89 active+clean; 2.1 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 826 op/s 2024-09-13T15:03:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.455 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:48.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:48.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:48.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:48.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:48.715 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:48.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:48.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:48.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:48.991 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:48.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:49.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:49.116 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:49.116 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:49.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:49.198 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:03:49.199 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:03:49.199 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-13T15:03:49.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd in 6 2024-09-13T15:03:49.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:49.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:49.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:49.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:49.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:49.586 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:49.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:49.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:49.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:49.861 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:49.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:03:49.888 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:49.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:49.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:50.019 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:50.116 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:03:50.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:50.155 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:03:50.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:50.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:50.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:50.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:50.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:03:50.428 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:50 smithi172 ceph-mon[100696]: pgmap v1161: 89 pgs: 89 active+clean; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 826 op/s 2024-09-13T15:03:50.428 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1675944079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:50.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:50 smithi154 ceph-mon[102314]: pgmap v1161: 89 pgs: 89 active+clean; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 826 op/s 2024-09-13T15:03:50.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1675944079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:50.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:50 smithi154 ceph-mon[107038]: pgmap v1161: 89 pgs: 89 active+clean; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 826 op/s 2024-09-13T15:03:50.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1675944079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:03:50.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:50.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:50.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:50.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:50.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:50.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:50.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:50.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:50.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:51.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:51.070 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:51.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:51.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:51.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:51.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:51.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:51.463 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:51 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:51.463 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1675944079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:51.463 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:51 smithi172 ceph-mon[100696]: osdmap e357: 8 total, 8 up, 7 in 2024-09-13T15:03:51.463 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:51 smithi172 ceph-mon[100696]: pgmap v1163: 89 pgs: 89 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 912 op/s 2024-09-13T15:03:51.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1675944079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[102314]: osdmap e357: 8 total, 8 up, 7 in 2024-09-13T15:03:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[102314]: pgmap v1163: 89 pgs: 89 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 912 op/s 2024-09-13T15:03:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:03:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1675944079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:03:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[107038]: osdmap e357: 8 total, 8 up, 7 in 2024-09-13T15:03:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-mon[107038]: pgmap v1163: 89 pgs: 89 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 912 op/s 2024-09-13T15:03:51.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:51.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:51.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:51.736 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:51.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:51.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:51.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:51.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:52.009 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:52.095 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:03:52.095 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:03:52.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:52.162 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:03:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:03:51] "GET /metrics HTTP/1.1" 200 36599 "" "Prometheus/2.51.0" 2024-09-13T15:03:52.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:52.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:52.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:52.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:52.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:52.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:52.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:52.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:52.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:52.837 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:52.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:52.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:53.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:53.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:03:53.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:53.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:53.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:53.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:53.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:53.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:53.471 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:53.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:53.549 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:53.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:53.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:53.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:53.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:53.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:53.971 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:53.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:54.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:54.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:54.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:54.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:54.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:54.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:54.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:54 smithi154 ceph-mon[102314]: pgmap v1164: 89 pgs: 89 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:54 smithi154 ceph-mon[107038]: pgmap v1164: 89 pgs: 89 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:54.480 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:54 smithi172 ceph-mon[100696]: pgmap v1164: 89 pgs: 89 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:03:54.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:54.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:54.631 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:54.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:54.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:54.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:54.908 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:54.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:54.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:03:55.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:55.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:55.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.215 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:55.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:55.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:55.361 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:55.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:55.488 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:55.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:03:55.622 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:03:55.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.749 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:55.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:55.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:55.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:55.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:55.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:56.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:56.238 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:56.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:56.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:56.390 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:56 smithi154 ceph-mon[102314]: pgmap v1165: 89 pgs: 89 active+clean; 1.7 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 811 op/s 2024-09-13T15:03:56.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:56 smithi154 ceph-mon[107038]: pgmap v1165: 89 pgs: 89 active+clean; 1.7 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 811 op/s 2024-09-13T15:03:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:56 smithi172 ceph-mon[100696]: pgmap v1165: 89 pgs: 89 active+clean; 1.7 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 811 op/s 2024-09-13T15:03:56.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.522 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:56.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:56.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:56.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:56.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:56.788 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:56.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:56.924 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:56.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:57.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:57.048 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:57.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:57.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:57.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:57.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:57.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:57.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:57.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:57.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:57.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:57.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:57.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:57.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:57.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:57.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:57.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:58.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:58.090 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:58.192 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:58.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:58.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:58.344 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:58.350 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:58.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:58 smithi154 ceph-mon[107038]: pgmap v1166: 89 pgs: 89 active+clean; 1.6 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:03:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:03:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:58 smithi154 ceph-mon[102314]: pgmap v1166: 89 pgs: 89 active+clean; 1.6 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:03:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:03:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:58.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:58.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:58 smithi172 ceph-mon[100696]: pgmap v1166: 89 pgs: 89 active+clean; 1.6 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:03:58.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:03:58.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:03:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:03:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:58.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:58.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:58.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:03:58.746 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:03:58.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:58.874 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:58.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:58.974 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:03:58.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:59.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:59.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:59.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:59.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:03:59.259 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:03:59.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:03:59.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:03:59.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:03:59.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:03:59.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:03:59.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:03:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:03:59.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:03:59.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:03:59.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:03:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:03:59.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:03:59.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:03:59.994 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:03:59.995 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:00.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:00.050 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:04:00.051 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:00.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:00.093 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:04:00.093 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:00.137 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:04:00.138 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:00.174 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:04:00.175 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:00.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:00.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:00.208 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:04:00.209 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:00.245 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:04:00.245 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:00.275 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:04:00.275 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:00.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:00.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:00.427 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:00 smithi154 ceph-mon[102314]: pgmap v1167: 89 pgs: 89 active+clean; 1.6 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:04:00.427 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3759901487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:00.427 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:00 smithi154 ceph-mon[107038]: pgmap v1167: 89 pgs: 89 active+clean; 1.6 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:04:00.427 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3759901487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:00.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:00.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:00 smithi172 ceph-mon[100696]: pgmap v1167: 89 pgs: 89 active+clean; 1.6 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:04:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3759901487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:00.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:00.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:00.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:00.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:00.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:00.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:00.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:00.979 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:00.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:00.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:01.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:01.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:01.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:01.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:01.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:01.366 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:01.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:01.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:01.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:01.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:01.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:01.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:01.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:01.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:01.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:01.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:02.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:02.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:02.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:02.150 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:02 smithi154 ceph-mon[102314]: pgmap v1168: 89 pgs: 89 active+clean; 1.4 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T15:04:02.151 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:02 smithi154 ceph-mon[107038]: pgmap v1168: 89 pgs: 89 active+clean; 1.4 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T15:04:02.153 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:04:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:04:01] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.51.0" 2024-09-13T15:04:02.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:02.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:02.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:02.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:02 smithi172 ceph-mon[100696]: pgmap v1168: 89 pgs: 89 active+clean; 1.4 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T15:04:02.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:02.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:02.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:02.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:02.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:02.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:02.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:02.942 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:02.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:03.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:03.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:03.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:03.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:03.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:03.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:03.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:03.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:03 smithi154 ceph-mon[102314]: pgmap v1169: 89 pgs: 89 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 677 op/s 2024-09-13T15:04:03.474 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:03 smithi154 ceph-mon[107038]: pgmap v1169: 89 pgs: 89 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 677 op/s 2024-09-13T15:04:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:03 smithi172 ceph-mon[100696]: pgmap v1169: 89 pgs: 89 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 677 op/s 2024-09-13T15:04:03.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:03.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:03.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:03.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:03.777 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:03.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:03.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:03.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:04.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:04.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:04.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:04.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:04.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:04.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:04.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:04:04.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:04.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:04.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:04.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:04.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:04.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:04.751 INFO:teuthology.orchestra.run.smithi154.stderr:marked in osd.6. 2024-09-13T15:04:04.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:04.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:04.904 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:04.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:04.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:04 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1177665321' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T15:04:04.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:04 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T15:04:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:04 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T15:04:04.686+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e357 definitely_dead 0 2024-09-13T15:04:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:04 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1177665321' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T15:04:04.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:04 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T15:04:05.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:05.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:05.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:05.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:05.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:04 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1177665321' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T15:04:05.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:04 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-13T15:04:05.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:05.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:05.384 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:05.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:05.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:05.516 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-13T15:04:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.530 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:05.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:05.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:05.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:05.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.778 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:05.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:05.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:05.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:05.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:06.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:06.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T15:04:06.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[102314]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T15:04:06.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[102314]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T15:04:06.013 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[102314]: pgmap v1172: 89 pgs: 6 peering, 83 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 672 op/s 2024-09-13T15:04:06.013 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T15:04:06.013 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[107038]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T15:04:06.013 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[107038]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T15:04:06.014 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:05 smithi154 ceph-mon[107038]: pgmap v1172: 89 pgs: 6 peering, 83 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 672 op/s 2024-09-13T15:04:06.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:06.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:06.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:06.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:05 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-13T15:04:06.435 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:05 smithi172 ceph-mon[100696]: osdmap e358: 8 total, 8 up, 8 in 2024-09-13T15:04:06.435 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:05 smithi172 ceph-mon[100696]: osdmap e359: 8 total, 8 up, 8 in 2024-09-13T15:04:06.435 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:05 smithi172 ceph-mon[100696]: pgmap v1172: 89 pgs: 6 peering, 83 active+clean; 1.2 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 672 op/s 2024-09-13T15:04:06.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:06.436 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:04:06.436 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:04:06.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213591 8.6 8.4404a6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.777053+0000", 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.466669687, 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4666883040000001, 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for peered", 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213591 2024-09-13T15:04:06.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777053+0000", 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777053+0000", 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777054+0000", 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.652e-06 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777062+0000", 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4120000000000004e-06 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777075+0000", 2024-09-13T15:04:06.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2938e-05 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777085+0000", 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0086e-05 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.781795+0000", 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047104310000000002 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.781806+0000", 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0275000000000001e-05 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005566+0000", 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.223760775 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:06.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005578+0000", 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1734e-05 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213619 8.6 8.8f597bc6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.806539+0000", 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.437183068, 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.437246987, 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:06.442 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for peered", 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213619 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806539+0000", 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806539+0000", 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806541+0000", 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9939999999999999e-06 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806547+0000", 2024-09-13T15:04:06.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5049999999999998e-06 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806560+0000", 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3427999999999999e-05 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806568+0000", 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1559999999999998e-06 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806637+0000", 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9758999999999994e-05 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806647+0000", 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.465e-06 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005656+0000", 2024-09-13T15:04:06.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.199009452 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005665+0000", 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5350000000000004e-06 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213624 8.6 8.ea41f7d6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:04:06.447 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.812828+0000", 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.4308945149999999, 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4309948299999999, 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for peered", 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213624 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812828+0000", 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812828+0000", 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812829+0000", 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.393e-06 2024-09-13T15:04:06.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812835+0000", 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7259999999999999e-06 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812851+0000", 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5515e-05 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812864+0000", 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3078000000000001e-05 2024-09-13T15:04:06.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812918+0000", 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4129000000000003e-05 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812924+0000", 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4189999999999999e-06 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005984+0000", 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.19305983800000001 2024-09-13T15:04:06.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005994+0000", 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0309000000000001e-05 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 3 2024-09-13T15:04:06.452 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:04:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.463 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:06.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:06.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:06.590 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:06.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:06.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:06.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:06.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:06.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:06.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:06.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:07.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:07.031 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:04:07.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:07.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:07.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: 8.6 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13360 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: 8.9 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13252 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: 8.5 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13329 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: 8.7 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13366 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: 8.3 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13297 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4184584156' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:07.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: 8.6 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13360 2024-09-13T15:04:07.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T15:04:07.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: 8.9 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13252 2024-09-13T15:04:07.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: 8.5 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13329 2024-09-13T15:04:07.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: 8.7 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13366 2024-09-13T15:04:07.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: 8.3 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13297 2024-09-13T15:04:07.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4184584156' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:07.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:07 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:07.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:07.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:04:07.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:07.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:07.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:07.483 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: 8.6 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13360 2024-09-13T15:04:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: osdmap e360: 8 total, 8 up, 8 in 2024-09-13T15:04:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: 8.9 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13252 2024-09-13T15:04:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: 8.5 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13329 2024-09-13T15:04:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: 8.7 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13366 2024-09-13T15:04:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: 8.3 starting backfill to osd.6 from (0'0,0'0] MAX to 357'13297 2024-09-13T15:04:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4184584156' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:07.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:07 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:07.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:07.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:07.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:07.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:07.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:07.803 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:07.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:07.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:07.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:08.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:08.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:08.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:08.208 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:08.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:08.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[102314]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T15:04:08.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[102314]: pgmap v1175: 89 pgs: 1 activating+remapped, 3 remapped+peering, 6 peering, 79 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 1130/55545 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-09-13T15:04:08.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:08.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:08.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[107038]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T15:04:08.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:08 smithi154 ceph-mon[107038]: pgmap v1175: 89 pgs: 1 activating+remapped, 3 remapped+peering, 6 peering, 79 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 1130/55545 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-09-13T15:04:08.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:08.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:08.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:08.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:08 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:08 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:08 smithi172 ceph-mon[100696]: osdmap e361: 8 total, 8 up, 8 in 2024-09-13T15:04:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:08 smithi172 ceph-mon[100696]: pgmap v1175: 89 pgs: 1 activating+remapped, 3 remapped+peering, 6 peering, 79 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 1130/55545 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-09-13T15:04:08.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:08.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:08.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:08.745 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:08.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:08.879 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:08.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:08.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:09.007 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:09.065 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:09.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:09.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:09.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:09.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:09.340 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:09 smithi154 ceph-mon[102314]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T15:04:09.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:09 smithi154 ceph-mon[107038]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T15:04:09.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:09.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:09.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:09 smithi172 ceph-mon[100696]: osdmap e362: 8 total, 8 up, 8 in 2024-09-13T15:04:09.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:09.568 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:09.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:09.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:09.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:09.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:09.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:09.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:09.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:04:09.999 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:10.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:10.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:10.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:10.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:10.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:10 smithi154 ceph-mon[102314]: pgmap v1177: 89 pgs: 1 activating+remapped, 3 remapped+peering, 6 peering, 79 active+clean; 1.1 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 453 op/s; 1130/55545 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-09-13T15:04:10.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:10 smithi154 ceph-mon[107038]: pgmap v1177: 89 pgs: 1 activating+remapped, 3 remapped+peering, 6 peering, 79 active+clean; 1.1 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 453 op/s; 1130/55545 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-09-13T15:04:10.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.441 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:10.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:10 smithi172 ceph-mon[100696]: pgmap v1177: 89 pgs: 1 activating+remapped, 3 remapped+peering, 6 peering, 79 active+clean; 1.1 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 453 op/s; 1130/55545 objects misplaced (2.034%); 0 B/s, 0 objects/s recovering 2024-09-13T15:04:10.516 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:04:10.517 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:04:10.517 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T15:04:10.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:04:10.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:10.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:10.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:10.738 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:10.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:10.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:10.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:11.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:11.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:11.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:11.156 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:04:11.156 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:11.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:11.193 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:04:11.193 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:11.232 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:04:11.233 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:11.270 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:04:11.270 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:11.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:11.301 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:04:11.301 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:11.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:11.337 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:04:11.338 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:11.369 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:04:11.370 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:11.401 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:04:11.401 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:11.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:11.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:11.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:11.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:11.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:11.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:11.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:11.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:11.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:11.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:12.022 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:12.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:12.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:12.155 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:12 smithi154 ceph-mon[102314]: pgmap v1178: 89 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 activating+remapped, 84 active+clean; 1005 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 4574/48123 objects misplaced (9.505%); 5.3 MiB/s, 90 objects/s recovering 2024-09-13T15:04:12.155 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2462270440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:12.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:12 smithi154 ceph-mon[107038]: pgmap v1178: 89 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 activating+remapped, 84 active+clean; 1005 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 4574/48123 objects misplaced (9.505%); 5.3 MiB/s, 90 objects/s recovering 2024-09-13T15:04:12.155 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2462270440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:12.156 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:04:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:04:11] "GET /metrics HTTP/1.1" 200 36594 "" "Prometheus/2.51.0" 2024-09-13T15:04:12.173 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:12.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:12.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:12.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:12.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:12 smithi172 ceph-mon[100696]: pgmap v1178: 89 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 1 activating+remapped, 84 active+clean; 1005 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 4574/48123 objects misplaced (9.505%); 5.3 MiB/s, 90 objects/s recovering 2024-09-13T15:04:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2462270440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:12.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:12.570 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:12.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:12.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:12.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:12.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:12.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:12.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:12.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:13.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:13.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:13.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:13.234 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:13.252 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:13.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:13.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:13.383 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:13.383 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:13.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:13.508 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:13.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:13.637 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:13.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:13.766 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:13.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:13.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:13.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:14.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:14.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:14.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:14.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:14.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:14.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:14.383 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:14 smithi154 ceph-mon[102314]: pgmap v1179: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 84 active+clean; 961 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 526 op/s; 4342/46011 objects misplaced (9.437%); 4.5 MiB/s, 77 objects/s recovering 2024-09-13T15:04:14.383 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:14 smithi154 ceph-mon[107038]: pgmap v1179: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 84 active+clean; 961 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 526 op/s; 4342/46011 objects misplaced (9.437%); 4.5 MiB/s, 77 objects/s recovering 2024-09-13T15:04:14.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:14 smithi172 ceph-mon[100696]: pgmap v1179: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 84 active+clean; 961 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 526 op/s; 4342/46011 objects misplaced (9.437%); 4.5 MiB/s, 77 objects/s recovering 2024-09-13T15:04:14.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:14.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:14.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:04:14.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:14.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:14.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:14.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:14.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:14.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:14.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:14.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:15.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:15.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:15.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:15.188 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:15.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:15.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:15.417 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:15.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:15.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:15.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:15.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:15.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:15.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:15.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:15.918 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:15.940 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:15.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:16.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:16.064 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:16.103 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:04:16.103 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:04:16.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:16.190 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:16.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:16.320 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:16.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:16.450 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:16.464 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:16 smithi154 ceph-mon[107038]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T15:04:16.465 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:16 smithi154 ceph-mon[107038]: pgmap v1181: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 84 active+clean; 896 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 527 op/s; 4004/42876 objects misplaced (9.339%); 4.0 MiB/s, 67 objects/s recovering 2024-09-13T15:04:16.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:16 smithi154 ceph-mon[102314]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T15:04:16.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:16 smithi154 ceph-mon[102314]: pgmap v1181: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 84 active+clean; 896 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 527 op/s; 4004/42876 objects misplaced (9.339%); 4.0 MiB/s, 67 objects/s recovering 2024-09-13T15:04:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:16 smithi172 ceph-mon[100696]: osdmap e363: 8 total, 8 up, 8 in 2024-09-13T15:04:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:16 smithi172 ceph-mon[100696]: pgmap v1181: 89 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_wait, 84 active+clean; 896 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 527 op/s; 4004/42876 objects misplaced (9.339%); 4.0 MiB/s, 67 objects/s recovering 2024-09-13T15:04:16.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:16.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:16.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:16.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:16.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:16.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:16.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:16.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:17.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:17.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:17.169 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:17.270 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:17 smithi154 ceph-mon[107038]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T15:04:17.270 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:17 smithi154 ceph-mon[102314]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T15:04:17.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:17.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:17.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:17.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:17 smithi172 ceph-mon[100696]: osdmap e364: 8 total, 8 up, 8 in 2024-09-13T15:04:17.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:17.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:17.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:17.728 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:17.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:04:17.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:17.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:17.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:17.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:18.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:18.059 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:04:18.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:18.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:18.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:18.286 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:18.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:18.426 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:18.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2940307954' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:18.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:18 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:18.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:18 smithi154 ceph-mon[102314]: pgmap v1183: 89 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 84 active+clean; 758 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 2734/36264 objects misplaced (7.539%); 10 MiB/s, 163 objects/s recovering 2024-09-13T15:04:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2940307954' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:18 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:18.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:18 smithi154 ceph-mon[107038]: pgmap v1183: 89 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 84 active+clean; 758 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 2734/36264 objects misplaced (7.539%); 10 MiB/s, 163 objects/s recovering 2024-09-13T15:04:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2940307954' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:18 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:04:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:18 smithi172 ceph-mon[100696]: pgmap v1183: 89 pgs: 1 peering, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 84 active+clean; 758 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 2734/36264 objects misplaced (7.539%); 10 MiB/s, 163 objects/s recovering 2024-09-13T15:04:18.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:18.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:18.658 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:04:18.663 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:04:18.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:18.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:18.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:18.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:18.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:18.965 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:18.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:19.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:19.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:19.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:19.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.248 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:19.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:19 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:19.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:19 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:19.377 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:19 smithi154 ceph-mon[102314]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T15:04:19.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:19 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:19.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:19 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:19.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:19 smithi154 ceph-mon[107038]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T15:04:19.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.485 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:19 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:19 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:04:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:19 smithi172 ceph-mon[100696]: osdmap e365: 8 total, 8 up, 8 in 2024-09-13T15:04:19.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:19.773 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1184,"stamp":"2024-09-13T15:04:18.050999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":795088893,"num_objects":12088,"num_object_clones":0,"num_object_copies":36264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2734,"num_objects_unfound":0,"num_objects_dirty":12088,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":221137,"num_write_kb":7441346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1520,"num_bytes_recovered":87390433,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130796,"ondisk_log_size":130796,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":272,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5892784,"kb_used_data":2507684,"kb_used_omap":40,"kb_used_meta":3384919,"kb_avail":744068432,"statfs":{"total":767960285184,"available":761926074368,"internally_reserved":0,"allocated":2567868416,"data_stored":2552229978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":3466157172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-259063808,"num_objects":-3953,"num_object_clones":0,"num_object_copies":-11859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1840,"num_objects_unfound":0,"num_objects_dirty":-3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3953,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":771,"num_bytes_recovered":50528256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.003073"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":55,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:08.045395+0000","last_change":"2024-09-13T15:01:14.837554+0000","last_active":"2024-09-13T15:04:08.045395+0000","last_peered":"2024-09-13T15:04:08.045395+0000","last_clean":"2024-09-13T15:04:08.045395+0000","last_became_active":"2024-09-13T15:01:14.837236+0000","last_became_peered":"2024-09-13T15:01:14.837236+0000","last_unstale":"2024-09-13T15:04:08.045395+0000","last_undegraded":"2024-09-13T15:04:08.045395+0000","last_fullsized":"2024-09-13T15:04:08.045395+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:49:18.510952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":105,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.028380+0000","last_change":"2024-09-13T15:00:15.944713+0000","last_active":"2024-09-13T15:04:07.028380+0000","last_peered":"2024-09-13T15:04:07.028380+0000","last_clean":"2024-09-13T15:04:07.028380+0000","last_became_active":"2024-09-13T15:00:15.944353+0000","last_became_peered":"2024-09-13T15:00:15.944353+0000","last_unstale":"2024-09-13T15:04:07.028380+0000","last_undegraded":"2024-09-13T15:04:07.028380+0000","last_fullsized":"2024-09-13T15:04:07.028380+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:22.078466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":211,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026605+0000","last_change":"2024-09-13T14:57:56.258528+0000","last_active":"2024-09-13T15:04:07.026605+0000","last_peered":"2024-09-13T15:04:07.026605+0000","last_clean":"2024-09-13T15:04:07.026605+0000","last_became_active":"2024-09-13T14:57:56.258116+0000","last_became_peered":"2024-09-13T14:57:56.258116+0000","last_unstale":"2024-09-13T15:04:07.026605+0000","last_undegraded":"2024-09-13T15:04:07.026605+0000","last_fullsized":"2024-09-13T15:04:07.026605+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:13.267817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":548,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.024638+0000","last_change":"2024-09-13T14:59:42.555312+0000","last_active":"2024-09-13T15:04:15.024638+0000","last_peered":"2024-09-13T15:04:15.024638+0000","last_clean":"2024-09-13T15:04:15.024638+0000","last_became_active":"2024-09-13T14:59:42.554397+0000","last_became_peered":"2024-09-13T14:59:42.554397+0000","last_unstale":"2024-09-13T15:04:15.024638+0000","last_undegraded":"2024-09-13T15:04:15.024638+0000","last_fullsized":"2024-09-13T15:04:15.024638+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:26.418960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":552,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.009672+0000","last_change":"2024-09-13T14:59:42.572046+0000","last_active":"2024-09-13T15:04:15.009672+0000","last_peered":"2024-09-13T15:04:15.009672+0000","last_clean":"2024-09-13T15:04:15.009672+0000","last_became_active":"2024-09-13T14:59:42.565943+0000","last_became_peered":"2024-09-13T14:59:42.565943+0000","last_unstale":"2024-09-13T15:04:15.009672+0000","last_undegraded":"2024-09-13T15:04:15.009672+0000","last_fullsized":"2024-09-13T15:04:15.009672+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:17:46.240539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":117,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.028248+0000","last_change":"2024-09-13T15:04:05.013988+0000","last_active":"2024-09-13T15:04:07.028248+0000","last_peered":"2024-09-13T15:04:07.028248+0000","last_clean":"2024-09-13T15:04:07.028248+0000","last_became_active":"2024-09-13T15:04:05.013696+0000","last_became_peered":"2024-09-13T15:04:05.013696+0000","last_unstale":"2024-09-13T15:04:07.028248+0000","last_undegraded":"2024-09-13T15:04:07.028248+0000","last_fullsized":"2024-09-13T15:04:07.028248+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:45.998297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":53,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008500+0000","last_change":"2024-09-13T15:01:17.867842+0000","last_active":"2024-09-13T15:04:15.008500+0000","last_peered":"2024-09-13T15:04:15.008500+0000","last_clean":"2024-09-13T15:04:15.008500+0000","last_became_active":"2024-09-13T15:01:17.867444+0000","last_became_peered":"2024-09-13T15:01:17.867444+0000","last_unstale":"2024-09-13T15:04:15.008500+0000","last_undegraded":"2024-09-13T15:04:15.008500+0000","last_fullsized":"2024-09-13T15:04:15.008500+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:14.910223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":172,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.007652+0000","last_change":"2024-09-13T15:04:05.021292+0000","last_active":"2024-09-13T15:04:15.007652+0000","last_peered":"2024-09-13T15:04:15.007652+0000","last_clean":"2024-09-13T15:04:15.007652+0000","last_became_active":"2024-09-13T15:04:05.011146+0000","last_became_peered":"2024-09-13T15:04:05.011146+0000","last_unstale":"2024-09-13T15:04:15.007652+0000","last_undegraded":"2024-09-13T15:04:15.007652+0000","last_fullsized":"2024-09-13T15:04:15.007652+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:00.137694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"362'13706","reported_seq":13747,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:13.611800+0000","last_change":"2024-09-13T15:00:24.023321+0000","last_active":"2024-09-13T15:04:13.611800+0000","last_peered":"2024-09-13T15:04:13.611800+0000","last_clean":"2024-09-13T15:04:13.611800+0000","last_became_active":"2024-09-13T15:00:24.022952+0000","last_became_peered":"2024-09-13T15:00:24.022952+0000","last_unstale":"2024-09-13T15:04:13.611800+0000","last_undegraded":"2024-09-13T15:04:13.611800+0000","last_fullsized":"2024-09-13T15:04:13.611800+0000","mapping_epoch":331,"log_start":"339'3900","ondisk_log_start":"339'3900","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:05:34.183228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55050240,"num_objects":840,"num_object_clones":0,"num_object_copies":2520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13706,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":119,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.028482+0000","last_change":"2024-09-13T15:00:15.946088+0000","last_active":"2024-09-13T15:04:07.028482+0000","last_peered":"2024-09-13T15:04:07.028482+0000","last_clean":"2024-09-13T15:04:07.028482+0000","last_became_active":"2024-09-13T15:00:15.945399+0000","last_became_peered":"2024-09-13T15:00:15.945399+0000","last_unstale":"2024-09-13T15:04:07.028482+0000","last_undegraded":"2024-09-13T15:04:07.028482+0000","last_fullsized":"2024-09-13T15:04:07.028482+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:20.043020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":257,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026362+0000","last_change":"2024-09-13T15:04:05.016419+0000","last_active":"2024-09-13T15:04:07.026362+0000","last_peered":"2024-09-13T15:04:07.026362+0000","last_clean":"2024-09-13T15:04:07.026362+0000","last_became_active":"2024-09-13T15:04:05.016123+0000","last_became_peered":"2024-09-13T15:04:05.016123+0000","last_unstale":"2024-09-13T15:04:07.026362+0000","last_undegraded":"2024-09-13T15:04:07.026362+0000","last_fullsized":"2024-09-13T15:04:07.026362+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:30.373414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"362'13641","reported_seq":13695,"reported_epoch":362,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:04:13.607591+0000","last_change":"2024-09-13T15:04:06.438401+0000","last_active":"2024-09-13T15:04:13.607591+0000","last_peered":"2024-09-13T15:04:13.607591+0000","last_clean":"2024-09-13T15:04:04.730182+0000","last_became_active":"2024-09-13T15:04:06.355346+0000","last_became_peered":"2024-09-13T15:04:06.355346+0000","last_unstale":"2024-09-13T15:04:13.607591+0000","last_undegraded":"2024-09-13T15:04:13.607591+0000","last_fullsized":"2024-09-13T15:04:13.607591+0000","mapping_epoch":359,"log_start":"345'5800","ondisk_log_start":"345'5800","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7841,"log_dups_size":0,"ondisk_log_size":7841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:30.496674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54722560,"num_objects":835,"num_object_clones":0,"num_object_copies":2505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":835,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13641,"num_write_kb":463232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":835}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":92,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.028764+0000","last_change":"2024-09-13T14:57:56.273194+0000","last_active":"2024-09-13T15:04:07.028764+0000","last_peered":"2024-09-13T15:04:07.028764+0000","last_clean":"2024-09-13T15:04:07.028764+0000","last_became_active":"2024-09-13T14:57:56.259881+0000","last_became_peered":"2024-09-13T14:57:56.259881+0000","last_unstale":"2024-09-13T15:04:07.028764+0000","last_undegraded":"2024-09-13T15:04:07.028764+0000","last_fullsized":"2024-09-13T15:04:07.028764+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:28.683153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":425,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036102+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:04:07.036102+0000","last_peered":"2024-09-13T15:04:07.036102+0000","last_clean":"2024-09-13T15:04:07.036102+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:04:07.036102+0000","last_undegraded":"2024-09-13T15:04:07.036102+0000","last_fullsized":"2024-09-13T15:04:07.036102+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"363'13831","reported_seq":13912,"reported_epoch":363,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:04:15.770103+0000","last_change":"2024-09-13T15:04:06.432975+0000","last_active":"2024-09-13T15:04:15.770103+0000","last_peered":"2024-09-13T15:04:15.770103+0000","last_clean":"2024-09-13T15:04:04.756301+0000","last_became_active":"2024-09-13T15:04:06.379326+0000","last_became_peered":"2024-09-13T15:04:06.379326+0000","last_unstale":"2024-09-13T15:04:15.770103+0000","last_undegraded":"2024-09-13T15:04:15.770103+0000","last_fullsized":"2024-09-13T15:04:15.770103+0000","mapping_epoch":359,"log_start":"346'6200","ondisk_log_start":"346'6200","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43188224,"num_objects":659,"num_object_clones":0,"num_object_copies":1977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":659,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13831,"num_write_kb":463680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":118,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.031736+0000","last_change":"2024-09-13T15:04:05.028892+0000","last_active":"2024-09-13T15:04:07.031736+0000","last_peered":"2024-09-13T15:04:07.031736+0000","last_clean":"2024-09-13T15:04:07.031736+0000","last_became_active":"2024-09-13T15:04:05.011144+0000","last_became_peered":"2024-09-13T15:04:05.011144+0000","last_unstale":"2024-09-13T15:04:07.031736+0000","last_undegraded":"2024-09-13T15:04:07.031736+0000","last_fullsized":"2024-09-13T15:04:07.031736+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:23.974259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":288,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.010373+0000","last_change":"2024-09-13T15:04:05.013040+0000","last_active":"2024-09-13T15:04:15.010373+0000","last_peered":"2024-09-13T15:04:15.010373+0000","last_clean":"2024-09-13T15:04:15.010373+0000","last_became_active":"2024-09-13T15:04:05.012788+0000","last_became_peered":"2024-09-13T15:04:05.012788+0000","last_unstale":"2024-09-13T15:04:15.010373+0000","last_undegraded":"2024-09-13T15:04:15.010373+0000","last_fullsized":"2024-09-13T15:04:15.010373+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:53.556798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"362'13765","reported_seq":15794,"reported_epoch":362,"state":"peering","last_fresh":"2024-09-13T15:04:15.007894+0000","last_change":"2024-09-13T15:04:15.014252+0000","last_active":"2024-09-13T15:04:15.014252+0000","last_peered":"2024-09-13T15:04:15.007894+0000","last_clean":"2024-09-13T15:04:15.007894+0000","last_became_active":"2024-09-13T15:04:06.365826+0000","last_became_peered":"2024-09-13T15:04:06.365826+0000","last_unstale":"2024-09-13T15:04:15.007894+0000","last_undegraded":"2024-09-13T15:04:15.007894+0000","last_fullsized":"2024-09-13T15:04:15.007894+0000","mapping_epoch":363,"log_start":"346'6300","ondisk_log_start":"346'6300","created":331,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51052544,"num_objects":779,"num_object_clones":0,"num_object_copies":2337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13765,"num_write_kb":465408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":63700992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":237,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:08.042309+0000","last_change":"2024-09-13T15:04:05.013208+0000","last_active":"2024-09-13T15:04:08.042309+0000","last_peered":"2024-09-13T15:04:08.042309+0000","last_clean":"2024-09-13T15:04:08.042309+0000","last_became_active":"2024-09-13T15:04:05.012896+0000","last_became_peered":"2024-09-13T15:04:05.012896+0000","last_unstale":"2024-09-13T15:04:08.042309+0000","last_undegraded":"2024-09-13T15:04:08.042309+0000","last_fullsized":"2024-09-13T15:04:08.042309+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:45.434822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":475,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008028+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:04:15.008028+0000","last_peered":"2024-09-13T15:04:15.008028+0000","last_clean":"2024-09-13T15:04:15.008028+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:04:15.008028+0000","last_undegraded":"2024-09-13T15:04:15.008028+0000","last_fullsized":"2024-09-13T15:04:15.008028+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"363'13784","reported_seq":13860,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.106954+0000","last_change":"2024-09-13T15:00:23.973213+0000","last_active":"2024-09-13T15:04:15.106954+0000","last_peered":"2024-09-13T15:04:15.106954+0000","last_clean":"2024-09-13T15:04:15.106954+0000","last_became_active":"2024-09-13T15:00:23.972885+0000","last_became_peered":"2024-09-13T15:00:23.972885+0000","last_unstale":"2024-09-13T15:04:15.106954+0000","last_undegraded":"2024-09-13T15:04:15.106954+0000","last_fullsized":"2024-09-13T15:04:15.106954+0000","mapping_epoch":331,"log_start":"346'6500","ondisk_log_start":"346'6500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:07:22.288214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":704,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13784,"num_write_kb":463616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":601,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.009752+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:04:15.009752+0000","last_peered":"2024-09-13T15:04:15.009752+0000","last_clean":"2024-09-13T15:04:15.009752+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:04:15.009752+0000","last_undegraded":"2024-09-13T15:04:15.009752+0000","last_fullsized":"2024-09-13T15:04:15.009752+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":367,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036308+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:04:07.036308+0000","last_peered":"2024-09-13T15:04:07.036308+0000","last_clean":"2024-09-13T15:04:07.036308+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:04:07.036308+0000","last_undegraded":"2024-09-13T15:04:07.036308+0000","last_fullsized":"2024-09-13T15:04:07.036308+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"363'13959","reported_seq":14035,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.676130+0000","last_change":"2024-09-13T15:00:24.023172+0000","last_active":"2024-09-13T15:04:15.676130+0000","last_peered":"2024-09-13T15:04:15.676130+0000","last_clean":"2024-09-13T15:04:15.676130+0000","last_became_active":"2024-09-13T15:00:24.022574+0000","last_became_peered":"2024-09-13T15:00:24.022574+0000","last_unstale":"2024-09-13T15:04:15.676130+0000","last_undegraded":"2024-09-13T15:04:15.676130+0000","last_fullsized":"2024-09-13T15:04:15.676130+0000","mapping_epoch":331,"log_start":"346'6500","ondisk_log_start":"346'6500","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7459,"log_dups_size":0,"ondisk_log_size":7459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:06.934046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46333952,"num_objects":707,"num_object_clones":0,"num_object_copies":2121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13959,"num_write_kb":469312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":165,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036333+0000","last_change":"2024-09-13T15:04:05.013503+0000","last_active":"2024-09-13T15:04:07.036333+0000","last_peered":"2024-09-13T15:04:07.036333+0000","last_clean":"2024-09-13T15:04:07.036333+0000","last_became_active":"2024-09-13T15:04:05.013204+0000","last_became_peered":"2024-09-13T15:04:05.013204+0000","last_unstale":"2024-09-13T15:04:07.036333+0000","last_undegraded":"2024-09-13T15:04:07.036333+0000","last_fullsized":"2024-09-13T15:04:07.036333+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:17.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":361,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036497+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:04:07.036497+0000","last_peered":"2024-09-13T15:04:07.036497+0000","last_clean":"2024-09-13T15:04:07.036497+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:04:07.036497+0000","last_undegraded":"2024-09-13T15:04:07.036497+0000","last_fullsized":"2024-09-13T15:04:07.036497+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"362'13761","reported_seq":13802,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:13.599089+0000","last_change":"2024-09-13T15:00:23.972644+0000","last_active":"2024-09-13T15:04:13.599089+0000","last_peered":"2024-09-13T15:04:13.599089+0000","last_clean":"2024-09-13T15:04:13.599089+0000","last_became_active":"2024-09-13T15:00:23.972370+0000","last_became_peered":"2024-09-13T15:00:23.972370+0000","last_unstale":"2024-09-13T15:04:13.599089+0000","last_undegraded":"2024-09-13T15:04:13.599089+0000","last_fullsized":"2024-09-13T15:04:13.599089+0000","mapping_epoch":331,"log_start":"345'5900","ondisk_log_start":"345'5900","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:55.756118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53936128,"num_objects":823,"num_object_clones":0,"num_object_copies":2469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13761,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":368,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036137+0000","last_change":"2024-09-13T15:00:15.945718+0000","last_active":"2024-09-13T15:04:07.036137+0000","last_peered":"2024-09-13T15:04:07.036137+0000","last_clean":"2024-09-13T15:04:07.036137+0000","last_became_active":"2024-09-13T15:00:15.944400+0000","last_became_peered":"2024-09-13T15:00:15.944400+0000","last_unstale":"2024-09-13T15:04:07.036137+0000","last_undegraded":"2024-09-13T15:04:07.036137+0000","last_fullsized":"2024-09-13T15:04:07.036137+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:23:47.992626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":394,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036769+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:04:07.036769+0000","last_peered":"2024-09-13T15:04:07.036769+0000","last_clean":"2024-09-13T15:04:07.036769+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:04:07.036769+0000","last_undegraded":"2024-09-13T15:04:07.036769+0000","last_fullsized":"2024-09-13T15:04:07.036769+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"363'13722","reported_seq":14405,"reported_epoch":363,"state":"active+remapped+backfilling","last_fresh":"2024-09-13T15:04:15.821829+0000","last_change":"2024-09-13T15:04:14.714199+0000","last_active":"2024-09-13T15:04:15.821829+0000","last_peered":"2024-09-13T15:04:15.821829+0000","last_clean":"2024-09-13T15:04:04.739588+0000","last_became_active":"2024-09-13T15:04:06.378443+0000","last_became_peered":"2024-09-13T15:04:06.378443+0000","last_unstale":"2024-09-13T15:04:15.821829+0000","last_undegraded":"2024-09-13T15:04:15.821829+0000","last_fullsized":"2024-09-13T15:04:15.821829+0000","mapping_epoch":359,"log_start":"346'6300","ondisk_log_start":"346'6300","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7422,"log_dups_size":0,"ondisk_log_size":7422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47185920,"num_objects":720,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":429,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13722,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":19529728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":720}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.009086+0000","last_change":"2024-09-13T15:01:17.859549+0000","last_active":"2024-09-13T15:04:15.009086+0000","last_peered":"2024-09-13T15:04:15.009086+0000","last_clean":"2024-09-13T15:04:15.009086+0000","last_became_active":"2024-09-13T15:01:17.858796+0000","last_became_peered":"2024-09-13T15:01:17.858796+0000","last_unstale":"2024-09-13T15:04:15.009086+0000","last_undegraded":"2024-09-13T15:04:15.009086+0000","last_fullsized":"2024-09-13T15:04:15.009086+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:32.898026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":344,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:08.049301+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:04:08.049301+0000","last_peered":"2024-09-13T15:04:08.049301+0000","last_clean":"2024-09-13T15:04:08.049301+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:04:08.049301+0000","last_undegraded":"2024-09-13T15:04:08.049301+0000","last_fullsized":"2024-09-13T15:04:08.049301+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"362'13613","reported_seq":13664,"reported_epoch":362,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-13T15:04:13.624356+0000","last_change":"2024-09-13T15:04:06.430388+0000","last_active":"2024-09-13T15:04:13.624356+0000","last_peered":"2024-09-13T15:04:13.624356+0000","last_clean":"2024-09-13T15:04:04.641039+0000","last_became_active":"2024-09-13T15:04:06.378532+0000","last_became_peered":"2024-09-13T15:04:06.378532+0000","last_unstale":"2024-09-13T15:04:13.624356+0000","last_undegraded":"2024-09-13T15:04:13.624356+0000","last_fullsized":"2024-09-13T15:04:13.624356+0000","mapping_epoch":359,"log_start":"339'3800","ondisk_log_start":"339'3800","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:19.064012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":811,"num_objects_unfound":0,"num_objects_dirty":811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13613,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,0,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":811}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":774,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:08.059305+0000","last_change":"2024-09-13T15:04:08.059305+0000","last_active":"2024-09-13T15:04:08.059305+0000","last_peered":"2024-09-13T15:04:08.059305+0000","last_clean":"2024-09-13T15:04:08.059305+0000","last_became_active":"2024-09-13T15:04:08.058902+0000","last_became_peered":"2024-09-13T15:04:08.058902+0000","last_unstale":"2024-09-13T15:04:08.059305+0000","last_undegraded":"2024-09-13T15:04:08.059305+0000","last_fullsized":"2024-09-13T15:04:08.059305+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.674125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":983,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026608+0000","last_change":"2024-09-13T15:00:15.945425+0000","last_active":"2024-09-13T15:04:07.026608+0000","last_peered":"2024-09-13T15:04:07.026608+0000","last_clean":"2024-09-13T15:04:07.026608+0000","last_became_active":"2024-09-13T15:00:15.944987+0000","last_became_peered":"2024-09-13T15:00:15.944987+0000","last_unstale":"2024-09-13T15:04:07.026608+0000","last_undegraded":"2024-09-13T15:04:07.026608+0000","last_fullsized":"2024-09-13T15:04:07.026608+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:58.158525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":562,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008224+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:04:15.008224+0000","last_peered":"2024-09-13T15:04:15.008224+0000","last_clean":"2024-09-13T15:04:15.008224+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:04:15.008224+0000","last_undegraded":"2024-09-13T15:04:15.008224+0000","last_fullsized":"2024-09-13T15:04:15.008224+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":195,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036431+0000","last_change":"2024-09-13T15:04:05.012948+0000","last_active":"2024-09-13T15:04:07.036431+0000","last_peered":"2024-09-13T15:04:07.036431+0000","last_clean":"2024-09-13T15:04:07.036431+0000","last_became_active":"2024-09-13T15:04:05.012616+0000","last_became_peered":"2024-09-13T15:04:05.012616+0000","last_unstale":"2024-09-13T15:04:07.036431+0000","last_undegraded":"2024-09-13T15:04:07.036431+0000","last_fullsized":"2024-09-13T15:04:07.036431+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:25:47.702692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"363'13780","reported_seq":13856,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.773783+0000","last_change":"2024-09-13T15:00:24.024002+0000","last_active":"2024-09-13T15:04:15.773783+0000","last_peered":"2024-09-13T15:04:15.773783+0000","last_clean":"2024-09-13T15:04:15.773783+0000","last_became_active":"2024-09-13T15:00:24.023708+0000","last_became_peered":"2024-09-13T15:00:24.023708+0000","last_unstale":"2024-09-13T15:04:15.773783+0000","last_undegraded":"2024-09-13T15:04:15.773783+0000","last_fullsized":"2024-09-13T15:04:15.773783+0000","mapping_epoch":331,"log_start":"346'6300","ondisk_log_start":"346'6300","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:06:46.507079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43778048,"num_objects":668,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13780,"num_write_kb":462336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":122,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.028527+0000","last_change":"2024-09-13T14:59:42.568409+0000","last_active":"2024-09-13T15:04:07.028527+0000","last_peered":"2024-09-13T15:04:07.028527+0000","last_clean":"2024-09-13T15:04:07.028527+0000","last_became_active":"2024-09-13T14:59:42.567972+0000","last_became_peered":"2024-09-13T14:59:42.567972+0000","last_unstale":"2024-09-13T15:04:07.028527+0000","last_undegraded":"2024-09-13T15:04:07.028527+0000","last_fullsized":"2024-09-13T15:04:07.028527+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:21.721692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":549,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.013037+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:04:15.013037+0000","last_peered":"2024-09-13T15:04:15.013037+0000","last_clean":"2024-09-13T15:04:15.013037+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:04:15.013037+0000","last_undegraded":"2024-09-13T15:04:15.013037+0000","last_fullsized":"2024-09-13T15:04:15.013037+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":544,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.009027+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:04:15.009027+0000","last_peered":"2024-09-13T15:04:15.009027+0000","last_clean":"2024-09-13T15:04:15.009027+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:04:15.009027+0000","last_undegraded":"2024-09-13T15:04:15.009027+0000","last_fullsized":"2024-09-13T15:04:15.009027+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"363'13910","reported_seq":13986,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.470334+0000","last_change":"2024-09-13T15:00:23.973296+0000","last_active":"2024-09-13T15:04:15.470334+0000","last_peered":"2024-09-13T15:04:15.470334+0000","last_clean":"2024-09-13T15:04:15.470334+0000","last_became_active":"2024-09-13T15:00:23.972661+0000","last_became_peered":"2024-09-13T15:00:23.972661+0000","last_unstale":"2024-09-13T15:04:15.470334+0000","last_undegraded":"2024-09-13T15:04:15.470334+0000","last_fullsized":"2024-09-13T15:04:15.470334+0000","mapping_epoch":331,"log_start":"346'6600","ondisk_log_start":"346'6600","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:58.633504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45744128,"num_objects":698,"num_object_clones":0,"num_object_copies":2094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13910,"num_write_kb":467456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":981,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:08.042904+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:04:08.042904+0000","last_peered":"2024-09-13T15:04:08.042904+0000","last_clean":"2024-09-13T15:04:08.042904+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:04:08.042904+0000","last_undegraded":"2024-09-13T15:04:08.042904+0000","last_fullsized":"2024-09-13T15:04:08.042904+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":612,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.018258+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:04:15.018258+0000","last_peered":"2024-09-13T15:04:15.018258+0000","last_clean":"2024-09-13T15:04:15.018258+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:04:15.018258+0000","last_undegraded":"2024-09-13T15:04:15.018258+0000","last_fullsized":"2024-09-13T15:04:15.018258+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":350,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.014553+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:04:15.014553+0000","last_peered":"2024-09-13T15:04:15.014553+0000","last_clean":"2024-09-13T15:04:15.014553+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:04:15.014553+0000","last_undegraded":"2024-09-13T15:04:15.014553+0000","last_fullsized":"2024-09-13T15:04:15.014553+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"362'13621","reported_seq":13662,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:13.642430+0000","last_change":"2024-09-13T15:00:24.022623+0000","last_active":"2024-09-13T15:04:13.642430+0000","last_peered":"2024-09-13T15:04:13.642430+0000","last_clean":"2024-09-13T15:04:13.642430+0000","last_became_active":"2024-09-13T15:00:24.022333+0000","last_became_peered":"2024-09-13T15:00:24.022333+0000","last_unstale":"2024-09-13T15:04:13.642430+0000","last_undegraded":"2024-09-13T15:04:13.642430+0000","last_fullsized":"2024-09-13T15:04:13.642430+0000","mapping_epoch":331,"log_start":"339'3800","ondisk_log_start":"339'3800","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:24.771552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53805056,"num_objects":821,"num_object_clones":0,"num_object_copies":2463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13621,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":286,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026604+0000","last_change":"2024-09-13T15:04:05.035515+0000","last_active":"2024-09-13T15:04:07.026604+0000","last_peered":"2024-09-13T15:04:07.026604+0000","last_clean":"2024-09-13T15:04:07.026604+0000","last_became_active":"2024-09-13T15:04:05.017008+0000","last_became_peered":"2024-09-13T15:04:05.017008+0000","last_unstale":"2024-09-13T15:04:07.026604+0000","last_undegraded":"2024-09-13T15:04:07.026604+0000","last_fullsized":"2024-09-13T15:04:07.026604+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:44.511395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":387,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036654+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:04:07.036654+0000","last_peered":"2024-09-13T15:04:07.036654+0000","last_clean":"2024-09-13T15:04:07.036654+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:04:07.036654+0000","last_undegraded":"2024-09-13T15:04:07.036654+0000","last_fullsized":"2024-09-13T15:04:07.036654+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":146,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026294+0000","last_change":"2024-09-13T15:00:14.949848+0000","last_active":"2024-09-13T15:04:07.026294+0000","last_peered":"2024-09-13T15:04:07.026294+0000","last_clean":"2024-09-13T15:04:07.026294+0000","last_became_active":"2024-09-13T15:00:14.949644+0000","last_became_peered":"2024-09-13T15:00:14.949644+0000","last_unstale":"2024-09-13T15:04:07.026294+0000","last_undegraded":"2024-09-13T15:04:07.026294+0000","last_fullsized":"2024-09-13T15:04:07.026294+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:17.762542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"363'13977","reported_seq":14053,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.694750+0000","last_change":"2024-09-13T15:00:24.024125+0000","last_active":"2024-09-13T15:04:15.694750+0000","last_peered":"2024-09-13T15:04:15.694750+0000","last_clean":"2024-09-13T15:04:15.694750+0000","last_became_active":"2024-09-13T15:00:24.023801+0000","last_became_peered":"2024-09-13T15:00:24.023801+0000","last_unstale":"2024-09-13T15:04:15.694750+0000","last_undegraded":"2024-09-13T15:04:15.694750+0000","last_fullsized":"2024-09-13T15:04:15.694750+0000","mapping_epoch":331,"log_start":"346'6300","ondisk_log_start":"346'6300","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7677,"log_dups_size":0,"ondisk_log_size":7677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:04:26.418853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46727168,"num_objects":713,"num_object_clones":0,"num_object_copies":2139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13977,"num_write_kb":470080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":236,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026534+0000","last_change":"2024-09-13T15:00:14.948015+0000","last_active":"2024-09-13T15:04:07.026534+0000","last_peered":"2024-09-13T15:04:07.026534+0000","last_clean":"2024-09-13T15:04:07.026534+0000","last_became_active":"2024-09-13T15:00:14.947799+0000","last_became_peered":"2024-09-13T15:00:14.947799+0000","last_unstale":"2024-09-13T15:04:07.026534+0000","last_undegraded":"2024-09-13T15:04:07.026534+0000","last_fullsized":"2024-09-13T15:04:07.026534+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:37.576480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1192,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.010683+0000","last_change":"2024-09-13T14:59:42.570065+0000","last_active":"2024-09-13T15:04:15.010683+0000","last_peered":"2024-09-13T15:04:15.010683+0000","last_clean":"2024-09-13T15:04:15.010683+0000","last_became_active":"2024-09-13T14:59:42.569452+0000","last_became_peered":"2024-09-13T14:59:42.569452+0000","last_unstale":"2024-09-13T15:04:15.010683+0000","last_undegraded":"2024-09-13T15:04:15.010683+0000","last_fullsized":"2024-09-13T15:04:15.010683+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:30:05.305260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":194,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.024178+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:04:15.024178+0000","last_peered":"2024-09-13T15:04:15.024178+0000","last_clean":"2024-09-13T15:04:15.024178+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:04:15.024178+0000","last_undegraded":"2024-09-13T15:04:15.024178+0000","last_fullsized":"2024-09-13T15:04:15.024178+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"363'13588","reported_seq":13664,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.279571+0000","last_change":"2024-09-13T15:00:24.024834+0000","last_active":"2024-09-13T15:04:15.279571+0000","last_peered":"2024-09-13T15:04:15.279571+0000","last_clean":"2024-09-13T15:04:15.279571+0000","last_became_active":"2024-09-13T15:00:24.024491+0000","last_became_peered":"2024-09-13T15:00:24.024491+0000","last_unstale":"2024-09-13T15:04:15.279571+0000","last_undegraded":"2024-09-13T15:04:15.279571+0000","last_fullsized":"2024-09-13T15:04:15.279571+0000","mapping_epoch":331,"log_start":"346'6300","ondisk_log_start":"346'6300","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7288,"log_dups_size":0,"ondisk_log_size":7288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:35.133306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49020928,"num_objects":748,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13588,"num_write_kb":458752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":604,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.010589+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:04:15.010589+0000","last_peered":"2024-09-13T15:04:15.010589+0000","last_clean":"2024-09-13T15:04:15.010589+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:04:15.010589+0000","last_undegraded":"2024-09-13T15:04:15.010589+0000","last_fullsized":"2024-09-13T15:04:15.010589+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":121,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.031813+0000","last_change":"2024-09-13T14:59:42.576338+0000","last_active":"2024-09-13T15:04:07.031813+0000","last_peered":"2024-09-13T15:04:07.031813+0000","last_clean":"2024-09-13T15:04:07.031813+0000","last_became_active":"2024-09-13T14:59:42.575875+0000","last_became_peered":"2024-09-13T14:59:42.575875+0000","last_unstale":"2024-09-13T15:04:07.031813+0000","last_undegraded":"2024-09-13T15:04:07.031813+0000","last_fullsized":"2024-09-13T15:04:07.031813+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:51.037045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":539,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008393+0000","last_change":"2024-09-13T14:59:42.572653+0000","last_active":"2024-09-13T15:04:15.008393+0000","last_peered":"2024-09-13T15:04:15.008393+0000","last_clean":"2024-09-13T15:04:15.008393+0000","last_became_active":"2024-09-13T14:59:42.572326+0000","last_became_peered":"2024-09-13T14:59:42.572326+0000","last_unstale":"2024-09-13T15:04:15.008393+0000","last_undegraded":"2024-09-13T15:04:15.008393+0000","last_fullsized":"2024-09-13T15:04:15.008393+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:10:02.284316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"363'13744","reported_seq":13820,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.570990+0000","last_change":"2024-09-13T15:00:24.023353+0000","last_active":"2024-09-13T15:04:15.570990+0000","last_peered":"2024-09-13T15:04:15.570990+0000","last_clean":"2024-09-13T15:04:15.570990+0000","last_became_active":"2024-09-13T15:00:24.023074+0000","last_became_peered":"2024-09-13T15:00:24.023074+0000","last_unstale":"2024-09-13T15:04:15.570990+0000","last_undegraded":"2024-09-13T15:04:15.570990+0000","last_fullsized":"2024-09-13T15:04:15.570990+0000","mapping_epoch":331,"log_start":"345'6100","ondisk_log_start":"345'6100","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:50.987256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13744,"num_write_kb":463104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":634,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.011490+0000","last_change":"2024-09-13T14:59:42.561566+0000","last_active":"2024-09-13T15:04:15.011490+0000","last_peered":"2024-09-13T15:04:15.011490+0000","last_clean":"2024-09-13T15:04:15.011490+0000","last_became_active":"2024-09-13T14:59:42.561224+0000","last_became_peered":"2024-09-13T14:59:42.561224+0000","last_unstale":"2024-09-13T15:04:15.011490+0000","last_undegraded":"2024-09-13T15:04:15.011490+0000","last_fullsized":"2024-09-13T15:04:15.011490+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:31.918624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1090,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.010914+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:04:15.010914+0000","last_peered":"2024-09-13T15:04:15.010914+0000","last_clean":"2024-09-13T15:04:15.010914+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:04:15.010914+0000","last_undegraded":"2024-09-13T15:04:15.010914+0000","last_fullsized":"2024-09-13T15:04:15.010914+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":346,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008746+0000","last_change":"2024-09-13T14:59:42.561395+0000","last_active":"2024-09-13T15:04:15.008746+0000","last_peered":"2024-09-13T15:04:15.008746+0000","last_clean":"2024-09-13T15:04:15.008746+0000","last_became_active":"2024-09-13T14:59:42.561100+0000","last_became_peered":"2024-09-13T14:59:42.561100+0000","last_unstale":"2024-09-13T15:04:15.008746+0000","last_undegraded":"2024-09-13T15:04:15.008746+0000","last_fullsized":"2024-09-13T15:04:15.008746+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:31.681550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"362'13612","reported_seq":13654,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:13.594605+0000","last_change":"2024-09-13T15:00:24.023000+0000","last_active":"2024-09-13T15:04:13.594605+0000","last_peered":"2024-09-13T15:04:13.594605+0000","last_clean":"2024-09-13T15:04:13.594605+0000","last_became_active":"2024-09-13T15:00:24.022727+0000","last_became_peered":"2024-09-13T15:00:24.022727+0000","last_unstale":"2024-09-13T15:04:13.594605+0000","last_undegraded":"2024-09-13T15:04:13.594605+0000","last_fullsized":"2024-09-13T15:04:13.594605+0000","mapping_epoch":331,"log_start":"339'3800","ondisk_log_start":"339'3800","created":331,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:00:22.954744+0000","last_clean_scrub_stamp":"2024-09-13T15:00:22.954744+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:54:35.731087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756504,"num_objects":806,"num_object_clones":0,"num_object_copies":2418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13612,"num_write_kb":461313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":480,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.010201+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:04:15.010201+0000","last_peered":"2024-09-13T15:04:15.010201+0000","last_clean":"2024-09-13T15:04:15.010201+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:04:15.010201+0000","last_undegraded":"2024-09-13T15:04:15.010201+0000","last_fullsized":"2024-09-13T15:04:15.010201+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":997,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:08.055398+0000","last_change":"2024-09-13T15:04:08.055398+0000","last_active":"2024-09-13T15:04:08.055398+0000","last_peered":"2024-09-13T15:04:08.055398+0000","last_clean":"2024-09-13T15:04:08.055398+0000","last_became_active":"2024-09-13T15:04:08.054838+0000","last_became_peered":"2024-09-13T15:04:08.054838+0000","last_unstale":"2024-09-13T15:04:08.055398+0000","last_undegraded":"2024-09-13T15:04:08.055398+0000","last_fullsized":"2024-09-13T15:04:08.055398+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:21.423390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":383,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036028+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:04:07.036028+0000","last_peered":"2024-09-13T15:04:07.036028+0000","last_clean":"2024-09-13T15:04:07.036028+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:04:07.036028+0000","last_undegraded":"2024-09-13T15:04:07.036028+0000","last_fullsized":"2024-09-13T15:04:07.036028+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":367,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036563+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:04:07.036563+0000","last_peered":"2024-09-13T15:04:07.036563+0000","last_clean":"2024-09-13T15:04:07.036563+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:04:07.036563+0000","last_undegraded":"2024-09-13T15:04:07.036563+0000","last_fullsized":"2024-09-13T15:04:07.036563+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":461,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036412+0000","last_change":"2024-09-13T15:04:05.025733+0000","last_active":"2024-09-13T15:04:07.036412+0000","last_peered":"2024-09-13T15:04:07.036412+0000","last_clean":"2024-09-13T15:04:07.036412+0000","last_became_active":"2024-09-13T15:04:05.014008+0000","last_became_peered":"2024-09-13T15:04:05.014008+0000","last_unstale":"2024-09-13T15:04:07.036412+0000","last_undegraded":"2024-09-13T15:04:07.036412+0000","last_fullsized":"2024-09-13T15:04:07.036412+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:10.184488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":23780,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":616,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.011116+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:04:15.011116+0000","last_peered":"2024-09-13T15:04:15.011116+0000","last_clean":"2024-09-13T15:04:15.011116+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:04:15.011116+0000","last_undegraded":"2024-09-13T15:04:15.011116+0000","last_fullsized":"2024-09-13T15:04:15.011116+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":94,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.031862+0000","last_change":"2024-09-13T14:57:56.271513+0000","last_active":"2024-09-13T15:04:07.031862+0000","last_peered":"2024-09-13T15:04:07.031862+0000","last_clean":"2024-09-13T15:04:07.031862+0000","last_became_active":"2024-09-13T14:57:56.258172+0000","last_became_peered":"2024-09-13T14:57:56.258172+0000","last_unstale":"2024-09-13T15:04:07.031862+0000","last_undegraded":"2024-09-13T15:04:07.031862+0000","last_fullsized":"2024-09-13T15:04:07.031862+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:41.410545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":238,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.025979+0000","last_change":"2024-09-13T14:57:56.258522+0000","last_active":"2024-09-13T15:04:07.025979+0000","last_peered":"2024-09-13T15:04:07.025979+0000","last_clean":"2024-09-13T15:04:07.025979+0000","last_became_active":"2024-09-13T14:57:56.258243+0000","last_became_peered":"2024-09-13T14:57:56.258243+0000","last_unstale":"2024-09-13T15:04:07.025979+0000","last_undegraded":"2024-09-13T15:04:07.025979+0000","last_fullsized":"2024-09-13T15:04:07.025979+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:19:02.030552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":589,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.015088+0000","last_change":"2024-09-13T15:04:05.014943+0000","last_active":"2024-09-13T15:04:15.015088+0000","last_peered":"2024-09-13T15:04:15.015088+0000","last_clean":"2024-09-13T15:04:15.015088+0000","last_became_active":"2024-09-13T15:04:05.014590+0000","last_became_peered":"2024-09-13T15:04:05.014590+0000","last_unstale":"2024-09-13T15:04:15.015088+0000","last_undegraded":"2024-09-13T15:04:15.015088+0000","last_fullsized":"2024-09-13T15:04:15.015088+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:37:58.181522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":571,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008630+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:04:15.008630+0000","last_peered":"2024-09-13T15:04:15.008630+0000","last_clean":"2024-09-13T15:04:15.008630+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:04:15.008630+0000","last_undegraded":"2024-09-13T15:04:15.008630+0000","last_fullsized":"2024-09-13T15:04:15.008630+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":633,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008857+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:04:15.008857+0000","last_peered":"2024-09-13T15:04:15.008857+0000","last_clean":"2024-09-13T15:04:15.008857+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:04:15.008857+0000","last_undegraded":"2024-09-13T15:04:15.008857+0000","last_fullsized":"2024-09-13T15:04:15.008857+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":165,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.027576+0000","last_change":"2024-09-13T15:04:05.011358+0000","last_active":"2024-09-13T15:04:07.027576+0000","last_peered":"2024-09-13T15:04:07.027576+0000","last_clean":"2024-09-13T15:04:07.027576+0000","last_became_active":"2024-09-13T15:04:05.011035+0000","last_became_peered":"2024-09-13T15:04:05.011035+0000","last_unstale":"2024-09-13T15:04:07.027576+0000","last_undegraded":"2024-09-13T15:04:07.027576+0000","last_fullsized":"2024-09-13T15:04:07.027576+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:52.740914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":597,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008969+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:04:15.008969+0000","last_peered":"2024-09-13T15:04:15.008969+0000","last_clean":"2024-09-13T15:04:15.008969+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:04:15.008969+0000","last_undegraded":"2024-09-13T15:04:15.008969+0000","last_fullsized":"2024-09-13T15:04:15.008969+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":554,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.009109+0000","last_change":"2024-09-13T15:01:17.859504+0000","last_active":"2024-09-13T15:04:15.009109+0000","last_peered":"2024-09-13T15:04:15.009109+0000","last_clean":"2024-09-13T15:04:15.009109+0000","last_became_active":"2024-09-13T15:01:17.849652+0000","last_became_peered":"2024-09-13T15:01:17.849652+0000","last_unstale":"2024-09-13T15:04:15.009109+0000","last_undegraded":"2024-09-13T15:04:15.009109+0000","last_fullsized":"2024-09-13T15:04:15.009109+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:04.098478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":570,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.016079+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:04:15.016079+0000","last_peered":"2024-09-13T15:04:15.016079+0000","last_clean":"2024-09-13T15:04:15.016079+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:04:15.016079+0000","last_undegraded":"2024-09-13T15:04:15.016079+0000","last_fullsized":"2024-09-13T15:04:15.016079+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":435,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008344+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:04:15.008344+0000","last_peered":"2024-09-13T15:04:15.008344+0000","last_clean":"2024-09-13T15:04:15.008344+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:04:15.008344+0000","last_undegraded":"2024-09-13T15:04:15.008344+0000","last_fullsized":"2024-09-13T15:04:15.008344+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":362,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036262+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:04:07.036262+0000","last_peered":"2024-09-13T15:04:07.036262+0000","last_clean":"2024-09-13T15:04:07.036262+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:04:07.036262+0000","last_undegraded":"2024-09-13T15:04:07.036262+0000","last_fullsized":"2024-09-13T15:04:07.036262+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":184,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-09-13T15:04:08.046565+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:04:08.046565+0000","last_peered":"2024-09-13T15:04:08.046565+0000","last_clean":"2024-09-13T15:04:08.046565+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:04:08.046565+0000","last_undegraded":"2024-09-13T15:04:08.046565+0000","last_fullsized":"2024-09-13T15:04:08.046565+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":542,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008497+0000","last_change":"2024-09-13T14:59:42.573105+0000","last_active":"2024-09-13T15:04:15.008497+0000","last_peered":"2024-09-13T15:04:15.008497+0000","last_clean":"2024-09-13T15:04:15.008497+0000","last_became_active":"2024-09-13T14:59:42.572543+0000","last_became_peered":"2024-09-13T14:59:42.572543+0000","last_unstale":"2024-09-13T15:04:15.008497+0000","last_undegraded":"2024-09-13T15:04:15.008497+0000","last_fullsized":"2024-09-13T15:04:15.008497+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:00.772607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":372,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036498+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:04:07.036498+0000","last_peered":"2024-09-13T15:04:07.036498+0000","last_clean":"2024-09-13T15:04:07.036498+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:04:07.036498+0000","last_undegraded":"2024-09-13T15:04:07.036498+0000","last_fullsized":"2024-09-13T15:04:07.036498+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":204,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026547+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T15:04:07.026547+0000","last_peered":"2024-09-13T15:04:07.026547+0000","last_clean":"2024-09-13T15:04:07.026547+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T15:04:07.026547+0000","last_undegraded":"2024-09-13T15:04:07.026547+0000","last_fullsized":"2024-09-13T15:04:07.026547+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":360,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.036180+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:04:07.036180+0000","last_peered":"2024-09-13T15:04:07.036180+0000","last_clean":"2024-09-13T15:04:07.036180+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:04:07.036180+0000","last_undegraded":"2024-09-13T15:04:07.036180+0000","last_fullsized":"2024-09-13T15:04:07.036180+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":232,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.026666+0000","last_change":"2024-09-13T15:04:05.027223+0000","last_active":"2024-09-13T15:04:07.026666+0000","last_peered":"2024-09-13T15:04:07.026666+0000","last_clean":"2024-09-13T15:04:07.026666+0000","last_became_active":"2024-09-13T15:04:05.014803+0000","last_became_peered":"2024-09-13T15:04:05.014803+0000","last_unstale":"2024-09-13T15:04:07.026666+0000","last_undegraded":"2024-09-13T15:04:07.026666+0000","last_fullsized":"2024-09-13T15:04:07.026666+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:23.094302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":607,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.015315+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:04:15.015315+0000","last_peered":"2024-09-13T15:04:15.015315+0000","last_clean":"2024-09-13T15:04:15.015315+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:04:15.015315+0000","last_undegraded":"2024-09-13T15:04:15.015315+0000","last_fullsized":"2024-09-13T15:04:15.015315+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":629,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.009032+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:04:15.009032+0000","last_peered":"2024-09-13T15:04:15.009032+0000","last_clean":"2024-09-13T15:04:15.009032+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:04:15.009032+0000","last_undegraded":"2024-09-13T15:04:15.009032+0000","last_fullsized":"2024-09-13T15:04:15.009032+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":393,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-09-13T15:04:07.035946+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:04:07.035946+0000","last_peered":"2024-09-13T15:04:07.035946+0000","last_clean":"2024-09-13T15:04:07.035946+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:04:07.035946+0000","last_undegraded":"2024-09-13T15:04:07.035946+0000","last_fullsized":"2024-09-13T15:04:07.035946+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":596,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-09-13T15:04:15.008311+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:04:15.008311+0000","last_peered":"2024-09-13T15:04:15.008311+0000","last_clean":"2024-09-13T15:04:15.008311+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:04:15.008311+0000","last_undegraded":"2024-09-13T15:04:15.008311+0000","last_fullsized":"2024-09-13T15:04:15.008311+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":790298648,"num_objects":12060,"num_object_clones":0,"num_object_copies":36180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2734,"num_objects_unfound":0,"num_objects_dirty":12060,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220014,"num_write_kb":7426305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1271,"num_bytes_recovered":83230720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2469277696,"data_stored":2469265480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129614,"ondisk_log_size":129614,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":27080,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":619504,"kb_used_data":263464,"kb_used_omap":1,"kb_used_meta":355966,"kb_avail":93125648,"statfs":{"total":95995035648,"available":95360663552,"internally_reserved":0,"allocated":269787136,"data_stored":267820405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":364509643},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496078,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501344,"kb_used_data":46936,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":93243808,"statfs":{"total":95995035648,"available":95481659392,"internally_reserved":0,"allocated":48062464,"data_stored":46111693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":465304007},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757713,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910620,"kb_used_data":428308,"kb_used_omap":1,"kb_used_meta":482302,"kb_avail":92834532,"statfs":{"total":95995035648,"available":95062560768,"internally_reserved":0,"allocated":438587392,"data_stored":436636354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":493877703},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019353,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":810148,"kb_used_data":273692,"kb_used_omap":1,"kb_used_meta":536446,"kb_avail":92935004,"statfs":{"total":95995035648,"available":95165444096,"internally_reserved":0,"allocated":280260608,"data_stored":278305836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549321158},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468238,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960936,"kb_used_data":337884,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":92784216,"statfs":{"total":95995035648,"available":95011037184,"internally_reserved":0,"allocated":345993216,"data_stored":344042203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":637988004},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273137,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":698848,"kb_used_data":502616,"kb_used_omap":12,"kb_used_meta":196211,"kb_avail":93046304,"statfs":{"total":95995035648,"available":95279415296,"internally_reserved":0,"allocated":514678784,"data_stored":512723902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":200920960},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795290,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898792,"kb_used_data":325660,"kb_used_omap":7,"kb_used_meta":573112,"kb_avail":92846360,"statfs":{"total":95995035648,"available":95074672640,"internally_reserved":0,"allocated":333475840,"data_stored":331524756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":586867314},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502072,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492592,"kb_used_data":329124,"kb_used_omap":10,"kb_used_meta":163445,"kb_avail":93252560,"statfs":{"total":95995035648,"available":95490621440,"internally_reserved":0,"allocated":337022976,"data_stored":335064829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":167368383},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327483392,"data_stored":327483392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":328335360,"data_stored":328335360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":479137792,"data_stored":479133720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336658432,"data_stored":336658432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":267649024,"data_stored":267649024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":421859328,"data_stored":421855256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":44957696,"data_stored":44957696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":263196672,"data_stored":263192600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:04:19.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-13T15:04:19.830 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:19.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:19.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:19.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:19.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:20.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:20.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:20.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:20.238 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:20.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:20.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:20.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[102314]: from='client.55615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:20.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[102314]: pgmap v1185: 89 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 84 active+clean; 680 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 2139/32529 objects misplaced (6.576%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T15:04:20.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[102314]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T15:04:20.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[102314]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T15:04:20.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[107038]: from='client.55615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:20.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[107038]: pgmap v1185: 89 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 84 active+clean; 680 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 2139/32529 objects misplaced (6.576%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T15:04:20.383 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[107038]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T15:04:20.383 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:20 smithi154 ceph-mon[107038]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T15:04:20.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:20.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:20.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:20 smithi172 ceph-mon[100696]: from='client.55615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:04:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:20 smithi172 ceph-mon[100696]: pgmap v1185: 89 pgs: 1 peering, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 84 active+clean; 680 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 2139/32529 objects misplaced (6.576%); 9.7 MiB/s, 155 objects/s recovering 2024-09-13T15:04:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:20 smithi172 ceph-mon[100696]: osdmap e366: 8 total, 8 up, 8 in 2024-09-13T15:04:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:20 smithi172 ceph-mon[100696]: osdmap e367: 8 total, 8 up, 8 in 2024-09-13T15:04:20.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:20.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:20.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:20.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:20.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:20.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:20.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:20.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:20.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:21.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:04:21.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:21.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:21.179 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:21.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:21.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:21.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:21.405 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:21.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:21.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:21.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:21.652 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:21.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:21.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:21.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:21.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:21.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:21.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:22.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:22.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:22.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:22 smithi154 ceph-mon[102314]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T15:04:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:22 smithi154 ceph-mon[102314]: pgmap v1189: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 602 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 1710/28767 objects misplaced (5.944%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T15:04:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:22 smithi154 ceph-mon[107038]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T15:04:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:22 smithi154 ceph-mon[107038]: pgmap v1189: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 602 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 1710/28767 objects misplaced (5.944%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T15:04:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:22.223 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:04:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:04:21] "GET /metrics HTTP/1.1" 200 36631 "" "Prometheus/2.51.0" 2024-09-13T15:04:22.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:22.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:22.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:22.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:22 smithi172 ceph-mon[100696]: osdmap e368: 8 total, 8 up, 8 in 2024-09-13T15:04:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:22 smithi172 ceph-mon[100696]: pgmap v1189: 89 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 86 active+clean; 602 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 1710/28767 objects misplaced (5.944%); 9.0 MiB/s, 143 objects/s recovering 2024-09-13T15:04:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:04:22.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:22.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:22.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:22.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:22.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:22.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:22.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:22.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:22.908 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:22.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:23.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:23.032 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:23.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:23.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:23.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:23 smithi154 ceph-mon[102314]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T15:04:23.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:23 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:23.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:23 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:23.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:23 smithi154 ceph-mon[107038]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T15:04:23.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:23 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:23.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:23 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:23.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:23.303 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:23.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:23.413 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:23 smithi172 ceph-mon[100696]: osdmap e369: 8 total, 8 up, 8 in 2024-09-13T15:04:23.413 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:23 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:23.414 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:23 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:23.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:23.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:23.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:23.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:23.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:23.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:23.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:23.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:23.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:24.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:24.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:24.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:24.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:24 smithi154 ceph-mon[102314]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T15:04:24.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:24 smithi154 ceph-mon[102314]: pgmap v1192: 89 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 86 active+clean; 571 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 1289/27267 objects misplaced (4.727%); 10 MiB/s, 166 objects/s recovering 2024-09-13T15:04:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:24 smithi154 ceph-mon[107038]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T15:04:24.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:24 smithi154 ceph-mon[107038]: pgmap v1192: 89 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 86 active+clean; 571 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 1289/27267 objects misplaced (4.727%); 10 MiB/s, 166 objects/s recovering 2024-09-13T15:04:24.255 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:24.290 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:24.366 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:24 smithi172 ceph-mon[100696]: osdmap e370: 8 total, 8 up, 8 in 2024-09-13T15:04:24.366 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:24 smithi172 ceph-mon[100696]: pgmap v1192: 89 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 86 active+clean; 571 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 1289/27267 objects misplaced (4.727%); 10 MiB/s, 166 objects/s recovering 2024-09-13T15:04:24.393 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.431 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:24.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:24.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.574 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:24.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:24.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:24.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:24.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:24.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:24.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:24.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:24.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:24.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:25.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:25.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:25.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:25.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:25.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.390 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:25.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:25.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:25.530 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:25.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:25.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:25.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:25.801 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:25.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:25.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:25.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:26.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:26.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:26.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:26 smithi154 ceph-mon[102314]: pgmap v1193: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 528 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 238/25203 objects misplaced (0.944%); 20 MiB/s, 326 objects/s recovering 2024-09-13T15:04:26.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:26 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:26.169 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:26 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:26.170 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:26 smithi154 ceph-mon[107038]: pgmap v1193: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 528 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 238/25203 objects misplaced (0.944%); 20 MiB/s, 326 objects/s recovering 2024-09-13T15:04:26.170 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:26 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:26.170 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:26 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:26.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:26.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:26.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:26.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:26.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:26 smithi172 ceph-mon[100696]: pgmap v1193: 89 pgs: 1 active+remapped+backfilling, 88 active+clean; 528 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 483 op/s; 238/25203 objects misplaced (0.944%); 20 MiB/s, 326 objects/s recovering 2024-09-13T15:04:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:26 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:26 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:26.492 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.522 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:26.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:26.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:26.655 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:26.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:26.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:26.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:26.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:26.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:27.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:27.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:27.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:27.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:27.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:27.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:27.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:27.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:27.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:27.678 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:04:27.678 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:27.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:27.709 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:04:27.710 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:27.748 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:04:27.748 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:27.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:27.782 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:04:27.782 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:27.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:27.818 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:04:27.818 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:27.852 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:04:27.852 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:27.885 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:04:27.885 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:27.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:27.924 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:04:27.924 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:27.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:27.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:28.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:28.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:28.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:28.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:28.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:28.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:28.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:28.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[102314]: pgmap v1194: 89 pgs: 89 active+clean; 419 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 485 op/s; 15 MiB/s, 243 objects/s recovering 2024-09-13T15:04:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2003164307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[107038]: pgmap v1194: 89 pgs: 89 active+clean; 419 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 485 op/s; 15 MiB/s, 243 objects/s recovering 2024-09-13T15:04:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2003164307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:28.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:28.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:28.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:28 smithi172 ceph-mon[100696]: pgmap v1194: 89 pgs: 89 active+clean; 419 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 485 op/s; 15 MiB/s, 243 objects/s recovering 2024-09-13T15:04:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:28 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2003164307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:28.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:28.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:28.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:28.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:28.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:28.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:28.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:28.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:28.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:28.905 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:28.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:29.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:29.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:29.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:29.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:29.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.319 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:29.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:29.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:29.453 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:29 smithi154 ceph-mon[102314]: pgmap v1195: 89 pgs: 89 active+clean; 316 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 572 op/s; 11 MiB/s, 183 objects/s recovering 2024-09-13T15:04:29.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:29 smithi154 ceph-mon[107038]: pgmap v1195: 89 pgs: 89 active+clean; 316 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 572 op/s; 11 MiB/s, 183 objects/s recovering 2024-09-13T15:04:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:29 smithi172 ceph-mon[100696]: pgmap v1195: 89 pgs: 89 active+clean; 316 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 572 op/s; 11 MiB/s, 183 objects/s recovering 2024-09-13T15:04:29.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:29.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:29.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:29.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:29.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:29.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:29.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:29.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:30.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:30.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:30.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:30.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:30.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:30.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:30.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:30.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:30.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:30.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:30.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:30.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:30.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:30.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:31.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:31.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:31.202 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:31.231 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:31.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:31.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:31.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:31.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:31.490 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:31.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:31.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:31.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:31.763 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:31.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:04:31.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:31.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:31.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:31.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:32.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:32.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:32.129 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:04:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:04:31] "GET /metrics HTTP/1.1" 200 36614 "" "Prometheus/2.51.0" 2024-09-13T15:04:32.131 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:04:32.131 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:04:32.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:04:32.131 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:04:32.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.131 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206474 7.7 7:ed16db3f:::benchmark_data_smithi154_225924_object96210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.131 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.640343+0000", 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.484959344, 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.086262802, 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206474 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.132 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640343+0000", 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640343+0000", 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640345+0000", 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.429e-06 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640350+0000", 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7770000000000002e-06 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640358+0000", 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6139999999999992e-06 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640366+0000", 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4900000000000003e-06 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640425+0000", 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9626999999999997e-05 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640437+0000", 2024-09-13T15:04:32.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1270999999999999e-05 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.630363+0000", 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.989926521 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.630452+0000", 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8906999999999994e-05 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:04:32.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.630833+0000", 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038098 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726226+0000", 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.095393194000000001 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726438+0000", 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000211998 2024-09-13T15:04:32.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726474+0000", 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5805e-05 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726524+0000", 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0071999999999998e-05 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.726606+0000", 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2117e-05 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206476 7.6 7:6e2524ad:::benchmark_data_smithi154_225924_object96212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.48484905499998, 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0031420820000001, 2024-09-13T15:04:32.139 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206476 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640454+0000", 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5899999999999995e-07 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640458+0000", 2024-09-13T15:04:32.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7950000000000001e-06 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640466+0000", 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1440000000000003e-06 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.640485+0000", 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8547999999999999e-05 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.682274+0000", 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041789204000000003 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.682283+0000", 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5620000000000004e-06 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.622873+0000", 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.94058993999999996 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.623013+0000", 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000140309 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.623452+0000", 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000438808 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637079+0000", 2024-09-13T15:04:32.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013627121000000001 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.639954+0000", 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002875444 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643431+0000", 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003476556 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643486+0000", 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4607000000000001e-05 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643596+0000", 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011028500000000001 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.146 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206483 7.f 7:fbb8947b:::benchmark_data_smithi154_225924_object96219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.666053+0000", 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.45925009799998, 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0757986070000001, 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206483 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666053+0000", 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666053+0000", 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666054+0000", 2024-09-13T15:04:32.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.37e-06 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666059+0000", 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.412e-06 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666071+0000", 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.151e-05 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666079+0000", 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0940000000000008e-06 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666131+0000", 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2080999999999998e-05 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.666138+0000", 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.311e-06 2024-09-13T15:04:32.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.712568+0000", 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.046429209 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.712690+0000", 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012265 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:04:32.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.713129+0000", 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043868999999999999 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741477+0000", 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028348172000000001 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741649+0000", 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017229699999999999 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741689+0000", 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9153999999999998e-05 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741753+0000", 2024-09-13T15:04:32.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3941000000000003e-05 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741851+0000", 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8715999999999995e-05 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206491 7.7 7:ea59bc92:::benchmark_data_smithi154_225924_object96227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.672662+0000", 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.45264048899998, 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1077584599999999, 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206491 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.155 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672662+0000", 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672662+0000", 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672664+0000", 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672669+0000", 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1930000000000001e-06 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672687+0000", 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8133000000000001e-05 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672697+0000", 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4590000000000004e-06 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672763+0000", 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5847999999999997e-05 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.672773+0000", 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8500000000000006e-06 2024-09-13T15:04:32.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742530+0000", 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.069756988 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742637+0000", 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107477 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742809+0000", 2024-09-13T15:04:32.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000171545 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.752976+0000", 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010167914 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.756045+0000", 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030688450000000002 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.780240+0000", 2024-09-13T15:04:32.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024194462 2024-09-13T15:04:32.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.780318+0000", 2024-09-13T15:04:32.161 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 1 pgp_num to 1 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8339999999999999e-05 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.780421+0000", 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102654 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.162 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206499 7.f 7:f6551031:::benchmark_data_smithi154_225924_object96235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.681433+0000", 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.443869925, 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085242949, 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206499 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681433+0000", 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681433+0000", 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681434+0000", 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.003e-06 2024-09-13T15:04:32.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681438+0000", 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3379999999999998e-06 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681450+0000", 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2055000000000001e-05 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681465+0000", 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4653e-05 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681546+0000", 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1155999999999993e-05 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.681562+0000", 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5843000000000001e-05 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.741946+0000", 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0603843799999999 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742108+0000", 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016193100000000001 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.742232+0000", 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012353700000000001 2024-09-13T15:04:32.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745195+0000", 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029633070000000001 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.752971+0000", 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077759500000000002 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.765956+0000", 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012985294 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.766039+0000", 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2367999999999995e-05 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.766676+0000", 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00063713400000000003 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.169 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206520 7.7 7:e2c3601b:::benchmark_data_smithi154_225924_object96256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.759086+0000", 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.36621723100001, 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0266216669999999, 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.170 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206520 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759086+0000", 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759086+0000", 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759087+0000", 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3349999999999999e-06 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759093+0000", 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3609999999999997e-06 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759107+0000", 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3456e-05 2024-09-13T15:04:32.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759114+0000", 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8040000000000004e-06 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759168+0000", 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4752e-05 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.759177+0000", 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3950000000000001e-06 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745626+0000", 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98644944899999998 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745746+0000", 2024-09-13T15:04:32.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120185 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.745871+0000", 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012463900000000001 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.752981+0000", 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007110384 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.762491+0000", 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095095030000000007 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785513+0000", 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023022154 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.177 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:04:32.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785604+0000", 2024-09-13T15:04:32.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0752000000000001e-05 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.785707+0000", 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103498 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206524 7.6 7:66f18573:::benchmark_data_smithi154_225924_object96260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.767148+0000", 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.35815468300001, 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.87661840599999996, 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206524 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767148+0000", 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767148+0000", 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767149+0000", 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1480000000000001e-06 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767152+0000", 2024-09-13T15:04:32.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.948e-06 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767160+0000", 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1820000000000006e-06 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767166+0000", 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999996e-06 2024-09-13T15:04:32.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767230+0000", 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4321999999999998e-05 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.767240+0000", 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0498e-05 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637108+0000", 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86986720900000003 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637235+0000", 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012727999999999999 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.637369+0000", 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013369400000000001 2024-09-13T15:04:32.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.639995+0000", 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002626729 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643141+0000", 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003145283 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643627+0000", 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048677600000000002 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643671+0000", 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3912999999999998e-05 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643767+0000", 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5163999999999999e-05 2024-09-13T15:04:32.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206531 7.6 7:68c59cad:::benchmark_data_smithi154_225924_object96267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:42.794323+0000", 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.33097995200001, 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.849595451, 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.187 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206531 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794323+0000", 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794323+0000", 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794324+0000", 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.097e-06 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794328+0000", 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1999999999999996e-06 2024-09-13T15:04:32.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794339+0000", 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0617e-05 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794345+0000", 2024-09-13T15:04:32.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3350000000000003e-06 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794406+0000", 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0608e-05 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:42.794418+0000", 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2455e-05 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.640076+0000", 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.84565835300000003 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.640222+0000", 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000145988 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.640353+0000", 2024-09-13T15:04:32.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013003900000000001 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643276+0000", 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029229880000000001 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643338+0000", 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2345e-05 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643773+0000", 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043504699999999998 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643813+0000", 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0383000000000003e-05 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.643918+0000", 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104996 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.194 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:206534 7.7 7:e3d4229e:::benchmark_data_smithi154_225924_object96270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:43.644006+0000", 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 288.48129653400002, 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14271251600000001, 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.195 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206534 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644006+0000", 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644006+0000", 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644007+0000", 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4499999999999995e-07 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644012+0000", 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0540000000000002e-06 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644023+0000", 2024-09-13T15:04:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0325e-05 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.644029+0000", 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5400000000000001e-06 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.749539+0000", 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.10551031600000001 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.749650+0000", 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011041500000000001 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.749794+0000", 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014454800000000001 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.756007+0000", 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062123350000000003 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.762527+0000", 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006519988 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.786560+0000", 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024033591 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.786614+0000", 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3634999999999997e-05 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:43.786719+0000", 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104824 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:213192 7.6 7:678df65b:::benchmark_data_smithi154_225924_object102928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T15:04:32.201 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:51.215821+0000", 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 280.90948165100002, 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.171609816, 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213192 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215821+0000", 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215821+0000", 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215822+0000", 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.252e-06 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215826+0000", 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.738e-06 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215843+0000", 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6759e-05 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215853+0000", 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0030000000000001e-05 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.215944+0000", 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0888000000000001e-05 2024-09-13T15:04:32.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.216082+0000", 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137749 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.216206+0000", 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012477300000000001 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.219412+0000", 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032052399999999998 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.219425+0000", 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3569e-05 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.387328+0000", 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.167902411 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.387386+0000", 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8041000000000002e-05 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.387431+0000", 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5365999999999999e-05 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.208 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:213201 7.f 7:f4496a28:::benchmark_data_smithi154_225924_object102937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-13T15:04:32.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:32.210 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:59:51.223062+0000", 2024-09-13T15:04:32.210 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 280.902240828, 2024-09-13T15:04:32.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.166977974, 2024-09-13T15:04:32.210 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.210 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213201 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223062+0000", 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223062+0000", 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223063+0000", 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1799999999999999e-06 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223068+0000", 2024-09-13T15:04:32.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5739999999999997e-06 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223082+0000", 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4725e-05 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223089+0000", 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6989999999999997e-06 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223144+0000", 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5009999999999997e-05 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223254+0000", 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109583 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.223354+0000", 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100076 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.228032+0000", 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046784319999999997 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.228141+0000", 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108785 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.389936+0000", 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.161794783 2024-09-13T15:04:32.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.389994+0000", 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8616000000000003e-05 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:59:51.390040+0000", 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5510999999999998e-05 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:58592 8.6 8:66025f50:::benchmark_data_smithi154_261841_object58591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.713052+0000", 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 206.41225090699999, 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.135725176, 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 58592 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.217 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713052+0000", 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713052+0000", 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713053+0000", 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0800000000000003e-07 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713073+0000", 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0619e-05 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713083+0000", 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6730000000000004e-06 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713089+0000", 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3679999999999998e-06 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.714597+0000", 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015075799999999999 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.714701+0000", 2024-09-13T15:04:32.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103541 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.714822+0000", 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121755 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721300+0000", 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064776909999999998 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848568+0000", 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12726848599999999 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848669+0000", 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100338 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848728+0000", 2024-09-13T15:04:32.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8811000000000003e-05 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848777+0000", 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9406000000000001e-05 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:58595 8.6 8:614e9bb7:::benchmark_data_smithi154_261841_object58594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:01:05.713590+0000", 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 206.411712349, 2024-09-13T15:04:32.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13530942500000001, 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 58595 2024-09-13T15:04:32.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713590+0000", 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713590+0000", 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713591+0000", 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9000000000000005e-07 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713612+0000", 2024-09-13T15:04:32.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1036e-05 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713625+0000", 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2054e-05 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.713632+0000", 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5939999999999996e-06 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.716759+0000", 2024-09-13T15:04:32.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003127037 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.716868+0000", 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108725 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.716984+0000", 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116328 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.721304+0000", 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043200670000000004 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848612+0000", 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12730782400000001 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848819+0000", 2024-09-13T15:04:32.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000206806 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848867+0000", 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7988000000000002e-05 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:01:05.848900+0000", 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2975999999999997e-05 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.230 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213001 8.6 8:6b996dc3:::benchmark_data_smithi154_261841_object96962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.765155+0000", 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 28.360147906000002, 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16601901799999999, 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213001 2024-09-13T15:04:32.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.765155+0000", 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.765155+0000", 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.765156+0000", 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4399999999999998e-07 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.765160+0000", 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1949999999999996e-06 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.765175+0000", 2024-09-13T15:04:32.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4928e-05 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.765182+0000", 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0520000000000004e-06 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.770971+0000", 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057891360000000003 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.771092+0000", 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012039100000000001 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.771206+0000", 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114351 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.778679+0000", 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074729619999999997 2024-09-13T15:04:32.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.778962+0000", 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028299599999999999 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.931037+0000", 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.152075557 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.931129+0000", 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1643000000000003e-05 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.931174+0000", 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4863000000000001e-05 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213009 8.e 8:797d07be:::benchmark_data_smithi154_261841_object96970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:04:32.237 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.782316+0000", 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 28.342987042000001, 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15592656099999999, 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213009 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782316+0000", 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782316+0000", 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782317+0000", 2024-09-13T15:04:32.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.426e-06 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782320+0000", 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0879999999999999e-06 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782328+0000", 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8299999999999996e-06 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782334+0000", 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2160000000000003e-06 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782560+0000", 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022557400000000001 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.782896+0000", 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033572000000000002 2024-09-13T15:04:32.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,4", 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.783224+0000", 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000328513 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.787285+0000", 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040611789999999998 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.789464+0000", 2024-09-13T15:04:32.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021787659999999999 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.937979+0000", 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.148514601 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.938193+0000", 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021434899999999999 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.938242+0000", 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9298999999999997e-05 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213012 8.6 8:6f4b810a:::benchmark_data_smithi154_261841_object96973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.791792+0000", 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 28.333511224999999, 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15176920299999999, 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.244 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213012 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.791792+0000", 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.791792+0000", 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.791794+0000", 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9999999999999999e-06 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.791800+0000", 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.127e-06 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.791820+0000", 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.001e-05 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.791829+0000", 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2050000000000008e-06 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.791909+0000", 2024-09-13T15:04:32.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0325000000000001e-05 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.792018+0000", 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010869 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.792130+0000", 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111606 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801586+0000", 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0094564669999999997 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801600+0000", 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4024e-05 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.943288+0000", 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.141687759 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.943465+0000", 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000176744 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.943561+0000", 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6246e-05 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.250 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213017 8.e 8:7f32ffd1:::benchmark_data_smithi154_261841_object96978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.801887+0000", 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 28.323415433000001, 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.142436012, 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213017 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801887+0000", 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801887+0000", 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801889+0000", 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6300000000000001e-06 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801896+0000", 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.939e-06 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801910+0000", 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4154999999999999e-05 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801918+0000", 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4549999999999998e-06 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801977+0000", 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9753999999999997e-05 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.802115+0000", 2024-09-13T15:04:32.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013806300000000001 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,4", 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.802345+0000", 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022938200000000001 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.805694+0000", 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033490249999999998 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.805826+0000", 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013185399999999999 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.944192+0000", 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.138366831 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.944244+0000", 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1767000000000001e-05 2024-09-13T15:04:32.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.944323+0000", 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9157000000000001e-05 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213591 8.6 8:65202200:::benchmark_data_smithi154_261841_object97552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.777053+0000", 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 27.348249938999999, 2024-09-13T15:04:32.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.682412553, 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213591 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777053+0000", 2024-09-13T15:04:32.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777053+0000", 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777054+0000", 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.652e-06 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777062+0000", 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4120000000000004e-06 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777075+0000", 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2938e-05 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.777085+0000", 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0086e-05 2024-09-13T15:04:32.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.781795+0000", 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047104310000000002 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.781806+0000", 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0275000000000001e-05 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005566+0000", 2024-09-13T15:04:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.223760775 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005578+0000", 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1734e-05 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.428166+0000", 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4225875969999999 2024-09-13T15:04:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.428465+0000", 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029924599999999997 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6,7", 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.428595+0000", 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013002000000000001 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.438227+0000", 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009632158 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.454660+0000", 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016433239999999998 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.454742+0000", 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1969000000000001e-05 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459288+0000", 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045453660000000003 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459342+0000", 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3928000000000003e-05 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459465+0000", 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000123726 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213619 8.6 8:63de9af1:::benchmark_data_smithi154_261841_object97580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.806539+0000", 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 27.318763319999999, 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6530886570000001, 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213619 2024-09-13T15:04:32.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806539+0000", 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806539+0000", 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806541+0000", 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9939999999999999e-06 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806547+0000", 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5049999999999998e-06 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806560+0000", 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3427999999999999e-05 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806568+0000", 2024-09-13T15:04:32.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1559999999999998e-06 2024-09-13T15:04:32.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806637+0000", 2024-09-13T15:04:32.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9758999999999994e-05 2024-09-13T15:04:32.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.806647+0000", 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.465e-06 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005656+0000", 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.199009452 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005665+0000", 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5350000000000004e-06 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.433088+0000", 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.427423103 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.433196+0000", 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010835100000000001 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6,7", 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.433302+0000", 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106173 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.438273+0000", 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049708790000000001 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.454705+0000", 2024-09-13T15:04:32.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016431962000000001 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.454778+0000", 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2857999999999999e-05 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459477+0000", 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046989989999999997 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459526+0000", 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8649000000000001e-05 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459628+0000", 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102389 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213624 8.6 8:6bef8257:::benchmark_data_smithi154_261841_object97585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:04:32.275 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.812828+0000", 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 27.312474767000001, 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6469735590000001, 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213624 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812828+0000", 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812828+0000", 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812829+0000", 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.393e-06 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812835+0000", 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7259999999999999e-06 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812851+0000", 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5515e-05 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:04:32.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812864+0000", 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3078000000000001e-05 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812918+0000", 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4129000000000003e-05 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.812924+0000", 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4189999999999999e-06 2024-09-13T15:04:32.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005984+0000", 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.19305983800000001 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.005994+0000", 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0309000000000001e-05 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:04:32.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.438309+0000", 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4323148489999999 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.438418+0000", 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010894 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6,7", 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.438524+0000", 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010569399999999999 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.454816+0000", 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016292088999999999 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.454856+0000", 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0394e-05 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.454907+0000", 2024-09-13T15:04:32.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0229999999999998e-05 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459645+0000", 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047386540000000001 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459696+0000", 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0543e-05 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:04:32.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.459802+0000", 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105759 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:04:32.284 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:04:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:32.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:32.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:32.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:32.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[102314]: pgmap v1196: 89 pgs: 89 active+clean; 198 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 8.4 MiB/s, 134 objects/s recovering 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/336964238' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/492902683' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[107038]: pgmap v1196: 89 pgs: 89 active+clean; 198 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 8.4 MiB/s, 134 objects/s recovering 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/336964238' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/492902683' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:32.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:32 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:32 smithi172 ceph-mon[100696]: pgmap v1196: 89 pgs: 89 active+clean; 198 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 8.4 MiB/s, 134 objects/s recovering 2024-09-13T15:04:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/336964238' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T15:04:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:32 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-13T15:04:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/492902683' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:32 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:32.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:32.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:32.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:32.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:32.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:32.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:32.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:32.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:32.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:33.084 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Removed 116036 objects 2024-09-13T15:04:33.084 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Clean up completed and total clean up time :148.127 2024-09-13T15:04:33.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:33.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:33.161 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:33.162 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:33.162 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:04:33.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:33.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:33.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:33.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:33.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:04:33.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T15:04:33.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:33.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[102314]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T15:04:33.462 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[102314]: pgmap v1198: 89 pgs: 89 active+clean; 150 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:04:33.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T15:04:33.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:33.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[107038]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T15:04:33.463 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:33 smithi154 ceph-mon[107038]: pgmap v1198: 89 pgs: 89 active+clean; 150 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:04:33.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:33.487 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:33 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:04:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:33 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:04:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:33 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-13T15:04:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:33 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:04:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:33 smithi172 ceph-mon[100696]: osdmap e371: 8 total, 8 up, 8 in 2024-09-13T15:04:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:33 smithi172 ceph-mon[100696]: pgmap v1198: 89 pgs: 89 active+clean; 150 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-09-13T15:04:33.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:33.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:33.712 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:33.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:33.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:33.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:33.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:33.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:33.983 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:33.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:34.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:34.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:34.123 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:04:34.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:34.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:34.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:34.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:34.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:34.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:34.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:34.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:34.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:34.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:34.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:34.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:34.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:34.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:34.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:34.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:35.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:35.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:35.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:35.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:35.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:35.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:35.414 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:35.422 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:35.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:35.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:35.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:35.601 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:35.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:35.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:35.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:35.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:35.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:35.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:35.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:35.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:36.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:36.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:36.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:36.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:36.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:36.357 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:36.372 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:36 smithi154 ceph-mon[102314]: pgmap v1199: 89 pgs: 89 active+clean; 65 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 1.4 MiB/s, 23 objects/s recovering 2024-09-13T15:04:36.373 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:36 smithi154 ceph-mon[107038]: pgmap v1199: 89 pgs: 89 active+clean; 65 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 1.4 MiB/s, 23 objects/s recovering 2024-09-13T15:04:36.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:36.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:36 smithi172 ceph-mon[100696]: pgmap v1199: 89 pgs: 89 active+clean; 65 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 1.4 MiB/s, 23 objects/s recovering 2024-09-13T15:04:36.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:04:36.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:36.610 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:36.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:36.737 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:36.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:36.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:36.861 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:36.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:36.989 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:36.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:37.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:37.116 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:37.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:37.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:37.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:37.368 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:37.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:37.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:37.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:37.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:37.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:37.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:37.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:37.872 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:37.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:37.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:37.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:38.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:38.126 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:38.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:38.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:38.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:38.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:38.453 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:38 smithi154 ceph-mon[102314]: pgmap v1200: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 661 op/s 2024-09-13T15:04:38.454 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:38 smithi154 ceph-mon[107038]: pgmap v1200: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 661 op/s 2024-09-13T15:04:38.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:38 smithi172 ceph-mon[100696]: pgmap v1200: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 661 op/s 2024-09-13T15:04:38.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:38.556 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:04:38.556 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:38.574 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:04:38.575 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:38.591 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:04:38.591 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:38.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:38.613 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:04:38.614 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:38.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:38.631 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:04:38.631 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:38.648 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:04:38.649 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:38.666 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:04:38.666 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:38.682 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:04:38.683 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:38.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:38.753 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:38.839 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-13T15:04:38.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-13T15:04:38.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:38.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:38.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:39.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:39.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:39.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:39.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:39.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:39.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:39.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:39.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:39.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:39.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:39 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1039063379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:39.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:39 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1039063379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:39.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:39 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1039063379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:39.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:39.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:39.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:39.659 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:39.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:39.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:39.808 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:04:39.808 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:04:39.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:39.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:39.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:40.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:40.021 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:04:40.021 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:04:40.021 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-13T15:04:40.022 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:04:40.022 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:04:40.022 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-13T15:04:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:40.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:40.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:40.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:40.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:40.288 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:40.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:40.416 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:40 smithi154 ceph-mon[102314]: pgmap v1201: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 497 op/s 2024-09-13T15:04:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:40 smithi154 ceph-mon[107038]: pgmap v1201: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 497 op/s 2024-09-13T15:04:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:40 smithi172 ceph-mon[100696]: pgmap v1201: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 497 op/s 2024-09-13T15:04:40.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:40.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:04:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:40.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:40.610 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:40.614 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:40.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:40.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:40.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:40.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:40.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:40.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:40.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:41.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:41.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:41.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:41.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:41.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:41.296 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:41.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:41.422 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:41 smithi154 ceph-mon[102314]: pgmap v1202: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 309 op/s 2024-09-13T15:04:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:41 smithi154 ceph-mon[107038]: pgmap v1202: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 309 op/s 2024-09-13T15:04:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:41 smithi172 ceph-mon[100696]: pgmap v1202: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 309 op/s 2024-09-13T15:04:41.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:41.546 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:41.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:41.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:04:41.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:41.673 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:41.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:41.801 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:41.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:41.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:41.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:41.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:42.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:42.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:42.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:42.176 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:04:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:04:41] "GET /metrics HTTP/1.1" 200 36614 "" "Prometheus/2.51.0" 2024-09-13T15:04:42.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:42.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:42.307 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:42.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:42.431 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:42.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:42.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:42.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:42.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:42.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:42.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:42.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:42.938 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:42.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:43.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:43.061 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:43.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:43.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:43.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:43.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:43.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:43.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:43.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:43.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:43.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:43.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:43.687 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:43.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:43.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:43.845 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:43.849 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:43.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:43.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:43.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:44.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:44.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:44.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:44.190 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:44 smithi154 ceph-mon[102314]: pgmap v1203: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 212 op/s 2024-09-13T15:04:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:44 smithi154 ceph-mon[107038]: pgmap v1203: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 212 op/s 2024-09-13T15:04:44.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:44.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:44.375 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:44.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:44.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:44.464 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:44.464 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:44.464 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:44 smithi172 ceph-mon[100696]: pgmap v1203: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 212 op/s 2024-09-13T15:04:44.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:44.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:44.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:44.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:44.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:44.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:44.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:44.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:44.932 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:44.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:45.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:45.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:45.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:45.179 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:45.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:45.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:45.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:45.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:45.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:45.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:45.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:45.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:45.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:45.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:45.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:45.922 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:45.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:46.024 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:46.048 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:46.121 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:04:46.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:46.172 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:46.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:46.296 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:46.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:46.420 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:46 smithi154 ceph-mon[102314]: pgmap v1204: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 194 op/s 2024-09-13T15:04:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:46 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/609025411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:46 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:46 smithi154 ceph-mon[107038]: pgmap v1204: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 194 op/s 2024-09-13T15:04:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:46 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/609025411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:46 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:46 smithi172 ceph-mon[100696]: pgmap v1204: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 194 op/s 2024-09-13T15:04:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:46 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/609025411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:46.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:46 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:04:46.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:46.544 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:46.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:46.668 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:46.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:46.791 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:46.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:46.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:46.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:47.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:47.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:47.130 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_3' removed 2024-09-13T15:04:47.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:47.162 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:47.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:47.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[102314]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T15:04:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1642493001' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:04:47.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:47.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:47.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[107038]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T15:04:47.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:47 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1642493001' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:04:47.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:47.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:47.387 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:47.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:47 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:04:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:47 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:04:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:47 smithi172 ceph-mon[100696]: osdmap e372: 8 total, 8 up, 8 in 2024-09-13T15:04:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:47 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1642493001' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:04:47.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:47.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:47.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:47.655 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:47.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:47.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:47.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:47.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:47.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:48.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:48.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:48.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:48.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:48.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:48.275 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:48.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:48 smithi154 ceph-mon[102314]: pgmap v1206: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:48.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1642493001' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:04:48.347 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:48 smithi154 ceph-mon[102314]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T15:04:48.347 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:48 smithi154 ceph-mon[107038]: pgmap v1206: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:48.348 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1642493001' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:04:48.348 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:48 smithi154 ceph-mon[107038]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T15:04:48.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:48.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:48 smithi172 ceph-mon[100696]: pgmap v1206: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1642493001' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:04:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:48 smithi172 ceph-mon[100696]: osdmap e373: 8 total, 8 up, 8 in 2024-09-13T15:04:48.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:48.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:48.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:48.649 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:48.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:48.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:48.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:48.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:48.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:48.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:04:49.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:49.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:49.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:49.152 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:49.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:49.275 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:49.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:49.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:49.413 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:49 smithi154 ceph-mon[102314]: pgmap v1208: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:49.413 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:49 smithi154 ceph-mon[107038]: pgmap v1208: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:49 smithi172 ceph-mon[100696]: pgmap v1208: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:49.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:49.530 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:49.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:49.654 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:49.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.779 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:49.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:49.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:49.907 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:49.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:50.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:50.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:50.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:50.159 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:50.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:50.286 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:50.377 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:04:50.377 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:04:50.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:50.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:50.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:50.541 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:50.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:50.668 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:50.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:50.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:50.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:50.920 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:50.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:51.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:51.047 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:51.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:51.174 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:51.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:51.301 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:51.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:51.428 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:51.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:51.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:51.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:51.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:51 smithi154 ceph-mon[102314]: pgmap v1209: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:51.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:51 smithi154 ceph-mon[107038]: pgmap v1209: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:51 smithi172 ceph-mon[100696]: pgmap v1209: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:51.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:51.556 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:51.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:51.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:51.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:51.809 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:51.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:51.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:51.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:52.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:52.062 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:52.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:52.187 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:04:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:04:51] "GET /metrics HTTP/1.1" 200 36544 "" "Prometheus/2.51.0" 2024-09-13T15:04:52.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:52.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:52.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:52.439 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:52.460 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T15:04:52.461 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T15:04:52.461 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-13T15:04:52.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-13T15:04:52.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:52.565 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:52.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:52.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:04:52.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:04:52.677 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:52.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:04:52.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:04:52.677 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:52.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:52.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:52.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:52.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:52.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:52.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:52.978 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:04:52.978 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:04:52.990 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:04:52.990 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:04:52.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:04:52.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:04:52.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:53.007 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:04:53.007 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:04:53.024 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:04:53.024 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:04:53.040 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:04:53.040 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:04:53.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:53.056 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:04:53.056 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:04:53.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:53.072 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:04:53.072 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:04:53.088 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:04:53.089 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:04:53.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:53.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:53.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.315 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:53.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:53.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:53.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:53.539 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:53.561 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:53.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:04:53.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:53.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:53.700 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:04:53.700 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-13T15:04:53.700 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:04:53.700 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:53 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/25069338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:53.701 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:53 smithi154 ceph-mon[102314]: pgmap v1210: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:53.701 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:53 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/25069338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:53.701 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:53 smithi154 ceph-mon[107038]: pgmap v1210: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:53.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:53.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:53.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:53.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:53.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:53.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:53.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:53 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/25069338' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:04:53.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:53 smithi172 ceph-mon[100696]: pgmap v1210: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:54.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:54.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:54.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:54.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:54.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:54.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:54.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:54.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:54.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:54.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:54.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:54.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:54.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:54.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:54.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:54.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:54.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:54.988 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:55.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:55.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:55.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:55.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:55.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:55.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:55.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:55.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:55.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:55.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:55.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:55.663 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:55.684 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:55.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:55.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:55.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:55.932 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:55.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:56.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:56.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:56.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:56.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:56.210 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:04:56.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:56.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:56.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:56.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:56 smithi154 ceph-mon[102314]: pgmap v1211: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:56.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:56 smithi154 ceph-mon[107038]: pgmap v1211: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:56 smithi172 ceph-mon[100696]: pgmap v1211: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:56.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:56.561 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:56.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:56.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:56.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:56.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:56.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:56.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:56.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:56.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:04:57.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:57.065 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:57.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:57.191 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:57.241 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:04:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=infra.usagestats t=2024-09-13T15:04:56.831196897Z level=info msg="Usage stats are ready to report" 2024-09-13T15:04:57.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:57.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:57.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:57.514 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:57.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:57.642 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:57.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:04:57.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:04:57.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:57.894 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:57.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:57.996 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:58.022 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:58.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:04:58.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:04:58.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:58.277 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:58.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:58.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:58.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:58 smithi154 ceph-mon[107038]: pgmap v1212: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:58.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:58.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:58.434 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:58 smithi154 ceph-mon[102314]: pgmap v1212: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:58.434 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:58.435 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:58 smithi172 ceph-mon[100696]: pgmap v1212: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:04:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:04:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:04:58.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:58.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:58.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:58.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:58.647 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:58.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:04:58.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:04:58.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:04:58.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:58.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:58.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:04:59.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:59.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:59.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:59.132 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_4' created 2024-09-13T15:04:59.135 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:04:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:59.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:59.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:59.279 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:59.336 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:04:59.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:04:59.337 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T15:04:59.337 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:04:59.337 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:04:59.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-13T15:04:59.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:04:59.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:59.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:04:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:59 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1719120142' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T15:04:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:59 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/73912829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:59.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:04:59 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:59.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:59 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1719120142' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T15:04:59.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:59 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/73912829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:59.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:04:59 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:59 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1719120142' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-13T15:04:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:59 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/73912829' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:04:59 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:04:59.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:04:59.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:04:59.543 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:04:59.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:04:59.653 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:04:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:04:59.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:59.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:04:59.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:04:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:04:59.903 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:04:59.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:00.004 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:00.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:00.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:00.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:00.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:00 smithi172 ceph-mon[100696]: pgmap v1213: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:00.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:00 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:00.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1719120142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T15:05:00.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:00 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:00.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:00 smithi172 ceph-mon[100696]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T15:05:00.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:00.280 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[102314]: pgmap v1213: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1719120142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[102314]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[107038]: pgmap v1213: 73 pgs: 73 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:00.313 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1719120142' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-13T15:05:00.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:00.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:00 smithi154 ceph-mon[107038]: osdmap e374: 8 total, 8 up, 8 in 2024-09-13T15:05:00.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-13T15:05:00.346 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:00.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:00.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:00.505 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:00.526 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:00.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:00.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:00.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:00.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:00.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:00.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:00.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:00.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:01.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:01.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:01.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:01.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:01.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:01 smithi172 ceph-mon[100696]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T15:05:01.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:01 smithi172 ceph-mon[100696]: pgmap v1216: 89 pgs: 10 unknown, 79 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:01.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:01.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:01.295 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:01 smithi154 ceph-mon[102314]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T15:05:01.295 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:01 smithi154 ceph-mon[102314]: pgmap v1216: 89 pgs: 10 unknown, 79 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:01.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:01 smithi154 ceph-mon[107038]: osdmap e375: 8 total, 8 up, 8 in 2024-09-13T15:05:01.295 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:01 smithi154 ceph-mon[107038]: pgmap v1216: 89 pgs: 10 unknown, 79 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:01.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:01.382 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:05:01.382 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T15:05:01.382 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:05:01.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156567 8.4 e352/331 8:2820a897:::benchmark_data_smithi154_261841_object40528:head v 352'9779, mlcod=352'9777)", 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 127.366380374, 2024-09-13T15:05:01.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.20225130599999999, 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:05:01.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7099999999999998e-07 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119138+0000", 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9909999999999998e-06 2024-09-13T15:05:01.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119143+0000", 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4099999999999999e-06 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119147+0000", 2024-09-13T15:05:01.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0779999999999997e-06 2024-09-13T15:05:01.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119321+0000", 2024-09-13T15:05:01.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000173067 2024-09-13T15:05:01.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119339+0000", 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8553e-05 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321283+0000", 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.20194368900000001 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321386+0000", 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103247 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156561 8.f e352/331 8:f0bec425:::benchmark_data_smithi154_261841_object40522:head v 352'9612, mlcod=352'9610)", 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 127.366337226, 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.19389062200000001, 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.495 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2099999999999998e-07 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119181+0000", 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0920000000000002e-06 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119188+0000", 2024-09-13T15:05:01.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7190000000000004e-06 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119192+0000", 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3100000000000002e-06 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.123961+0000", 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047686250000000003 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.123979+0000", 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.312999+0000", 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18902035 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313068+0000", 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9308999999999997e-05 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.499 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156569 8.4 e352/331 8:2859ecb2:::benchmark_data_smithi154_261841_object40530:head v 352'9780, mlcod=352'9777)", 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.124429+0000", 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 127.36108576300001, 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.196985044, 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124429+0000", 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124429+0000", 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124430+0000", 2024-09-13T15:05:01.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9100000000000002e-07 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124437+0000", 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4540000000000004e-06 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124448+0000", 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0913999999999999e-05 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124454+0000", 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5989999999999998e-06 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313126+0000", 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18867229199999999 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313146+0000", 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9276999999999999e-05 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321390+0000", 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082445820000000003 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321414+0000", 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4034999999999999e-05 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.504 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156572 8.a e352/331 8:5a825b83:::benchmark_data_smithi154_261841_object40533:head v 352'9705, mlcod=352'9703)", 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 127.358096553, 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18567791, 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3900000000000003e-07 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127425+0000", 2024-09-13T15:05:01.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0079999999999999e-06 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127436+0000", 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210000000000001e-05 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127442+0000", 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6950000000000003e-06 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127512+0000", 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9210999999999999e-05 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127529+0000", 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7428999999999999e-05 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313056+0000", 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18552676900000001 2024-09-13T15:05:01.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313096+0000", 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0649000000000003e-05 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156571 8.7 e352/331 8:e2978e79:::benchmark_data_smithi154_261841_object40532:head v 352'9816, mlcod=352'9814)", 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.127461+0000", 2024-09-13T15:05:01.509 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 127.358053645, 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18564062200000001, 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127461+0000", 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127461+0000", 2024-09-13T15:05:01.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127462+0000", 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5699999999999999e-07 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127466+0000", 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2169999999999996e-06 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127472+0000", 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9510000000000003e-06 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127476+0000", 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2889999999999999e-06 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127877+0000", 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000400911 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127895+0000", 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7652999999999999e-05 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313074+0000", 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18517932400000001 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313102+0000", 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8019999999999999e-05 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156568 8.3 e352/331 8:cb5d79f8:::benchmark_data_smithi154_261841_object40529:head v 352'9697, mlcod=352'9693)", 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.131003+0000", 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 127.354512274, 2024-09-13T15:05:01.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18894451300000001, 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131003+0000", 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131003+0000", 2024-09-13T15:05:01.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131004+0000", 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2139999999999999e-06 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131009+0000", 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4129999999999997e-06 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.312989+0000", 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18197933299999999 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313005+0000", 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5883e-05 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313072+0000", 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7657999999999995e-05 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313098+0000", 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5267999999999999e-05 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.319919+0000", 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068212050000000003 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.319947+0000", 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8538999999999999e-05 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213001 8.6 e357/331 8:6b996dc3:::benchmark_data_smithi154_261841_object96962:head v 357'13323, mlcod=357'13321)", 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.777341+0000", 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.708174274999998, 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15349352199999999, 2024-09-13T15:05:01.519 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.520 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.520 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.520 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:01.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777341+0000", 2024-09-13T15:05:01.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777341+0000", 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777342+0000", 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.093e-06 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777350+0000", 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6380000000000004e-06 2024-09-13T15:05:01.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777361+0000", 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1187e-05 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777368+0000", 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6320000000000008e-06 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777461+0000", 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2388999999999999e-05 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.777481+0000", 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.001e-05 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930792+0000", 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.153311535 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930834+0000", 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2038e-05 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:212989 8.3 e357/331 8:ce4aa1c2:::benchmark_data_smithi154_261841_object96950:head v 357'13253, mlcod=357'13250)", 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.780796+0000", 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.704719355999998, 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15463231999999999, 2024-09-13T15:05:01.526 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780796+0000", 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780796+0000", 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780797+0000", 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2999999999999999e-07 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780801+0000", 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.51e-06 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780812+0000", 2024-09-13T15:05:01.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0917999999999999e-05 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780819+0000", 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1110000000000002e-06 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780934+0000", 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114931 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780952+0000", 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7924e-05 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.935382+0000", 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.15442998099999999 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.935428+0000", 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6115000000000001e-05 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.530 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:212997 8.a e357/331 8:5338dd39:::benchmark_data_smithi154_261841_object96958:head v 357'13304, mlcod=357'13301)", 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.780839+0000", 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.704676347000003, 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.149123333, 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780839+0000", 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780839+0000", 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780839+0000", 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0699999999999998e-07 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780843+0000", 2024-09-13T15:05:01.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5719999999999999e-06 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780848+0000", 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.806e-06 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.780853+0000", 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0890000000000002e-06 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.781093+0000", 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024048400000000001 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.781109+0000", 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6252e-05 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929909+0000", 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.148800139 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929962+0000", 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2683999999999999e-05 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:212996 8.1 e357/331 8:8a8dfc78:::benchmark_data_smithi154_261841_object96957:head v 357'13463, mlcod=357'13459)", 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.785068+0000", 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.700447083, 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14488778299999999, 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785068+0000", 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785068+0000", 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785069+0000", 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.144e-06 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785076+0000", 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8329999999999996e-06 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785093+0000", 2024-09-13T15:05:01.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6650000000000002e-05 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785103+0000", 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0735e-05 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785358+0000", 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00025500300000000001 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785379+0000", 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0699999999999998e-05 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801627+0000", 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016248385000000001 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929956+0000", 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12832833299999999 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213003 8.7 e357/331 8:eb02f4bf:::benchmark_data_smithi154_261841_object96964:head v 357'13329, mlcod=357'13327)", 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.785129+0000", 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.700385691999998, 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.144914508, 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785129+0000", 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785129+0000", 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785130+0000", 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0100000000000001e-07 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785133+0000", 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.523e-06 2024-09-13T15:05:01.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785139+0000", 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0260000000000004e-06 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785144+0000", 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.617e-06 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785331+0000", 2024-09-13T15:05:01.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018719799999999999 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.785355+0000", 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3896e-05 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.801631+0000", 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016276024 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930044+0000", 2024-09-13T15:05:01.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12841292300000001 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213009 8.e e357/331 8:797d07be:::benchmark_data_smithi154_261841_object96970:head v 357'13452, mlcod=357'13450)", 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.786542+0000", 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.698972775000001, 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14925197800000001, 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.786542+0000", 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.786542+0000", 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.786543+0000", 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5000000000000002e-07 2024-09-13T15:05:01.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.786549+0000", 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2179999999999996e-06 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.786560+0000", 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0705e-05 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.786566+0000", 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1430000000000003e-06 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.788893+0000", 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002327352 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.788912+0000", 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8227999999999999e-05 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.935749+0000", 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14683702000000001 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.935794+0000", 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5562000000000002e-05 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213004 8.a e357/331 8:5ed2a9ba:::benchmark_data_smithi154_261841_object96965:head v 357'13305, mlcod=357'13301)", 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.790089+0000", 2024-09-13T15:05:01.548 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.695426067, 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13989953699999999, 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790089+0000", 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790089+0000", 2024-09-13T15:05:01.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790090+0000", 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0109999999999999e-06 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790097+0000", 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-06 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790108+0000", 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1753000000000001e-05 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790115+0000", 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9970000000000001e-06 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.795231+0000", 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051156860000000004 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.795255+0000", 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4541000000000001e-05 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929965+0000", 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13470953799999999 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929989+0000", 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3530999999999999e-05 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.552 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213010 8.1 e357/331 8:83bfe91f:::benchmark_data_smithi154_261841_object96971:head v 357'13464, mlcod=357'13461)", 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.790200+0000", 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.695315016000002, 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13979267100000001, 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790200+0000", 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790200+0000", 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790201+0000", 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8699999999999995e-07 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790205+0000", 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.138e-06 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790212+0000", 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8499999999999996e-06 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790216+0000", 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5530000000000003e-06 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790277+0000", 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0856999999999997e-05 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.790294+0000", 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7053000000000001e-05 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929961+0000", 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13966720199999999 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929993+0000", 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1531e-05 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213012 8.6 e357/331 8:6f4b810a:::benchmark_data_smithi154_261841_object96973:head v 357'13324, mlcod=357'13321)", 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.797309+0000", 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.688205637000003, 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.142196036, 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.557 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797309+0000", 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797309+0000", 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797310+0000", 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7600000000000006e-07 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797318+0000", 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2690000000000002e-06 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797332+0000", 2024-09-13T15:05:01.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3922e-05 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797339+0000", 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7819999999999994e-06 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799532+0000", 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021929359999999999 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.560 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799550+0000", 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7387000000000001e-05 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.939424+0000", 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13987434000000001 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.939505+0000", 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1423999999999998e-05 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.561 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213011 8.3 e357/331 8:cdfa85e7:::benchmark_data_smithi154_261841_object96972:head v 357'13254, mlcod=357'13251)", 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.797408+0000", 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.688107285000001, 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13804796799999999, 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797408+0000", 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797408+0000", 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797408+0000", 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0500000000000004e-07 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797413+0000", 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.04e-06 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797422+0000", 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4889999999999995e-06 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797427+0000", 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0490000000000003e-06 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799970+0000", 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025431300000000002 2024-09-13T15:05:01.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799989+0000", 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8949000000000001e-05 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.935432+0000", 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.135443498 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.935456+0000", 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3308000000000001e-05 2024-09-13T15:05:01.565 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213015 8.a e357/331 8:5b821479:::benchmark_data_smithi154_261841_object96976:head v 357'13306, mlcod=357'13302)", 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.797445+0000", 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.688069882999997, 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13256847699999999, 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797445+0000", 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797445+0000", 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797445+0000", 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2700000000000001e-07 2024-09-13T15:05:01.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797449+0000", 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2610000000000002e-06 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797454+0000", 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3299999999999998e-06 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.797458+0000", 2024-09-13T15:05:01.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.783e-06 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.798787+0000", 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001329123 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.798805+0000", 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8481000000000001e-05 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929991+0000", 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13118566200000001 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930014+0000", 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2609999999999999e-05 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213014 8.3 e357/331 8:c2e5fee0:::benchmark_data_smithi154_261841_object96975:head v 357'13255, mlcod=357'13251)", 2024-09-13T15:05:01.570 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.799348+0000", 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.686166925999999, 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.1505793, 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799348+0000", 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799348+0000", 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799349+0000", 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.16e-07 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799355+0000", 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0719999999999996e-06 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799365+0000", 2024-09-13T15:05:01.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0360000000000001e-05 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.799372+0000", 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7649999999999996e-06 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930153+0000", 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13078083400000001 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930171+0000", 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8434999999999999e-05 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.949885+0000", 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019713221 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.949927+0000", 2024-09-13T15:05:01.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2796999999999997e-05 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213017 8.e e357/331 8:7f32ffd1:::benchmark_data_smithi154_261841_object96978:head v 357'13453, mlcod=357'13450)", 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.802430+0000", 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.683084645999998, 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14165534799999999, 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.802430+0000", 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.802430+0000", 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.802431+0000", 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6000000000000003e-07 2024-09-13T15:05:01.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.802437+0000", 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1109999999999996e-06 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929909+0000", 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12747172400000001 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929928+0000", 2024-09-13T15:05:01.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9097e-05 2024-09-13T15:05:01.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930516+0000", 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058808999999999999 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930536+0000", 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9995000000000001e-05 2024-09-13T15:05:01.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.944018+0000", 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013481472 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.944086+0000", 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8099e-05 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:213019 8.f e357/331 8:fdb9afea:::benchmark_data_smithi154_261841_object96980:head v 357'13151, mlcod=357'13149)", 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:03.803951+0000", 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.681564354000002, 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.139993479, 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.803951+0000", 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.803951+0000", 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.803951+0000", 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2600000000000002e-07 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:05:01.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.803958+0000", 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1489999999999999e-06 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929932+0000", 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12597441500000001 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.929942+0000", 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0030999999999999e-05 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930072+0000", 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013001899999999999 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.930096+0000", 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3595999999999999e-05 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.943898+0000", 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013802587 2024-09-13T15:05:01.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:03.943944+0000", 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5955999999999998e-05 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:05:01.586 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:05:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:01.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:01.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:01.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:01.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:01.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:01.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:01.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:01.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:02.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:02.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:02.148 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:05:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:05:01] "GET /metrics HTTP/1.1" 200 34243 "" "Prometheus/2.51.0" 2024-09-13T15:05:02.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:02.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:02.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:02.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:02.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.453 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:02.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:02 smithi154 ceph-mon[102314]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:05:02.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:02 smithi154 ceph-mon[102314]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T15:05:02.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:02 smithi154 ceph-mon[107038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:05:02.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:02 smithi154 ceph-mon[107038]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T15:05:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:02 smithi172 ceph-mon[100696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:05:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:02 smithi172 ceph-mon[100696]: osdmap e376: 8 total, 8 up, 8 in 2024-09-13T15:05:02.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:02.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:02.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:02.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:02.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:02.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:02.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:02.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:02.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:03.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:03.072 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:03.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:03.195 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:03.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:03 smithi154 ceph-mon[102314]: pgmap v1218: 89 pgs: 7 unknown, 82 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:03.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:03 smithi154 ceph-mon[107038]: pgmap v1218: 89 pgs: 7 unknown, 82 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:03.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:03.319 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:03.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:03.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:03 smithi172 ceph-mon[100696]: pgmap v1218: 89 pgs: 7 unknown, 82 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:03.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:03.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:03.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:03.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:05:03.760 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:05:03.778 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:03.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:03.795 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:05:03.795 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:03.812 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:05:03.812 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:03.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:03.828 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:05:03.828 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:03.844 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:05:03.844 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:03.860 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:05:03.860 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:03.877 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:05:03.877 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:03.917 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:03.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:03.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:04.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:04.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:04.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:04.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:04.192 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:04 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1736648683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:04.193 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:04 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1736648683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:04.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:05:04.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:04.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:04.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:04.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:04.437 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:04 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1736648683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:04.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:04.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:04.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:04.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:04.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:04.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:04.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:04.904 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:04.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:04.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:04.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:05.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:05.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:05.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:05.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:05.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:05.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:05:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:05.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:05.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:05.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:05.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:05 smithi154 ceph-mon[102314]: pgmap v1219: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:05.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:05 smithi154 ceph-mon[107038]: pgmap v1219: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:05 smithi172 ceph-mon[100696]: pgmap v1219: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:05.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:05.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:05.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:05.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:05.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:05.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:05.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:05.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:05.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:05.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:06.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:06.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:06.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:06.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:06.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:06.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:06.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:06.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:06.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:06.568 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:06.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:06.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:06.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:06.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:06.818 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:06.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:05:06.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:06.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:06.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:07.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:07.063 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:05:07.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:05:07.063 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-13T15:05:07.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd out 7 2024-09-13T15:05:07.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:07.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:07.207 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:07.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:07.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:07.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:07.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:07.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:07.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:07.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:07.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:07.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:07.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:07.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:07.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:08.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:08.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-13T15:05:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:08.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:08.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:08.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:08.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:08.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:08.455 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:08.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:08 smithi154 ceph-mon[107038]: pgmap v1220: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:08.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4050084012' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:08.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:08 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:08.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:08 smithi154 ceph-mon[102314]: pgmap v1220: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:08.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4050084012' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:08.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:08 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:08 smithi172 ceph-mon[100696]: pgmap v1220: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4050084012' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:08 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:05:08.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:08.582 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:08.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:08.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:08.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:08.831 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:08.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:08.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:08.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:09.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:09.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:09.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:09.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:09.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:09.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:09.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:09.426 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:09 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:05:09.426 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:09 smithi154 ceph-mon[102314]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T15:05:09.426 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:09 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:05:09.426 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:09 smithi154 ceph-mon[107038]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T15:05:09.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:09.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:09.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:09 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:05:09.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:09 smithi172 ceph-mon[100696]: osdmap e377: 8 total, 8 up, 8 in 2024-09-13T15:05:09.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:09.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:09.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:09.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:09.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:09.804 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:09.804 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:09.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:09.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:09.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:09.961 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:09.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:10.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:10.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:10.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:10.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:10.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:10.335 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:10.348 DEBUG:teuthology.orchestra.run.smithi154:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-13T15:05:10.351 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T15:05:10.351 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T15:05:10.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:10.444 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:hints = 1 2024-09-13T15:05:10.444 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T15:05:10.444 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object prefix: benchmark_data_smithi154_291684 2024-09-13T15:05:10.444 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:05:10.444 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 0 0 0 0 0 0 - 0 2024-09-13T15:05:10.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:10 smithi154 ceph-mon[102314]: pgmap v1222: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:10.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:10 smithi154 ceph-mon[102314]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:05:10.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:10 smithi154 ceph-mon[107038]: pgmap v1222: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:10.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:10 smithi154 ceph-mon[107038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:05:10.473 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:10 smithi172 ceph-mon[100696]: pgmap v1222: 89 pgs: 89 active+clean; 4.6 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-13T15:05:10.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:10 smithi172 ceph-mon[100696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:05:10.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:10.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:10.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:10.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:10.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:10.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:10.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:11.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:11.034 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:11.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:11.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:11.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:11.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:11.343 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:11 smithi154 ceph-mon[107038]: pgmap v1223: 89 pgs: 89 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s wr, 26 op/s 2024-09-13T15:05:11.344 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:11 smithi154 ceph-mon[102314]: pgmap v1223: 89 pgs: 89 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s wr, 26 op/s 2024-09-13T15:05:11.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:11.444 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 1 16 3938 3922 245.111 245.125 0.00262436 0.00406703 2024-09-13T15:05:11.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:11 smithi172 ceph-mon[100696]: pgmap v1223: 89 pgs: 89 active+clean; 21 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.7 MiB/s wr, 26 op/s 2024-09-13T15:05:11.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:11.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:11.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:11.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:11.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:11.857 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:11.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:11.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:11.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:12.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:12.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:12.137 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:05:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:05:11] "GET /metrics HTTP/1.1" 200 34243 "" "Prometheus/2.51.0" 2024-09-13T15:05:12.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:12.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:12.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:12.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:12.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 2 16 7700 7684 240.107 235.125 0.00268273 0.00415656 2024-09-13T15:05:12.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:12.545 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:12.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:12.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.681 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:12.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:12.782 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:12.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:12.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:12.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:12.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:13.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:13.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:13.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.238 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:13.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:13.292 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:13.293 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:13.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:13.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:13.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 3 15 10979 10964 228.388 205 0.00454136 0.00437129 2024-09-13T15:05:13.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:13.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:05:13.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:13.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:13.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:13.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:13.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:13.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:13.971 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:13.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:14.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:14.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:14.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:14 smithi154 ceph-mon[102314]: pgmap v1224: 89 pgs: 89 active+clean; 53 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s wr, 77 op/s 2024-09-13T15:05:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:14 smithi154 ceph-mon[107038]: pgmap v1224: 89 pgs: 89 active+clean; 53 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s wr, 77 op/s 2024-09-13T15:05:14.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:14.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:14.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:14.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:14.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 4 16 13997 13981 218.427 188.562 0.0081616 0.0045688 2024-09-13T15:05:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:14 smithi172 ceph-mon[100696]: pgmap v1224: 89 pgs: 89 active+clean; 53 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.8 MiB/s wr, 77 op/s 2024-09-13T15:05:14.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:14.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:14.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:14.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:14.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:14.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:14.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:14.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:14.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:15.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:15.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:15.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:15.220 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:15.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.361 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:15.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:15.362 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:05:15.362 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:15.396 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:05:15.396 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:15.432 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:05:15.432 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:15.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 5 16 16653 16637 207.936 166 0.00617184 0.00480122 2024-09-13T15:05:15.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:15.467 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:05:15.467 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:15.496 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:15.501 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:05:15.501 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:15.535 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:15 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T15:05:15.412+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e377 definitely_dead 0 2024-09-13T15:05:15.545 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:05:15.545 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:15.579 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:05:15.580 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:15.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:15.613 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:05:15.614 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:15.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:15.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:15.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:15.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:15.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:15.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:15.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:16.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:16.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:16.147 INFO:teuthology.orchestra.run.smithi154.stderr:marked out osd.7. 2024-09-13T15:05:16.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:16.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:16.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:16.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:16.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.435 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:16.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:16.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 6 16 18387 18371 191.342 108.375 0.00187203 0.00497253 2024-09-13T15:05:16.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[107038]: pgmap v1225: 89 pgs: 89 active+clean; 376 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s wr, 594 op/s 2024-09-13T15:05:16.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4106998952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:16.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/369620425' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T15:05:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T15:05:16.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[107038]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T15:05:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[102314]: pgmap v1225: 89 pgs: 89 active+clean; 376 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s wr, 594 op/s 2024-09-13T15:05:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4106998952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/369620425' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T15:05:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T15:05:16.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:16 smithi154 ceph-mon[102314]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T15:05:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:16 smithi172 ceph-mon[100696]: pgmap v1225: 89 pgs: 89 active+clean; 376 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s wr, 594 op/s 2024-09-13T15:05:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4106998952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/369620425' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T15:05:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:16 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-13T15:05:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:16 smithi172 ceph-mon[100696]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-13T15:05:16.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:16.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:16.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:16.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:16.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:16.812 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:16.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:16.940 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:16.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:17.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:17.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:17.145 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:05:17.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:17.196 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:17.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:17.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:17.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:17.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 7 16 18387 18371 164.008 0 - 0.00497253 2024-09-13T15:05:17.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:17 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T15:05:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:17 smithi154 ceph-mon[102314]: osdmap e378: 8 total, 8 up, 7 in 2024-09-13T15:05:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:17 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3059154216' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:17.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:17 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T15:05:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:17 smithi154 ceph-mon[107038]: osdmap e378: 8 total, 8 up, 7 in 2024-09-13T15:05:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:17 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3059154216' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:17 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-13T15:05:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:17 smithi172 ceph-mon[100696]: osdmap e378: 8 total, 8 up, 7 in 2024-09-13T15:05:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:17 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3059154216' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:05:17.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:17.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:17.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:17.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:17.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:17.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:17.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:17.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:17.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:18.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:18.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:18.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:18.209 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:18.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:18.304 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[102314]: pgmap v1227: 89 pgs: 89 active+clean; 944 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-13T15:05:18.304 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:18.304 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3059154216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:18.304 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[102314]: osdmap e379: 8 total, 8 up, 7 in 2024-09-13T15:05:18.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[107038]: pgmap v1227: 89 pgs: 89 active+clean; 944 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-13T15:05:18.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:18.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3059154216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:18.305 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:18 smithi154 ceph-mon[107038]: osdmap e379: 8 total, 8 up, 7 in 2024-09-13T15:05:18.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:18.372 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:18.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 8 16 18688 18672 145.859 9.40625 0.0104444 0.00684866 2024-09-13T15:05:18.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:18 smithi172 ceph-mon[100696]: pgmap v1227: 89 pgs: 89 active+clean; 944 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-13T15:05:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:18 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:05:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3059154216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:05:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:18 smithi172 ceph-mon[100696]: osdmap e379: 8 total, 8 up, 7 in 2024-09-13T15:05:18.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:18.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:18.665 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:18.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:18.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:18.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:18.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:18.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:19.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:19.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:19.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:19.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:19.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:19.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:05:19.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:19.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:19.445 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 9 16 20404 20388 141.568 107.25 0.017212 0.00705218 2024-09-13T15:05:19.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:19 smithi154 ceph-mon[107038]: osdmap e380: 8 total, 8 up, 7 in 2024-09-13T15:05:19.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:19 smithi154 ceph-mon[107038]: pgmap v1230: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 84 active+clean; 1.1 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 173 MiB/s wr, 2.77k op/s; 2280/52263 objects degraded (4.363%); 1068/52263 objects misplaced (2.044%); 662 KiB/s, 9 objects/s recovering 2024-09-13T15:05:19.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:19 smithi154 ceph-mon[102314]: osdmap e380: 8 total, 8 up, 7 in 2024-09-13T15:05:19.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:19 smithi154 ceph-mon[102314]: pgmap v1230: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 84 active+clean; 1.1 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 173 MiB/s wr, 2.77k op/s; 2280/52263 objects degraded (4.363%); 1068/52263 objects misplaced (2.044%); 662 KiB/s, 9 objects/s recovering 2024-09-13T15:05:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:19.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:19 smithi172 ceph-mon[100696]: osdmap e380: 8 total, 8 up, 7 in 2024-09-13T15:05:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:19 smithi172 ceph-mon[100696]: pgmap v1230: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+undersized+remapped, 84 active+clean; 1.1 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 173 MiB/s wr, 2.77k op/s; 2280/52263 objects degraded (4.363%); 1068/52263 objects misplaced (2.044%); 662 KiB/s, 9 objects/s recovering 2024-09-13T15:05:19.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:19.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:19.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:19.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:19.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:19.902 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:19.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:20.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:20.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:05:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:20.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:20.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:20.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:20.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:20.272 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:20 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 2280/52263 objects degraded (4.363%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:20.272 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:20 smithi154 ceph-mon[107038]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T15:05:20.273 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:20 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 2280/52263 objects degraded (4.363%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:20.273 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:20 smithi154 ceph-mon[102314]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T15:05:20.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:20.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:20.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:20.423 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:20.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 10 16 21984 21968 137.272 98.75 0.0141373 0.00727407 2024-09-13T15:05:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:20 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 2280/52263 objects degraded (4.363%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:20 smithi172 ceph-mon[100696]: osdmap e381: 8 total, 8 up, 7 in 2024-09-13T15:05:20.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:20.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:20.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:20.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:20.756 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:05:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:05:20.756 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:05:20.757 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:05:20.757 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:05:20.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:20.843 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:20.945 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:20.987 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:20.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:21.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:21.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:21.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:21.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:21.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:21.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:21.446 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 11 16 23538 23522 133.622 97.125 0.00300793 0.00747646 2024-09-13T15:05:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:21 smithi154 ceph-mon[102314]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T15:05:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:21 smithi154 ceph-mon[102314]: pgmap v1233: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 80 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s; 5678/61623 objects degraded (9.214%); 2014/61623 objects misplaced (3.268%); 3.8 MiB/s, 64 objects/s recovering 2024-09-13T15:05:21.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:21 smithi154 ceph-mon[107038]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T15:05:21.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:21 smithi154 ceph-mon[107038]: pgmap v1233: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 80 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s; 5678/61623 objects degraded (9.214%); 2014/61623 objects misplaced (3.268%); 3.8 MiB/s, 64 objects/s recovering 2024-09-13T15:05:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:21 smithi172 ceph-mon[100696]: osdmap e382: 8 total, 8 up, 7 in 2024-09-13T15:05:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:21 smithi172 ceph-mon[100696]: pgmap v1233: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 80 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s; 5678/61623 objects degraded (9.214%); 2014/61623 objects misplaced (3.268%); 3.8 MiB/s, 64 objects/s recovering 2024-09-13T15:05:21.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:21.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:21.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:21.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:21.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:21.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:21.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:21.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:21.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:22.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:22.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:22.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:22.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:05:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:05:21] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-09-13T15:05:22.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:22.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:22.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:22.446 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 12 16 24976 24960 129.976 89.875 0.00920877 0.00768495 2024-09-13T15:05:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:22.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:22.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:22.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:22.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:22.730 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:22.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:22.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:22.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:22.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:22.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:22.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:23.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:23.111 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:23.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:23.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:23.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:23.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:23.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 13 16 26154 26138 125.641 73.625 0.0061522 0.00795182 2024-09-13T15:05:23.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:23.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:23.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:23.632 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:23.682 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:05:23.682 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:05:23.682 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T15:05:23.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T15:05:23.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:23.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:23.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:23.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:23.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:23.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:24.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:24.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:24.152 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:24.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:24.281 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:24.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:24.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:24.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 14 16 27414 27398 122.292 78.75 0.00499731 0.00816967 2024-09-13T15:05:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:24 smithi154 ceph-mon[102314]: pgmap v1234: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 78 active+clean; 1.3 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s; 7985/64365 objects degraded (12.406%); 2014/64365 objects misplaced (3.129%); 2.4 MiB/s, 43 objects/s recovering 2024-09-13T15:05:24.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:24 smithi154 ceph-mon[107038]: pgmap v1234: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 78 active+clean; 1.3 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s; 7985/64365 objects degraded (12.406%); 2014/64365 objects misplaced (3.129%); 2.4 MiB/s, 43 objects/s recovering 2024-09-13T15:05:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:24 smithi172 ceph-mon[100696]: pgmap v1234: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+undersized+remapped, 78 active+clean; 1.3 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s; 7985/64365 objects degraded (12.406%); 2014/64365 objects misplaced (3.129%); 2.4 MiB/s, 43 objects/s recovering 2024-09-13T15:05:24.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:24.533 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:24.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:24.661 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:24.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:24.791 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:24.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:24.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:24.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:24.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:25.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:25.045 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:25.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:25.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:25.171 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:25.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:25.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:25.411 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:25.424 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7985/64365 objects degraded (12.406%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:25.443 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:25.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 15 16 28656 28640 119.314 77.625 0.00711166 0.00837542 2024-09-13T15:05:25.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7985/64365 objects degraded (12.406%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:25.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7985/64365 objects degraded (12.406%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:25.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:25.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:25.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:25.707 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:25.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:25.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:25.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:25.982 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:25.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:26.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:26.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:26.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:26.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:26.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:26.379 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:26.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 16 16 29889 29873 116.673 77.0625 0.00253619 0.00845407 2024-09-13T15:05:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:26 smithi154 ceph-mon[102314]: pgmap v1235: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 80 active+clean; 1.4 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 7985/71016 objects degraded (11.244%); 1278/71016 objects misplaced (1.800%); 8.7 MiB/s, 143 objects/s recovering 2024-09-13T15:05:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:26 smithi154 ceph-mon[107038]: pgmap v1235: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 80 active+clean; 1.4 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 7985/71016 objects degraded (11.244%); 1278/71016 objects misplaced (1.800%); 8.7 MiB/s, 143 objects/s recovering 2024-09-13T15:05:26.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:26 smithi172 ceph-mon[100696]: pgmap v1235: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 80 active+clean; 1.4 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 7985/71016 objects degraded (11.244%); 1278/71016 objects misplaced (1.800%); 8.7 MiB/s, 143 objects/s recovering 2024-09-13T15:05:26.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:26.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:26.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:26.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:26.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:26.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:26.872 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:26.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:26.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:26.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:27.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:27.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:27.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:27.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:27.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:27.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:27.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 17 16 30344 30328 111.483 28.4375 0.00354068 0.00896337 2024-09-13T15:05:27.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:27 smithi154 ceph-mon[102314]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T15:05:27.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:27 smithi154 ceph-mon[102314]: pgmap v1237: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 80 active+clean; 1.7 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s; 7985/85017 objects degraded (9.392%); 932/85017 objects misplaced (1.096%); 9.9 MiB/s, 162 objects/s recovering 2024-09-13T15:05:27.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:27 smithi154 ceph-mon[107038]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T15:05:27.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:27 smithi154 ceph-mon[107038]: pgmap v1237: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 80 active+clean; 1.7 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s; 7985/85017 objects degraded (9.392%); 932/85017 objects misplaced (1.096%); 9.9 MiB/s, 162 objects/s recovering 2024-09-13T15:05:27.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:27 smithi172 ceph-mon[100696]: osdmap e383: 8 total, 8 up, 7 in 2024-09-13T15:05:27.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:27 smithi172 ceph-mon[100696]: pgmap v1237: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 80 active+clean; 1.7 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s; 7985/85017 objects degraded (9.392%); 932/85017 objects misplaced (1.096%); 9.9 MiB/s, 162 objects/s recovering 2024-09-13T15:05:27.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:27.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:27.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:27.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:27.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:27.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:27.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:27.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:27.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:28.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:28.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:28.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:28.177 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:28.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:28.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:28.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:28.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:28.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 18 16 31438 31422 109.087 68.375 0.00253182 0.00909861 2024-09-13T15:05:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:28 smithi154 ceph-mon[102314]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T15:05:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:05:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:28 smithi154 ceph-mon[107038]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T15:05:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:05:28.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:28 smithi172 ceph-mon[100696]: osdmap e384: 8 total, 8 up, 7 in 2024-09-13T15:05:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:05:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:28.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:28.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:28.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:28.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:28.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:28.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:28.931 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:28.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:28.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:29.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:29.088 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:29.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:29.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:29.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:29.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:29.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 19 16 32372 32356 106.418 58.375 0.00201133 0.00932329 2024-09-13T15:05:29.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:29.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:29.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:29 smithi154 ceph-mon[102314]: pgmap v1239: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 81 active+clean; 1.8 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 7985/89901 objects degraded (8.882%); 600/89901 objects misplaced (0.667%); 11 MiB/s, 176 objects/s recovering 2024-09-13T15:05:29.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:29 smithi154 ceph-mon[107038]: pgmap v1239: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 81 active+clean; 1.8 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 7985/89901 objects degraded (8.882%); 600/89901 objects misplaced (0.667%); 11 MiB/s, 176 objects/s recovering 2024-09-13T15:05:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:29 smithi172 ceph-mon[100696]: pgmap v1239: 89 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 81 active+clean; 1.8 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 7985/89901 objects degraded (8.882%); 600/89901 objects misplaced (0.667%); 11 MiB/s, 176 objects/s recovering 2024-09-13T15:05:29.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:29.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:29.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:29.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:29.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:29.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:29.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:29.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:29.966 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:29.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:30.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:30.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:30.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:30.232 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:30.334 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:30.361 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:30.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:05:30.447141+0000 min lat: 0.0012263 max lat: 2.16284 avg lat: 0.00975709 2024-09-13T15:05:30.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:05:30.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 20 16 32794 32778 102.416 26.375 0.0067603 0.00975709 2024-09-13T15:05:30.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:30 smithi154 ceph-mon[102314]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T15:05:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 7985/89901 objects degraded (8.882%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:30.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:30 smithi154 ceph-mon[107038]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T15:05:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 7985/89901 objects degraded (8.882%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:30 smithi172 ceph-mon[100696]: osdmap e385: 8 total, 8 up, 7 in 2024-09-13T15:05:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 7985/89901 objects degraded (8.882%), 7 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:30.491 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:30.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:30.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:30.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:30.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:30.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:30.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:30.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:31.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:31.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:31.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:31.179 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:31.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:31.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:31.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:31.436 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:31 smithi154 ceph-mon[102314]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T15:05:31.436 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:31 smithi154 ceph-mon[102314]: pgmap v1242: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 762 op/s; 6859/96309 objects degraded (7.122%); 741/96309 objects misplaced (0.769%); 17 MiB/s, 266 objects/s recovering 2024-09-13T15:05:31.436 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:31 smithi154 ceph-mon[107038]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T15:05:31.436 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:31 smithi154 ceph-mon[107038]: pgmap v1242: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 762 op/s; 6859/96309 objects degraded (7.122%); 741/96309 objects misplaced (0.769%); 17 MiB/s, 266 objects/s recovering 2024-09-13T15:05:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:31.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 21 16 34201 34185 101.726 87.9375 0.0198062 0.00982419 2024-09-13T15:05:31.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:31 smithi172 ceph-mon[100696]: osdmap e386: 8 total, 8 up, 7 in 2024-09-13T15:05:31.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:31 smithi172 ceph-mon[100696]: pgmap v1242: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 762 op/s; 6859/96309 objects degraded (7.122%); 741/96309 objects misplaced (0.769%); 17 MiB/s, 266 objects/s recovering 2024-09-13T15:05:31.525 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:05:31.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:31.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:31.583 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:31.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:31.699 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:31.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:31.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:31.928 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:31.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:31.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:31.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:05:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:05:31] "GET /metrics HTTP/1.1" 200 36619 "" "Prometheus/2.51.0" 2024-09-13T15:05:32.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:32.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:32.144 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:32.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:32.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:32.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:32.364 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:32.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 22 16 35368 35352 100.417 72.9375 0.01234 0.00995192 2024-09-13T15:05:32.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:32.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:32.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:32.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:32.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:32.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:32.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:32.793 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1293661176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:32.793 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1293661176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:32.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:05:32.908 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:32.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:32.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:32.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:32.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:32.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:32.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:32.909 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:32.913 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:32.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:32.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1293661176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:33.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:33.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:33.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:33.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:33.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:33.324 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:33.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:33.447 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 23 16 36622 36606 99.4587 78.375 0.00714324 0.010046 2024-09-13T15:05:33.451 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:33.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:33.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:33.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:33.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:33.722 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:33.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:33.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:33.868 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:33 smithi154 ceph-mon[102314]: pgmap v1243: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.0 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 47 MiB/s wr, 747 op/s; 6859/98484 objects degraded (6.965%); 741/98484 objects misplaced (0.752%); 14 MiB/s, 219 objects/s recovering 2024-09-13T15:05:33.869 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:33 smithi154 ceph-mon[107038]: pgmap v1243: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.0 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 47 MiB/s wr, 747 op/s; 6859/98484 objects degraded (6.965%); 741/98484 objects misplaced (0.752%); 14 MiB/s, 219 objects/s recovering 2024-09-13T15:05:33.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:33.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:33.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:33.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:33 smithi172 ceph-mon[100696]: pgmap v1243: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.0 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 47 MiB/s wr, 747 op/s; 6859/98484 objects degraded (6.965%); 741/98484 objects misplaced (0.752%); 14 MiB/s, 219 objects/s recovering 2024-09-13T15:05:34.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:34.104 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:34.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:34.247 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:34.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:34.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:34.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 24 16 37974 37958 98.8351 84.5 0.0146404 0.0101105 2024-09-13T15:05:34.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:34.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:34.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:34.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:34.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:34.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:34.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:34.926 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:34.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:35.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:35.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:35.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:35.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:35.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:35.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:35.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:35.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:35 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6859/98484 objects degraded (6.965%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:35.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:35 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6859/98484 objects degraded (6.965%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:35.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:35.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 25 16 39381 39365 98.3989 87.9375 0.00827105 0.0101563 2024-09-13T15:05:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:35 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6859/98484 objects degraded (6.965%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:35.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:35.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:35.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:35.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:35.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:35.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:35.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:35.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:36.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:36.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:36.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:36.231 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:36.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:36.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:36.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:36.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 26 16 40707 40691 97.8016 82.875 0.00882589 0.0102174 2024-09-13T15:05:36.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:36 smithi154 ceph-mon[102314]: pgmap v1244: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 774 op/s; 6859/103434 objects degraded (6.631%); 741/103434 objects misplaced (0.716%); 10 MiB/s, 166 objects/s recovering 2024-09-13T15:05:36.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:36 smithi154 ceph-mon[107038]: pgmap v1244: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 774 op/s; 6859/103434 objects degraded (6.631%); 741/103434 objects misplaced (0.716%); 10 MiB/s, 166 objects/s recovering 2024-09-13T15:05:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:36 smithi172 ceph-mon[100696]: pgmap v1244: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 774 op/s; 6859/103434 objects degraded (6.631%); 741/103434 objects misplaced (0.716%); 10 MiB/s, 166 objects/s recovering 2024-09-13T15:05:36.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:36.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:36.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:36.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:36.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:36.804 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:36.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:36.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:36.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:37.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:37.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:37.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:37.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:37.310 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:37.338 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:37.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:37.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 27 16 42065 42049 97.3216 84.875 0.0259244 0.010267 2024-09-13T15:05:37.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:37.612 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:37.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:37.740 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:37.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:37.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:37.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:37.978 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:37.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:38.045 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:38.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:38.173 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:38.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:38 smithi154 ceph-mon[102314]: pgmap v1245: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 6859/117204 objects degraded (5.852%); 242/117204 objects misplaced (0.206%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:05:38.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:38 smithi154 ceph-mon[107038]: pgmap v1245: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 6859/117204 objects degraded (5.852%); 242/117204 objects misplaced (0.206%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:05:38.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:38.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:38.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:38.448 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 28 16 43426 43410 96.8835 85.0625 0.0174527 0.0103148 2024-09-13T15:05:38.451 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:38 smithi172 ceph-mon[100696]: pgmap v1245: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 6859/117204 objects degraded (5.852%); 242/117204 objects misplaced (0.206%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:05:38.554 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:38.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:38.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:38.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:38.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:38.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:38.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:38.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:38.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:39.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:39.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:39.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:39.232 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:39.334 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:39.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:39.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 29 16 44387 44371 95.6137 60.0625 0.00239317 0.0103887 2024-09-13T15:05:39.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:39.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:05:39.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:39.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:39.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:39.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:39.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:39.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:39.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:39.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:39.976 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:39.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:40.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:40.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:40.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:40.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:40.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:40.352 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:40.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 30 16 44665 44649 93.0057 17.375 0.00879689 0.0107395 2024-09-13T15:05:40.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:40 smithi154 ceph-mon[102314]: pgmap v1246: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 6859/123501 objects degraded (5.554%); 242/123501 objects misplaced (0.196%); 9.4 MiB/s, 149 objects/s recovering 2024-09-13T15:05:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:40 smithi154 ceph-mon[102314]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T15:05:40.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:40 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 6859/123501 objects degraded (5.554%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:40 smithi154 ceph-mon[107038]: pgmap v1246: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 6859/123501 objects degraded (5.554%); 242/123501 objects misplaced (0.196%); 9.4 MiB/s, 149 objects/s recovering 2024-09-13T15:05:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:40 smithi154 ceph-mon[107038]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T15:05:40.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:40 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 6859/123501 objects degraded (5.554%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:40.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:40 smithi172 ceph-mon[100696]: pgmap v1246: 89 pgs: 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 6859/123501 objects degraded (5.554%); 242/123501 objects misplaced (0.196%); 9.4 MiB/s, 149 objects/s recovering 2024-09-13T15:05:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:40 smithi172 ceph-mon[100696]: osdmap e387: 8 total, 8 up, 7 in 2024-09-13T15:05:40.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:40 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 6859/123501 objects degraded (5.554%), 6 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:40.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:40.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:40.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:40.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:40.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:40.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:40.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:40.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:41.019 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:41.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:41.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:41.249 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:41.273 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:41.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:41.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:41.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:41.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 31 16 46015 45999 92.7271 84.375 0.0146848 0.010779 2024-09-13T15:05:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:41 smithi154 ceph-mon[102314]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T15:05:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:41 smithi154 ceph-mon[102314]: pgmap v1249: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s; 5711/131499 objects degraded (4.343%); 859/131499 objects misplaced (0.653%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T15:05:41.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:41 smithi154 ceph-mon[107038]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T15:05:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:41 smithi154 ceph-mon[107038]: pgmap v1249: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s; 5711/131499 objects degraded (4.343%); 859/131499 objects misplaced (0.653%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T15:05:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:41 smithi172 ceph-mon[100696]: osdmap e388: 8 total, 8 up, 7 in 2024-09-13T15:05:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:41 smithi172 ceph-mon[100696]: pgmap v1249: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 86 MiB/s wr, 1.37k op/s; 5711/131499 objects degraded (4.343%); 859/131499 objects misplaced (0.653%); 8.0 MiB/s, 128 objects/s recovering 2024-09-13T15:05:41.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:41.542 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:41.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:41.668 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:41.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:41.796 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:41.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:41.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:41.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:42.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:42.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:42.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:42.202 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:05:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:05:41] "GET /metrics HTTP/1.1" 200 36619 "" "Prometheus/2.51.0" 2024-09-13T15:05:42.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:42.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:42.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:42.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:42.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 32 15 47309 47294 92.3584 80.9375 0.0138061 0.0108217 2024-09-13T15:05:42.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:42.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:42.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:42.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:42.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:42.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:42.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:42.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:43.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:43.028 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:43.060 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:05:43.072 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:43.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:43.171 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:43.210 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:43 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1760479154' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:43.211 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:43 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:43.211 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:43.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:43 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1760479154' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:43.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:43 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:43.212 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:43.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:43.298 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:43.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:43.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:43.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 33 16 48486 48470 91.7868 73.5 0.0234299 0.0108885 2024-09-13T15:05:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:43 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1760479154' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:43 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:05:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:43.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:43.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-13T15:05:43.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:43.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:43.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:43.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:43.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:43.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:43.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:43.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:44.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:44.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[102314]: osdmap e389: 8 total, 8 up, 7 in 2024-09-13T15:05:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[102314]: pgmap v1251: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s; 4545/134409 objects degraded (3.381%); 1728/134409 objects misplaced (1.286%); 8.6 MiB/s, 137 objects/s recovering 2024-09-13T15:05:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[107038]: osdmap e389: 8 total, 8 up, 7 in 2024-09-13T15:05:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:44 smithi154 ceph-mon[107038]: pgmap v1251: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s; 4545/134409 objects degraded (3.381%); 1728/134409 objects misplaced (1.286%); 8.6 MiB/s, 137 objects/s recovering 2024-09-13T15:05:44.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:44.261 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:44.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:44.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:44 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:05:44.391 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:44 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:05:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:44 smithi172 ceph-mon[100696]: osdmap e389: 8 total, 8 up, 7 in 2024-09-13T15:05:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:44 smithi172 ceph-mon[100696]: pgmap v1251: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 60 MiB/s wr, 955 op/s; 4545/134409 objects degraded (3.381%); 1728/134409 objects misplaced (1.286%); 8.6 MiB/s, 137 objects/s recovering 2024-09-13T15:05:44.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:44.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 34 16 49728 49712 91.3701 77.625 0.0228864 0.0109381 2024-09-13T15:05:44.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:44.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:44.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:44.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:44.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:44.851 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:44.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:44.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:44.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:45.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.144 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:45.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:45.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:45.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:45.373 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:45.399 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:45.448 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:45 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4545/134409 objects degraded (3.381%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:45.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:45 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4545/134409 objects degraded (3.381%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:45.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 35 16 50966 50950 90.97 77.375 0.0116476 0.0109862 2024-09-13T15:05:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:45 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4545/134409 objects degraded (3.381%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:45.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:45.534 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:45.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:45.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:45.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:45.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:45.906 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:45.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:45.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:46.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:46.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:46.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:46.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:46.202 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:46.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:46.325 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:46.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:05:46.404 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:46 smithi154 ceph-mon[102314]: pgmap v1252: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 54 MiB/s wr, 865 op/s; 4545/139080 objects degraded (3.268%); 1728/139080 objects misplaced (1.242%); 8.6 MiB/s, 137 objects/s recovering 2024-09-13T15:05:46.404 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:46 smithi154 ceph-mon[107038]: pgmap v1252: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 54 MiB/s wr, 865 op/s; 4545/139080 objects degraded (3.268%); 1728/139080 objects misplaced (1.242%); 8.6 MiB/s, 137 objects/s recovering 2024-09-13T15:05:46.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:46.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 36 16 52110 52094 90.4291 71.5 0.00546261 0.0110515 2024-09-13T15:05:46.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:46 smithi172 ceph-mon[100696]: pgmap v1252: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 2.8 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 54 MiB/s wr, 865 op/s; 4545/139080 objects degraded (3.268%); 1728/139080 objects misplaced (1.242%); 8.6 MiB/s, 137 objects/s recovering 2024-09-13T15:05:46.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:46.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:46.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:46.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:46.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:46.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:46.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:46.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:47.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:47.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:47.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:47.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:47.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:47.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:47.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:47.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 37 16 53232 53216 89.8802 70.125 0.00811382 0.0111207 2024-09-13T15:05:47.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:47.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:47.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:47.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:47.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:47.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:47.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:47.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:47.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:48.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:48.092 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:48.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:48.232 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:48.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:48.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:48.395 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:48 smithi154 ceph-mon[102314]: pgmap v1253: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s; 4545/151860 objects degraded (2.993%); 1059/151860 objects misplaced (0.697%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:05:48.397 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:48 smithi154 ceph-mon[107038]: pgmap v1253: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s; 4545/151860 objects degraded (2.993%); 1059/151860 objects misplaced (0.697%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:05:48.408 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:48.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 38 16 54274 54258 89.2286 65.125 0.0177316 0.0111997 2024-09-13T15:05:48.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:48 smithi172 ceph-mon[100696]: pgmap v1253: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s; 4545/151860 objects degraded (2.993%); 1059/151860 objects misplaced (0.697%); 12 MiB/s, 188 objects/s recovering 2024-09-13T15:05:48.499 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:48.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:48.626 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:48.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:48.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:48.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:48.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:48.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:49.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:49.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:49.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:49.296 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:49.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:49.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:49.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:49.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 39 16 55307 55291 88.596 64.5625 0.0124075 0.0112812 2024-09-13T15:05:49.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:49.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:49.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:49.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:49.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:49.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:49.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:49.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:49.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:50.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:50.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:50.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:50.273 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:50.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:50.395 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:50 smithi154 ceph-mon[102314]: pgmap v1254: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s; 4545/157407 objects degraded (2.887%); 1059/157407 objects misplaced (0.673%); 10 MiB/s, 167 objects/s recovering 2024-09-13T15:05:50.395 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:50 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4545/157407 objects degraded (2.887%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:50.395 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:50 smithi154 ceph-mon[107038]: pgmap v1254: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s; 4545/157407 objects degraded (2.887%); 1059/157407 objects misplaced (0.673%); 10 MiB/s, 167 objects/s recovering 2024-09-13T15:05:50.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:50 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4545/157407 objects degraded (2.887%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:50.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:50.449 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:05:50.449299+0000 min lat: 0.0012263 max lat: 2.16284 avg lat: 0.0113296 2024-09-13T15:05:50.450 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:05:50.450 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 40 16 56482 56466 88.2169 73.4375 0.0136366 0.0113296 2024-09-13T15:05:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:50 smithi172 ceph-mon[100696]: pgmap v1254: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s; 4545/157407 objects degraded (2.887%); 1059/157407 objects misplaced (0.673%); 10 MiB/s, 167 objects/s recovering 2024-09-13T15:05:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:50 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4545/157407 objects degraded (2.887%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:50.513 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:50.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:50.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:50.579 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:05:50.580 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:05:50.644 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:50.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:50.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:50.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:50.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:50.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:50.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:51.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:51.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:51.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:51.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:51.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:51.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:51.450 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 41 16 57489 57473 87.6001 62.9375 0.00270024 0.0113562 2024-09-13T15:05:51.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:51 smithi154 ceph-mon[102314]: pgmap v1255: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 4545/165540 objects degraded (2.746%); 618/165540 objects misplaced (0.373%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T15:05:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:51 smithi154 ceph-mon[102314]: osdmap e390: 8 total, 8 up, 7 in 2024-09-13T15:05:51.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:51 smithi154 ceph-mon[107038]: pgmap v1255: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 4545/165540 objects degraded (2.746%); 618/165540 objects misplaced (0.373%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T15:05:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:51 smithi154 ceph-mon[107038]: osdmap e390: 8 total, 8 up, 7 in 2024-09-13T15:05:51.478 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:51 smithi172 ceph-mon[100696]: pgmap v1255: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 4545/165540 objects degraded (2.746%); 618/165540 objects misplaced (0.373%); 8.8 MiB/s, 140 objects/s recovering 2024-09-13T15:05:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:51 smithi172 ceph-mon[100696]: osdmap e390: 8 total, 8 up, 7 in 2024-09-13T15:05:51.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:51.601 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:51.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:51.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:51.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:51.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:51.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:51.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:05:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:05:51] "GET /metrics HTTP/1.1" 200 36621 "" "Prometheus/2.51.0" 2024-09-13T15:05:51.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:51.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:52.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:52.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:05:52.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:52.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:52.225 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:52.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:52.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:52.450 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 42 16 57688 57672 85.8106 12.4375 0.00252192 0.011601 2024-09-13T15:05:52.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:52.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:52.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:52.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:52.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:52.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:52.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:52.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:52.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:52.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:52.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:52.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:53.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:53.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:53.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:53.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:53.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:53.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:53.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:53.450 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 43 16 57940 57924 84.1813 15.75 0.00424207 0.0118723 2024-09-13T15:05:53.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:53.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:53 smithi154 ceph-mon[102314]: osdmap e391: 8 total, 8 up, 7 in 2024-09-13T15:05:53.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:53 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:05:53.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:53 smithi154 ceph-mon[102314]: pgmap v1258: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 4545/168600 objects degraded (2.696%); 313/168600 objects misplaced (0.186%); 11 MiB/s, 176 objects/s recovering 2024-09-13T15:05:53.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:53 smithi154 ceph-mon[107038]: osdmap e391: 8 total, 8 up, 7 in 2024-09-13T15:05:53.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:53 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:05:53.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:53 smithi154 ceph-mon[107038]: pgmap v1258: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 4545/168600 objects degraded (2.696%); 313/168600 objects misplaced (0.186%); 11 MiB/s, 176 objects/s recovering 2024-09-13T15:05:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:53 smithi172 ceph-mon[100696]: osdmap e391: 8 total, 8 up, 7 in 2024-09-13T15:05:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:53 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:05:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:53 smithi172 ceph-mon[100696]: pgmap v1258: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s; 4545/168600 objects degraded (2.696%); 313/168600 objects misplaced (0.186%); 11 MiB/s, 176 objects/s recovering 2024-09-13T15:05:53.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:53.597 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:53.626 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:53.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:53.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:53.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:53.892 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:53.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:53.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:54.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:54.132 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:54.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:54.167 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:54 smithi172 ceph-mon[100696]: osdmap e392: 8 total, 8 up, 7 in 2024-09-13T15:05:54.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:54.298 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:54.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:54.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:54.450 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 44 16 58909 58893 83.6444 60.5625 0.020308 0.0119494 2024-09-13T15:05:54.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:54 smithi154 ceph-mon[102314]: osdmap e392: 8 total, 8 up, 7 in 2024-09-13T15:05:54.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:54 smithi154 ceph-mon[107038]: osdmap e392: 8 total, 8 up, 7 in 2024-09-13T15:05:54.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:54.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:54.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:54.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:54.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:54.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:54.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:54.968 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:54.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:55.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.103 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:55.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:55.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:55.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:55.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:55.363 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:55.451 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 45 16 59903 59887 83.1661 62.125 0.0289445 0.0120193 2024-09-13T15:05:55.457 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:55 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 4545/168600 objects degraded (2.696%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:55.458 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:55 smithi154 ceph-mon[102314]: pgmap v1260: 89 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 811 op/s; 3429/172023 objects degraded (1.993%); 1075/172023 objects misplaced (0.625%); 11 MiB/s, 183 objects/s recovering 2024-09-13T15:05:55.458 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:55 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 4545/168600 objects degraded (2.696%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:55.458 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:55 smithi154 ceph-mon[107038]: pgmap v1260: 89 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 811 op/s; 3429/172023 objects degraded (1.993%); 1075/172023 objects misplaced (0.625%); 11 MiB/s, 183 objects/s recovering 2024-09-13T15:05:55.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:55.490 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:55.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:55 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 4545/168600 objects degraded (2.696%), 4 pgs degraded (PG_DEGRADED) 2024-09-13T15:05:55.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:55 smithi172 ceph-mon[100696]: pgmap v1260: 89 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 51 MiB/s wr, 811 op/s; 3429/172023 objects degraded (1.993%); 1075/172023 objects misplaced (0.625%); 11 MiB/s, 183 objects/s recovering 2024-09-13T15:05:55.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:55.618 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:55.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:55.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:55.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:55.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:56.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:56.043 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:56.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:56.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:56.271 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:05:56.271 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:05:56.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:56.307 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:56.311 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:05:56.311 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:05:56.351 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:05:56.351 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:05:56.388 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:05:56.388 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:05:56.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:56.420 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:05:56.421 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:05:56.450 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 46 16 60824 60808 82.6094 57.5625 0.0102698 0.0120979 2024-09-13T15:05:56.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:56.460 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:05:56.461 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:05:56.494 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:05:56.495 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:05:56.526 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:05:56.526 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:05:56.527 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:56 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2750012369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:56.527 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:56 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2750012369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:56.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:56.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:56.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:56 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2750012369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:05:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:56.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:56.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:56.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:56.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:57.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:57.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:57.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:57.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:57.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:57.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:57.377 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:57 smithi154 ceph-mon[102314]: pgmap v1261: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 46 MiB/s wr, 742 op/s; 2288/178902 objects degraded (1.279%); 1476/178902 objects misplaced (0.825%); 15 MiB/s, 233 objects/s recovering 2024-09-13T15:05:57.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:57 smithi154 ceph-mon[107038]: pgmap v1261: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 46 MiB/s wr, 742 op/s; 2288/178902 objects degraded (1.279%); 1476/178902 objects misplaced (0.825%); 15 MiB/s, 233 objects/s recovering 2024-09-13T15:05:57.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:57.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:57.452 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 47 16 61859 61843 82.2255 64.6875 0.00817482 0.0121549 2024-09-13T15:05:57.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:57.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:57.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:57 smithi172 ceph-mon[100696]: pgmap v1261: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 46 MiB/s wr, 742 op/s; 2288/178902 objects degraded (1.279%); 1476/178902 objects misplaced (0.825%); 15 MiB/s, 233 objects/s recovering 2024-09-13T15:05:57.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:57.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:57.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:57.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:57.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:57.833 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:05:57.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:57.960 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:57.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:58.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:58.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:58.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:58.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:58.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:58.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:05:58.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-13T15:05:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:58.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:05:58.451 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 48 16 62869 62853 81.8275 63.125 0.0176687 0.0122149 2024-09-13T15:05:58.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:58.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:58.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:58.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:58.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:58.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:58.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:05:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:58.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:58.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:05:59.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:05:59.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:05:59.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:05:59.218 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:05:59.218 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:05:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:59.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:05:59.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:05:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:59.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:05:59.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:59.351 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:59.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:05:59.452 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 49 16 63772 63756 81.309 56.4375 0.0267252 0.0122918 2024-09-13T15:05:59.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:05:59.469 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:05:59.469 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:05:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:05:59.480 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:05:59.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:05:59.608 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:05:59.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:05:59.737 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:05:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:05:59.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:05:59.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:05:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:05:59.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:05:59.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:00.016 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:06:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:00.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:00.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:00.145 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:00.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:00 smithi154 ceph-mon[102314]: pgmap v1262: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 758 op/s; 2288/183594 objects degraded (1.246%); 968/183594 objects misplaced (0.527%); 15 MiB/s, 240 objects/s recovering 2024-09-13T15:06:00.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3527966553' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:00.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:00 smithi154 ceph-mon[107038]: pgmap v1262: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 758 op/s; 2288/183594 objects degraded (1.246%); 968/183594 objects misplaced (0.527%); 15 MiB/s, 240 objects/s recovering 2024-09-13T15:06:00.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3527966553' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:00.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:00.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:00.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:00.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:00 smithi172 ceph-mon[100696]: pgmap v1262: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 47 MiB/s wr, 758 op/s; 2288/183594 objects degraded (1.246%); 968/183594 objects misplaced (0.527%); 15 MiB/s, 240 objects/s recovering 2024-09-13T15:06:00.394 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3527966553' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:06:00.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:00.454 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 50 16 64825 64809 80.9947 65.8125 0.0212905 0.0123403 2024-09-13T15:06:00.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:00.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:00.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:00.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:00.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:00.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:00.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:00.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:01.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3527966553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[102314]: osdmap e393: 8 total, 8 up, 7 in 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2288/183594 objects degraded (1.246%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3527966553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[107038]: osdmap e393: 8 total, 8 up, 7 in 2024-09-13T15:06:01.123 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2288/183594 objects degraded (1.246%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:01.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:01.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:01.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:01.152 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:01.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:01.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:01.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:01.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:01.455 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 51 16 65902 65886 80.7261 67.3125 0.0177157 0.0123811 2024-09-13T15:06:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:01 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3527966553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:06:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:01 smithi172 ceph-mon[100696]: osdmap e393: 8 total, 8 up, 7 in 2024-09-13T15:06:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:01 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2288/183594 objects degraded (1.246%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:01 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:01.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:01 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:01.517 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:01.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:01.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:01.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:01.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:01.807 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:01.833 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:01.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:01.884 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:06:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:06:01] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.51.0" 2024-09-13T15:06:01.910 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:01.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:02.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:02.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:02.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:02.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:02 smithi154 ceph-mon[102314]: pgmap v1264: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 2288/191130 objects degraded (1.197%); 968/191130 objects misplaced (0.506%); 13 MiB/s, 200 objects/s recovering 2024-09-13T15:06:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:02 smithi154 ceph-mon[107038]: pgmap v1264: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 2288/191130 objects degraded (1.197%); 968/191130 objects misplaced (0.506%); 13 MiB/s, 200 objects/s recovering 2024-09-13T15:06:02.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:02.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:02.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:02.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:02.455 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 52 16 66731 66715 80.1701 51.8125 0.00427273 0.0124047 2024-09-13T15:06:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:02 smithi172 ceph-mon[100696]: pgmap v1264: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 2288/191130 objects degraded (1.197%); 968/191130 objects misplaced (0.506%); 13 MiB/s, 200 objects/s recovering 2024-09-13T15:06:02.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:02.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:02.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:02.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:02.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:02.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:02.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:02.918 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:02.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:03.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:03.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:03.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:03.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:06:03.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:03.199 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(9.9 394/379 [PushOp(9:9258837b:::benchmark_data_smithi154_291684_object14385:head, version: 377'899, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:9258837b:::benchmark_data_smithi154_291684_object14385:head@377'899, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:06:02.599018+0000", 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.00063836899999999996, 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00068912900000000002, 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599018+0000", 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599018+0000", 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599019+0000", 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1100000000000005e-07 2024-09-13T15:06:03.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599050+0000", 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0664999999999999e-05 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599071+0000", 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1546999999999999e-05 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599097+0000", 2024-09-13T15:06:03.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5678e-05 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599190+0000", 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3249000000000001e-05 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:06:02.599204+0000", 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3906999999999999e-05 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:03.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:03.204 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T15:06:03.204 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 1 2024-09-13T15:06:03.204 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:06:03.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.344 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:03.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:03.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:03.457 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 53 16 67073 67057 79.0578 21.375 0.0100845 0.0126427 2024-09-13T15:06:03.468 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:03.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:03 smithi154 ceph-mon[102314]: osdmap e394: 8 total, 8 up, 7 in 2024-09-13T15:06:03.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:03 smithi154 ceph-mon[107038]: osdmap e394: 8 total, 8 up, 7 in 2024-09-13T15:06:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:03 smithi172 ceph-mon[100696]: osdmap e394: 8 total, 8 up, 7 in 2024-09-13T15:06:03.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:03.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:03.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:03.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:03.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:03.864 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:03.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:03.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:04.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:04.119 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:04.145 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:04.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:04.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:04.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:04.417 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:04.457 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 54 16 68087 68071 78.7673 63.375 0.025766 0.0126893 2024-09-13T15:06:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:04 smithi154 ceph-mon[102314]: osdmap e395: 8 total, 8 up, 7 in 2024-09-13T15:06:04.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:04 smithi154 ceph-mon[102314]: pgmap v1267: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s; 2288/194022 objects degraded (1.179%); 629/194022 objects misplaced (0.324%); 8.8 MiB/s, 141 objects/s recovering 2024-09-13T15:06:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:04 smithi154 ceph-mon[107038]: osdmap e395: 8 total, 8 up, 7 in 2024-09-13T15:06:04.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:04 smithi154 ceph-mon[107038]: pgmap v1267: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s; 2288/194022 objects degraded (1.179%); 629/194022 objects misplaced (0.324%); 8.8 MiB/s, 141 objects/s recovering 2024-09-13T15:06:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:04 smithi172 ceph-mon[100696]: osdmap e395: 8 total, 8 up, 7 in 2024-09-13T15:06:04.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:04 smithi172 ceph-mon[100696]: pgmap v1267: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s; 2288/194022 objects degraded (1.179%); 629/194022 objects misplaced (0.324%); 8.8 MiB/s, 141 objects/s recovering 2024-09-13T15:06:04.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:04.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:04.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:04.684 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:04.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:04.831 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:04.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:04.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:04.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:04.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:05.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:05.108 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:05.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:05.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:05.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:05.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:05.406 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 2288/194022 objects degraded (1.179%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:05.406 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 2288/194022 objects degraded (1.179%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:05.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:06:05.457 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 55 16 69031 69015 78.4078 59 0.00439398 0.0126895 2024-09-13T15:06:05.476 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 2288/194022 objects degraded (1.179%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:06:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.499 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:05.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:05.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:05.624 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:05.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:05.747 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:05.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:05.870 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:05.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:05.995 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:05.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:06.096 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:06.135 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:06.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:06 smithi154 ceph-mon[102314]: osdmap e396: 8 total, 8 up, 7 in 2024-09-13T15:06:06.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:06 smithi154 ceph-mon[102314]: pgmap v1269: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 2288/197901 objects degraded (1.156%); 375/197901 objects misplaced (0.189%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T15:06:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:06 smithi154 ceph-mon[107038]: osdmap e396: 8 total, 8 up, 7 in 2024-09-13T15:06:06.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:06 smithi154 ceph-mon[107038]: pgmap v1269: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 2288/197901 objects degraded (1.156%); 375/197901 objects misplaced (0.189%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T15:06:06.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:06.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:06.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:06.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:06.458 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 56 16 69413 69397 77.4334 23.875 0.0251426 0.0129076 2024-09-13T15:06:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:06 smithi172 ceph-mon[100696]: osdmap e396: 8 total, 8 up, 7 in 2024-09-13T15:06:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:06 smithi172 ceph-mon[100696]: pgmap v1269: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 2288/197901 objects degraded (1.156%); 375/197901 objects misplaced (0.189%); 7.3 MiB/s, 116 objects/s recovering 2024-09-13T15:06:06.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:06.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:06.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:06.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:06.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:06.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:06.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:06.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:07.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:07.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:07.159 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:07.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:07 smithi154 ceph-mon[102314]: osdmap e397: 8 total, 8 up, 7 in 2024-09-13T15:06:07.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:07 smithi154 ceph-mon[107038]: osdmap e397: 8 total, 8 up, 7 in 2024-09-13T15:06:07.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:07.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:07.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:07.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:07.458 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 57 16 70437 70421 77.197 64 0.0133104 0.0129468 2024-09-13T15:06:07.473 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:06:07.473 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:07 smithi172 ceph-mon[100696]: osdmap e397: 8 total, 8 up, 7 in 2024-09-13T15:06:07.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:07.523 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:06:07.524 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:07.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:07.562 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:06:07.563 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:07.599 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:07.621 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:06:07.621 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:07.658 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:07.668 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:06:07.668 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:07.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:07.701 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:06:07.701 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:07.734 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:06:07.734 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:07.767 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:06:07.767 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:07.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:07.807 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:07.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:07.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:07.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:08.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:08.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:08.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:08.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:08.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:08.458 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 58 16 71409 71393 76.9133 60.75 0.00889171 0.0129956 2024-09-13T15:06:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:08 smithi154 ceph-mon[102314]: pgmap v1271: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 48 MiB/s wr, 763 op/s; 1164/205629 objects degraded (0.566%); 723/205629 objects misplaced (0.352%); 13 MiB/s, 203 objects/s recovering 2024-09-13T15:06:08.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2888416838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:08.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:08 smithi154 ceph-mon[107038]: pgmap v1271: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 48 MiB/s wr, 763 op/s; 1164/205629 objects degraded (0.566%); 723/205629 objects misplaced (0.352%); 13 MiB/s, 203 objects/s recovering 2024-09-13T15:06:08.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2888416838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:08.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:08 smithi172 ceph-mon[100696]: pgmap v1271: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 48 MiB/s wr, 763 op/s; 1164/205629 objects degraded (0.566%); 723/205629 objects misplaced (0.352%); 13 MiB/s, 203 objects/s recovering 2024-09-13T15:06:08.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2888416838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:08.514 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:08.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:08.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:08.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:08.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.767 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:08.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:08.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:08.895 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:08.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:08.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:09.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:09.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:09.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:09.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:09.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:09.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:09.428 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:09.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 59 16 72364 72348 76.6213 59.6875 0.00360102 0.0130426 2024-09-13T15:06:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:09 smithi154 ceph-mon[102314]: pgmap v1272: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 54 MiB/s wr, 865 op/s; 1532/209769 objects misplaced (0.730%); 14 MiB/s, 228 objects/s recovering 2024-09-13T15:06:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:09 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1164/205629 objects degraded (0.566%), 1 pg degraded) 2024-09-13T15:06:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:09 smithi154 ceph-mon[107038]: pgmap v1272: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 54 MiB/s wr, 865 op/s; 1532/209769 objects misplaced (0.730%); 14 MiB/s, 228 objects/s recovering 2024-09-13T15:06:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:09 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1164/205629 objects degraded (0.566%), 1 pg degraded) 2024-09-13T15:06:09.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:09 smithi172 ceph-mon[100696]: pgmap v1272: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 54 MiB/s wr, 865 op/s; 1532/209769 objects misplaced (0.730%); 14 MiB/s, 228 objects/s recovering 2024-09-13T15:06:09.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:09 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1164/205629 objects degraded (0.566%), 1 pg degraded) 2024-09-13T15:06:09.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:09.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:09.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:09.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:09.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:09.826 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:09.928 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:09.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:09.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:10.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:10.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:10.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:10.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:10.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:10.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:10.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:10.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:06:10.458593+0000 min lat: 0.0012263 max lat: 2.16284 avg lat: 0.013119 2024-09-13T15:06:10.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:06:10.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 60 16 73175 73159 76.189 50.6875 0.0210375 0.013119 2024-09-13T15:06:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:10.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:10.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:10.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:10.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:10.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:10.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:10.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:10.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:11.002 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:11.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:11.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:11.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:11.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:11.287 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:11.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:11.411 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:11.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 61 16 74043 74027 75.8293 54.25 0.035967 0.013164 2024-09-13T15:06:11.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:11.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:11.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:11.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:11.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:11.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:11.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:11.964 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:11.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:12.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:12.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:12.150 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:06:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:06:11] "GET /metrics HTTP/1.1" 200 36627 "" "Prometheus/2.51.0" 2024-09-13T15:06:12.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:12.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:12.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:12.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:12.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 62 16 75031 75015 75.6022 61.75 0.0161139 0.0132216 2024-09-13T15:06:12.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:12 smithi154 ceph-mon[102314]: pgmap v1273: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s; 1173/216693 objects misplaced (0.541%); 14 MiB/s, 217 objects/s recovering 2024-09-13T15:06:12.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:12 smithi154 ceph-mon[107038]: pgmap v1273: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s; 1173/216693 objects misplaced (0.541%); 14 MiB/s, 217 objects/s recovering 2024-09-13T15:06:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:12 smithi172 ceph-mon[100696]: pgmap v1273: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s; 1173/216693 objects misplaced (0.541%); 14 MiB/s, 217 objects/s recovering 2024-09-13T15:06:12.492 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:12.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:12.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:12.725 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:12.747 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:12.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:12.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:12.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:12.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:13.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:13.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:13.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:13.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:13.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:13.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:13.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 63 16 76003 75987 75.3664 60.75 0.0139452 0.0132625 2024-09-13T15:06:13.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:13.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:13.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:13.672 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:13.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:13.798 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:13.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:13.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:13.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:14.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:14.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:14.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:14.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:14.292 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:14.292 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:14.293 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:14 smithi172 ceph-mon[100696]: pgmap v1274: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 883 op/s; 1173/219192 objects misplaced (0.535%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:06:14.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:14.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:14.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:14.459 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 64 16 77058 77042 75.219 65.9375 0.0332054 0.013288 2024-09-13T15:06:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:14 smithi154 ceph-mon[102314]: pgmap v1274: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 883 op/s; 1173/219192 objects misplaced (0.535%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:06:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:14 smithi154 ceph-mon[107038]: pgmap v1274: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 883 op/s; 1173/219192 objects misplaced (0.535%); 12 MiB/s, 185 objects/s recovering 2024-09-13T15:06:14.484 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:14.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:14.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:14.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T15:06:14.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:14.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:14.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:14.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:14.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:15.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:15.066 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:15.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:15.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:15.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:15.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:15.379 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:15.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:15.463 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 65 16 78062 78046 75.0228 62.75 0.0156428 0.0133239 2024-09-13T15:06:15.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:15 smithi154 ceph-mon[102314]: pgmap v1275: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 53 MiB/s wr, 850 op/s; 883/223440 objects misplaced (0.395%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:06:15.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:15 smithi154 ceph-mon[107038]: pgmap v1275: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 53 MiB/s wr, 850 op/s; 883/223440 objects misplaced (0.395%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:06:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:15 smithi172 ceph-mon[100696]: pgmap v1275: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 53 MiB/s wr, 850 op/s; 883/223440 objects misplaced (0.395%); 11 MiB/s, 177 objects/s recovering 2024-09-13T15:06:15.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:15.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:15.630 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:15.732 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:15.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:15.864 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:15.892 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:15.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:15.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:16.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:16.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:16.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:16.236 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:16.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:16.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:16.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:16.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:16.464 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 66 16 79114 79098 74.8813 65.75 0.0143471 0.0133465 2024-09-13T15:06:16.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:16.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:16.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:16.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:16.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:16.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:16.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:16.951 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:16.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:17.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:17.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:17.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:06:17.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:17.198 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:17.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.321 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:17.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:17.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:17.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:17.464 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 67 16 79339 79323 73.9737 14.0625 0.0188064 0.0135021 2024-09-13T15:06:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:17 smithi154 ceph-mon[102314]: osdmap e398: 8 total, 8 up, 7 in 2024-09-13T15:06:17.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:17 smithi154 ceph-mon[102314]: pgmap v1277: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 58 MiB/s wr, 935 op/s; 557/233706 objects misplaced (0.238%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T15:06:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:17 smithi154 ceph-mon[107038]: osdmap e398: 8 total, 8 up, 7 in 2024-09-13T15:06:17.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:17 smithi154 ceph-mon[107038]: pgmap v1277: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 58 MiB/s wr, 935 op/s; 557/233706 objects misplaced (0.238%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T15:06:17.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:17.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:17.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:17.721 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:17 smithi172 ceph-mon[100696]: osdmap e398: 8 total, 8 up, 7 in 2024-09-13T15:06:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:17 smithi172 ceph-mon[100696]: pgmap v1277: 89 pgs: 2 active+recovering+undersized+remapped, 87 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 58 MiB/s wr, 935 op/s; 557/233706 objects misplaced (0.238%); 8.3 MiB/s, 132 objects/s recovering 2024-09-13T15:06:17.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:17.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:17.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:17.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:18.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:18.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:18.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:18.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:18.280 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:18.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:18.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:18.464 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 68 16 80411 80395 73.8711 67 0.00626754 0.0135299 2024-09-13T15:06:18.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.547 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:18.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:18.559 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:18 smithi154 ceph-mon[107038]: osdmap e399: 8 total, 8 up, 7 in 2024-09-13T15:06:18.559 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:18 smithi154 ceph-mon[102314]: osdmap e399: 8 total, 8 up, 7 in 2024-09-13T15:06:18.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:18.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:18 smithi172 ceph-mon[100696]: osdmap e399: 8 total, 8 up, 7 in 2024-09-13T15:06:18.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:18.797 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:18.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:18.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:18.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:19.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:19.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:19.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:19.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:19.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:19.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:19.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:19.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:19.417 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:19.464 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 69 16 80645 80629 73.0126 14.625 0.0040513 0.0136895 2024-09-13T15:06:19.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:19 smithi154 ceph-mon[102314]: osdmap e400: 8 total, 8 up, 7 in 2024-09-13T15:06:19.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:19 smithi154 ceph-mon[102314]: pgmap v1280: 89 pgs: 1 peering, 1 active+recovering+undersized+remapped, 87 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 38/237651 objects misplaced (0.016%); 12 MiB/s, 189 objects/s recovering 2024-09-13T15:06:19.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:19 smithi154 ceph-mon[107038]: osdmap e400: 8 total, 8 up, 7 in 2024-09-13T15:06:19.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:19 smithi154 ceph-mon[107038]: pgmap v1280: 89 pgs: 1 peering, 1 active+recovering+undersized+remapped, 87 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 38/237651 objects misplaced (0.016%); 12 MiB/s, 189 objects/s recovering 2024-09-13T15:06:19.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:19.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:19.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:19.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:19.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:19 smithi172 ceph-mon[100696]: osdmap e400: 8 total, 8 up, 7 in 2024-09-13T15:06:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:19 smithi172 ceph-mon[100696]: pgmap v1280: 89 pgs: 1 peering, 1 active+recovering+undersized+remapped, 87 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s; 38/237651 objects misplaced (0.016%); 12 MiB/s, 189 objects/s recovering 2024-09-13T15:06:19.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:19.797 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:19.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:19.925 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:19.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:19.926 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:19.927 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:20.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:20.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:20.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:20.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:20.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:20.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:20.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:20.465 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 70 16 81469 81453 72.7044 51.5 0.0130773 0.0137478 2024-09-13T15:06:20.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:20.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:20.596 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:20.617 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:20 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:20.617 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:20 smithi154 ceph-mon[102314]: osdmap e401: 8 total, 8 up, 7 in 2024-09-13T15:06:20.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:20 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:20.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:20 smithi154 ceph-mon[107038]: osdmap e401: 8 total, 8 up, 7 in 2024-09-13T15:06:20.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:20 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-13T15:06:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:20 smithi172 ceph-mon[100696]: osdmap e401: 8 total, 8 up, 7 in 2024-09-13T15:06:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:20.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:20.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:20.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:20.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:20.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:21.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:21.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:21.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:21.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:21.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:21.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:21.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:21.465 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 71 16 82536 82520 72.6196 66.6875 0.0269839 0.0137645 2024-09-13T15:06:21.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:21.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:21.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:21.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:21 smithi154 ceph-mon[107038]: pgmap v1282: 89 pgs: 1 peering, 88 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 40 MiB/s wr, 637 op/s; 7.4 MiB/s, 117 objects/s recovering 2024-09-13T15:06:21.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:21 smithi154 ceph-mon[102314]: pgmap v1282: 89 pgs: 1 peering, 88 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 40 MiB/s wr, 637 op/s; 7.4 MiB/s, 117 objects/s recovering 2024-09-13T15:06:21.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:21 smithi172 ceph-mon[100696]: pgmap v1282: 89 pgs: 1 peering, 88 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 40 MiB/s wr, 637 op/s; 7.4 MiB/s, 117 objects/s recovering 2024-09-13T15:06:21.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:21.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:21.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:21.984 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:21.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:22.069 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:06:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:06:21] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.51.0" 2024-09-13T15:06:22.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:22.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:06:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.162 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:22.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:22.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.291 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:22.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:22.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:22.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:22.467 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 72 16 83544 83528 72.4843 63 0.038808 0.0137894 2024-09-13T15:06:22.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:22.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:22.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:22 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T15:06:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:22 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T15:06:22.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:22 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-13T15:06:22.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:22.856 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:22.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:22.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:22.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:23.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:23.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:23.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:23.250 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:23.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:23.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:23.434 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[107038]: pgmap v1283: 89 pgs: 1 peering, 88 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 38 MiB/s wr, 609 op/s; 5.8 MiB/s, 92 objects/s recovering 2024-09-13T15:06:23.435 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:06:23.467 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 73 16 84488 84472 72.2993 59 0.0205299 0.0138246 2024-09-13T15:06:23.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:23.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:23.651 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:23.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:23.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[102314]: pgmap v1283: 89 pgs: 1 peering, 88 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 38 MiB/s wr, 609 op/s; 5.8 MiB/s, 92 objects/s recovering 2024-09-13T15:06:23.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2503366720' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:23.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:23.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:23.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:23.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2503366720' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:23.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:23.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:23.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:23 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:23 smithi172 ceph-mon[100696]: pgmap v1283: 89 pgs: 1 peering, 88 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 38 MiB/s wr, 609 op/s; 5.8 MiB/s, 92 objects/s recovering 2024-09-13T15:06:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:23 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2503366720' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:23.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:23 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:23.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:23 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:23.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:23 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:23.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:23.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:23.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:23.954 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:23.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:24.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:24.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:24.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:24.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:24.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:24.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:24.467 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 74 16 85499 85483 72.1761 63.1875 0.00993759 0.0138488 2024-09-13T15:06:24.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:24.513 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:24.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:24.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:24.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:24 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:24.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:24 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:24.642 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:24 smithi154 ceph-mon[102314]: osdmap e402: 8 total, 8 up, 7 in 2024-09-13T15:06:24.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:24 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:24.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:24 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:24.643 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:24 smithi154 ceph-mon[107038]: osdmap e402: 8 total, 8 up, 7 in 2024-09-13T15:06:24.653 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:24.658 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:24.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:24 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:24 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:24.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:24 smithi172 ceph-mon[100696]: osdmap e402: 8 total, 8 up, 7 in 2024-09-13T15:06:24.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:24.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:24.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:24.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:25.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:25.070 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:25.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:25.197 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:25.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:25.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:25.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:25.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:25.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:25.467 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 75 16 86415 86399 71.9771 57.25 0.00858523 0.0138874 2024-09-13T15:06:25.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:25.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:25.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:25.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:06:25.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:06:25.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:25.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[102314]: pgmap v1285: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 46 MiB/s wr, 741 op/s; 5.2 MiB/s, 82 objects/s recovering 2024-09-13T15:06:25.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:06:25.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:06:25.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:25.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:25 smithi154 ceph-mon[107038]: pgmap v1285: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 46 MiB/s wr, 741 op/s; 5.2 MiB/s, 82 objects/s recovering 2024-09-13T15:06:25.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:06:25.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:06:25.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:25.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:25 smithi172 ceph-mon[100696]: pgmap v1285: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 46 MiB/s wr, 741 op/s; 5.2 MiB/s, 82 objects/s recovering 2024-09-13T15:06:25.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:25.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:25.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:25.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:25.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:26.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:26.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:26.148 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:26.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:26.282 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:26.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:06:26.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:26.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:26.468 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 76 16 87422 87406 71.8581 62.9375 0.014884 0.0139108 2024-09-13T15:06:26.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.571 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:26.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:26.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:26.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:26.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:26.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:26.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:26.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:26.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:27.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:27.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:27.211 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:27.236 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:27.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:27.364 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:27.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:27.467 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 77 16 88447 88431 71.7568 64.0625 0.00338253 0.0139297 2024-09-13T15:06:27.486 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:27.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:27.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:27.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:27.760 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:27.862 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:27.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:27.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:27.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:28.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:28.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:28.140 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:28.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:28.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:28.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:28.468 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 78 16 89419 89403 71.6156 60.75 0.00814069 0.0139575 2024-09-13T15:06:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:28 smithi154 ceph-mon[107038]: pgmap v1286: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 MiB/s wr, 877 op/s; 304 KiB/s, 4 objects/s recovering 2024-09-13T15:06:28.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:28.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:28 smithi154 ceph-mon[102314]: pgmap v1286: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 MiB/s wr, 877 op/s; 304 KiB/s, 4 objects/s recovering 2024-09-13T15:06:28.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:28 smithi172 ceph-mon[100696]: pgmap v1286: 89 pgs: 89 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 MiB/s wr, 877 op/s; 304 KiB/s, 4 objects/s recovering 2024-09-13T15:06:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:28.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:28.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:28.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:28.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:28.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:28.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:28.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:29.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:29.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:29.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:29.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:29.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-13T15:06:29.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:29.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:29.468 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 79 16 90442 90426 71.5184 63.9375 0.00729915 0.0139767 2024-09-13T15:06:29.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:29 smithi154 ceph-mon[107038]: pgmap v1287: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s; 251 KiB/s, 3 objects/s recovering 2024-09-13T15:06:29.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:29 smithi154 ceph-mon[102314]: pgmap v1287: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s; 251 KiB/s, 3 objects/s recovering 2024-09-13T15:06:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:29 smithi172 ceph-mon[100696]: pgmap v1287: 89 pgs: 89 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s; 251 KiB/s, 3 objects/s recovering 2024-09-13T15:06:29.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:29.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:29.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:29.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:29.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:29.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:29.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:30.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:30.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:30.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:30.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:30.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:30.297 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:30.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:30.424 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:30.451 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:30.469 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:06:30.468874+0000 min lat: 0.0012263 max lat: 2.16284 avg lat: 0.0139993 2024-09-13T15:06:30.469 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:06:30.469 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 80 16 91437 91421 71.4006 62.1875 0.00971924 0.0139993 2024-09-13T15:06:30.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:30.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:30.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:30.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:30.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:30.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:30.824 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:30.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:30.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:30.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:31.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:31.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:31.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:31.237 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:31.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:31.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:31.470 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 81 16 92411 92395 71.2698 60.875 0.0216394 0.0140248 2024-09-13T15:06:31.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.513 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:31.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:31.563 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:31.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:31.631 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:06:31.631 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:31.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:31.670 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:06:31.671 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:31.705 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:06:31.705 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:31.745 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:06:31.745 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:31.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:31.777 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:06:31.777 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:31.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:31.815 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:06:31.816 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:31.849 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:06:31.849 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:31.886 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:06:31.886 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:31.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:31.932 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:31.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:31.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:06:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:06:31] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-13T15:06:32.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:32.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:32.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:32.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:32.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:32.331 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:32 smithi154 ceph-mon[102314]: pgmap v1288: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:06:32.331 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4155151818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:32.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:32 smithi154 ceph-mon[107038]: pgmap v1288: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:06:32.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4155151818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:32.338 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:32.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:32.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:32.470 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 82 16 93285 93269 71.0668 54.625 0.0237219 0.0140656 2024-09-13T15:06:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:32 smithi172 ceph-mon[100696]: pgmap v1288: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 944 op/s 2024-09-13T15:06:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4155151818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:32.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:32.593 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:32.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:32.721 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:32.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:32.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:32.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:32.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:32.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:33.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:33.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:33.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:33.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:33.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:33.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:33 smithi154 ceph-mon[102314]: pgmap v1289: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 970 op/s 2024-09-13T15:06:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:33 smithi154 ceph-mon[107038]: pgmap v1289: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 970 op/s 2024-09-13T15:06:33.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 83 16 94173 94157 70.8793 55.5 0.0206783 0.0141028 2024-09-13T15:06:33.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:33 smithi172 ceph-mon[100696]: pgmap v1289: 89 pgs: 89 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 970 op/s 2024-09-13T15:06:33.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:33.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.647 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:33.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:33.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:33.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:33.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:33.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:33.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:34.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:34.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:34.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:34.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:34.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:34.297 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:06:34.297 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:06:34.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:06:34.297 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:06:34.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.297 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172063 7.8 7:1f1ab487:::benchmark_data_smithi154_225924_object61799:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.253924+0000", 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 517.85377995099998, 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.15242054, 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172063 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.253924+0000", 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.253924+0000", 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.253927+0000", 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2069999999999998e-06 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.253932+0000", 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3539999999999999e-06 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.253945+0000", 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3412e-05 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.253955+0000", 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5489999999999995e-06 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.258179+0000", 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042243330000000003 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.258190+0000", 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0361e-05 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.404717+0000", 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.14652752 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.406345+0000", 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016278040000000001 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172080 7.8 7:19a23b20:::benchmark_data_smithi154_225924_object61816:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.254059+0000", 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 517.85364549999997, 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1662550600000001, 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:06:34.304 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172080 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254059+0000", 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254059+0000", 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254060+0000", 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2600000000000002e-07 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254063+0000", 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.783e-06 2024-09-13T15:06:34.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254073+0000", 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9110000000000005e-06 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254081+0000", 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5190000000000003e-06 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.258225+0000", 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041444280000000003 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.258423+0000", 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019790000000000001 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.406500+0000", 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.148076772 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.406747+0000", 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024738499999999998 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.406970+0000", 2024-09-13T15:06:34.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022314399999999999 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.413082+0000", 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061116820000000002 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.416830+0000", 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037481540000000001 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.420135+0000", 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003305021 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.420201+0000", 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5386e-05 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.420314+0000", 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113249 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45383.0:172082 7.8 7:19e7ec74:::benchmark_data_smithi154_225924_object61818:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e286)", 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T14:57:56.254092+0000", 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 517.85361202199999, 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1664129889999999, 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.312 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45383", 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172082 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254092+0000", 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254092+0000", 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254093+0000", 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6400000000000003e-07 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254095+0000", 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.221e-06 2024-09-13T15:06:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254101+0000", 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.254106+0000", 2024-09-13T15:06:34.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5619999999999997e-06 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.258484+0000", 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043783390000000002 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:56.258492+0000", 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4800000000000004e-06 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.411393+0000", 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1529011920000001 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.411464+0000", 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0693999999999998e-05 2024-09-13T15:06:34.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.411534+0000", 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0193999999999999e-05 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.413111+0000", 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001576991 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.416929+0000", 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038178309999999998 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.420355+0000", 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034266399999999999 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.420413+0000", 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7389000000000002e-05 2024-09-13T15:06:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T14:57:57.420505+0000", 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2639999999999994e-05 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18374 9.c 9:3ff0438f:::benchmark_data_smithi154_291684_object18373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.169900+0000", 2024-09-13T15:06:34.320 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.937804815999996, 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0942832500000002, 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18374 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169900+0000", 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169900+0000", 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169901+0000", 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.635e-06 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169926+0000", 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4556999999999999e-05 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169939+0000", 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3517e-05 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169948+0000", 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3920000000000003e-06 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.170069+0000", 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121261 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.170079+0000", 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0298e-05 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.156239+0000", 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98616019200000005 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.156252+0000", 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2894999999999999e-05 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.258169+0000", 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1019164889999999 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.258339+0000", 2024-09-13T15:06:34.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017045799999999999 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.258511+0000", 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000171965 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.261168+0000", 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002657066 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.261210+0000", 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2236e-05 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.262996+0000", 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001785888 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.263061+0000", 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4194000000000002e-05 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.264183+0000", 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011222070000000001 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.329 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32202 9.3 9:cf9537f3:::benchmark_data_smithi154_291684_object32201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.217477+0000", 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.890227475000003, 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96497188899999997, 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32202 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.217477+0000", 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.217477+0000", 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.217477+0000", 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5400000000000001e-07 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.217501+0000", 2024-09-13T15:06:34.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3144999999999999e-05 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.217508+0000", 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1149999999999996e-06 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.217513+0000", 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0679999999999996e-06 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.258713+0000", 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.041200447000000001 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.258722+0000", 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0860000000000008e-06 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.179621+0000", 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.92089855300000001 2024-09-13T15:06:34.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.179738+0000", 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011668200000000001 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.179835+0000", 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7019000000000001e-05 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182327+0000", 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024924299999999999 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182372+0000", 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4712999999999997e-05 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182381+0000", 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4490000000000005e-06 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182420+0000", 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8698000000000001e-05 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182449+0000", 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8929999999999999e-05 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32204 9.3 9:cd2bdd99:::benchmark_data_smithi154_291684_object32203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.219858+0000", 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.887846830000001, 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96595554400000005, 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32204 2024-09-13T15:06:34.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.219858+0000", 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.219858+0000", 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.219869+0000", 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1456e-05 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.219900+0000", 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1007000000000003e-05 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.219912+0000", 2024-09-13T15:06:34.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2138e-05 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.219919+0000", 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2919999999999998e-06 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.258749+0000", 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038830075999999998 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.258757+0000", 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0530000000000001e-06 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.181424+0000", 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.92266771999999997 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.181531+0000", 2024-09-13T15:06:34.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010665799999999999 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.181625+0000", 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4130999999999997e-05 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.183420+0000", 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017951110000000001 2024-09-13T15:06:34.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.183432+0000", 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1513e-05 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.185715+0000", 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022828449999999999 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.185766+0000", 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1542999999999998e-05 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.185813+0000", 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7000999999999999e-05 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32208 9.3 9:cbfcf0b6:::benchmark_data_smithi154_291684_object32207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.220784+0000", 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.886919956, 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.96525115299999997, 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32208 2024-09-13T15:06:34.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.220784+0000", 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.220784+0000", 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.220786+0000", 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1289999999999999e-06 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.220836+0000", 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0825999999999998e-05 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.257043+0000", 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036206617000000003 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.257057+0000", 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.454e-05 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.258879+0000", 2024-09-13T15:06:34.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018210100000000001 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.258887+0000", 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4800000000000001e-06 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182529+0000", 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.92364231200000002 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182613+0000", 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3619e-05 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.182743+0000", 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013001500000000001 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.185899+0000", 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031556219999999999 2024-09-13T15:06:34.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.185950+0000", 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1495000000000001e-05 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.185962+0000", 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1905999999999999e-05 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186004+0000", 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1693000000000002e-05 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186036+0000", 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1888999999999999e-05 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32225 9.3 9:cac84025:::benchmark_data_smithi154_291684_object32224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.271139+0000", 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.836565342, 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.91505057999999995, 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32225 2024-09-13T15:06:34.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271139+0000", 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271139+0000", 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271140+0000", 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3680000000000001e-06 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271165+0000", 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4216999999999999e-05 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271176+0000", 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1508999999999999e-05 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271184+0000", 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5109999999999997e-06 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271257+0000", 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3538e-05 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271268+0000", 2024-09-13T15:06:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0474e-05 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.183451+0000", 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.91218368699999997 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.183530+0000", 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.86e-05 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.183614+0000", 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4388999999999995e-05 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186045+0000", 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002431047 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186082+0000", 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6406999999999999e-05 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186121+0000", 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8773999999999998e-05 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186160+0000", 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9128999999999998e-05 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186190+0000", 2024-09-13T15:06:34.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.993e-05 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32228 9.3 9:c908c69a:::benchmark_data_smithi154_291684_object32227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.271276+0000", 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.836428251000001, 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.91504623500000004, 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.361 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32228 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271276+0000", 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271276+0000", 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271277+0000", 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8800000000000002e-07 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271306+0000", 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8991000000000001e-05 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271314+0000", 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.67e-06 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271320+0000", 2024-09-13T15:06:34.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6219999999999997e-06 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271378+0000", 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7747999999999997e-05 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.271383+0000", 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0989999999999998e-06 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.184510+0000", 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.91312676999999998 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.184643+0000", 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000133898 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.184775+0000", 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013145299999999999 2024-09-13T15:06:34.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186091+0000", 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013163840000000001 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186222+0000", 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013057300000000001 2024-09-13T15:06:34.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186255+0000", 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2669000000000002e-05 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186295+0000", 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0629e-05 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186322+0000", 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7141e-05 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32239 9.3 9:cec85314:::benchmark_data_smithi154_291684_object32238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.277817+0000", 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.829887739, 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.91580333700000005, 2024-09-13T15:06:34.369 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32239 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277817+0000", 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277817+0000", 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277818+0000", 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1090000000000001e-06 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277841+0000", 2024-09-13T15:06:34.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3139999999999999e-05 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277851+0000", 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9790000000000005e-06 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277857+0000", 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1149999999999999e-06 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277928+0000", 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1111000000000004e-05 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.277936+0000", 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.174e-06 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186347+0000", 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.90841108000000004 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186447+0000", 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100128 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.186610+0000", 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000162409 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.189471+0000", 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028612070000000002 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.191760+0000", 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022889400000000002 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193476+0000", 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001715924 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193576+0000", 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100388 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193620+0000", 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3633000000000001e-05 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.376 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32260 9.3 9:ce3eb2fe:::benchmark_data_smithi154_291684_object32259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.290832+0000", 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.816872107999998, 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.90837569500000004, 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.377 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32260 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290832+0000", 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290832+0000", 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290834+0000", 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.339e-06 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290854+0000", 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0064e-05 2024-09-13T15:06:34.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290874+0000", 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0049999999999999e-05 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290880+0000", 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6800000000000004e-06 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290935+0000", 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.503e-05 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.290944+0000", 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6200000000000005e-06 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.187868+0000", 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89692434799999998 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.188017+0000", 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014904 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.188160+0000", 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000142707 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193664+0000", 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005503572 2024-09-13T15:06:34.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193700+0000", 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6557999999999998e-05 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199116+0000", 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054155990000000001 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199164+0000", 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8303999999999997e-05 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199208+0000", 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3784e-05 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.384 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32276 9.3 9:c2a75d80:::benchmark_data_smithi154_291684_object32275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.301255+0000", 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.806448891000002, 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.89807046499999998, 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32276 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.385 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301255+0000", 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301255+0000", 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301256+0000", 2024-09-13T15:06:34.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.61e-07 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301280+0000", 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3343999999999999e-05 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301290+0000", 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0386e-05 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301297+0000", 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7689999999999999e-06 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301371+0000", 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3990000000000001e-05 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.301380+0000", 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6390000000000004e-06 2024-09-13T15:06:34.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.189656+0000", 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.88827590099999998 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.189787+0000", 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013118200000000001 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.189916+0000", 2024-09-13T15:06:34.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000128615 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193734+0000", 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038176970000000001 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199099+0000", 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053649689999999998 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199258+0000", 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000158966 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199295+0000", 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7481999999999998e-05 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199326+0000", 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0763999999999999e-05 2024-09-13T15:06:34.391 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32308 9.3 9:cc982876:::benchmark_data_smithi154_291684_object32307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.327931+0000", 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.7797731, 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.87146295600000001, 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.392 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32308 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.327931+0000", 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.327931+0000", 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.327933+0000", 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8509999999999999e-06 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.327960+0000", 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6471000000000001e-05 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.327971+0000", 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1154e-05 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.327977+0000", 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6170000000000003e-06 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.328065+0000", 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7886000000000005e-05 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.328076+0000", 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0909e-05 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.191798+0000", 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86372189099999996 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.191962+0000", 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000164279 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.192106+0000", 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014320700000000001 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193773+0000", 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016673289999999999 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199247+0000", 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054740250000000004 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199332+0000", 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4952999999999998e-05 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199366+0000", 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.43e-05 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199394+0000", 2024-09-13T15:06:34.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8084000000000001e-05 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32322 9.3 9:ca334aa3:::benchmark_data_smithi154_291684_object32321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.335734+0000", 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.771970422999999, 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.86555974499999999, 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.399 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32322 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335734+0000", 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335734+0000", 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335734+0000", 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.46e-07 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335790+0000", 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6107000000000003e-05 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335799+0000", 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4270000000000008e-06 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335805+0000", 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6540000000000004e-06 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335892+0000", 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7224000000000004e-05 2024-09-13T15:06:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.335900+0000", 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5750000000000001e-06 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193804+0000", 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.85790339999999998 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.193948+0000", 2024-09-13T15:06:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014457900000000001 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.194093+0000", 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000144807 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199400+0000", 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053071009999999998 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199438+0000", 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7506999999999999e-05 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201203+0000", 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017656379999999999 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201251+0000", 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7836000000000001e-05 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201294+0000", 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2444000000000002e-05 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32348 9.3 9:c28a786b:::benchmark_data_smithi154_291684_object32347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.358450+0000", 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.749254649999997, 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.84301177100000002, 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32348 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358450+0000", 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358450+0000", 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358451+0000", 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.528e-06 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358476+0000", 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.429e-05 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358488+0000", 2024-09-13T15:06:34.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2235999999999999e-05 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358495+0000", 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0910000000000004e-06 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358567+0000", 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.216e-05 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.358578+0000", 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1442e-05 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.195529+0000", 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83695077200000001 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.195674+0000", 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000144604 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.195809+0000", 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013491800000000001 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199405+0000", 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035959770000000002 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199468+0000", 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3560999999999997e-05 2024-09-13T15:06:34.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201333+0000", 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001864697 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201383+0000", 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9972999999999997e-05 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201461+0000", 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8522000000000005e-05 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32353 9.3 9:c6116d0e:::benchmark_data_smithi154_291684_object32352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.362057+0000", 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.745647403999996, 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83958481399999996, 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.414 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32353 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362057+0000", 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362057+0000", 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362058+0000", 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.403e-06 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362087+0000", 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8943000000000001e-05 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362114+0000", 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7025e-05 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362120+0000", 2024-09-13T15:06:34.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5840000000000002e-06 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362174+0000", 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4355999999999999e-05 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.362181+0000", 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.888e-06 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.197270+0000", 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83508906999999999 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.197414+0000", 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014413699999999999 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.197591+0000", 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017686700000000001 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199409+0000", 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018176480000000001 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199497+0000", 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8095000000000003e-05 2024-09-13T15:06:34.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201568+0000", 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020711589999999999 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201610+0000", 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2218000000000001e-05 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201642+0000", 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1421000000000003e-05 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.422 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32357 9.3 9:c82cc226:::benchmark_data_smithi154_291684_object32356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.365437+0000", 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.742267283999993, 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.84656181900000005, 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32357 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365437+0000", 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365437+0000", 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365438+0000", 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2890000000000001e-06 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365455+0000", 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7079999999999999e-05 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365466+0000", 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1072e-05 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365472+0000", 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6779999999999997e-06 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365561+0000", 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8555999999999997e-05 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.365573+0000", 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2346e-05 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199520+0000", 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83394677299999997 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199641+0000", 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120906 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.199761+0000", 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119967 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.207496+0000", 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077356070000000002 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.207537+0000", 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0243000000000001e-05 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.211906+0000", 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043696489999999998 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.211952+0000", 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6184999999999999e-05 2024-09-13T15:06:34.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.211999+0000", 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6468000000000003e-05 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32363 9.3 9:cdd0d056:::benchmark_data_smithi154_291684_object32362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.430 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.370125+0000", 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.737579292999996, 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.84198689999999998, 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32363 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370125+0000", 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370125+0000", 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370127+0000", 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.697e-06 2024-09-13T15:06:34.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370150+0000", 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3552000000000001e-05 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370164+0000", 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3903000000000001e-05 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370172+0000", 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6510000000000001e-06 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370234+0000", 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1734000000000002e-05 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.370241+0000", 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8569999999999995e-06 2024-09-13T15:06:34.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201686+0000", 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83144416700000001 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.201985+0000", 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029908300000000001 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.202098+0000", 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011369100000000001 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.207549+0000", 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054502409999999998 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.207574+0000", 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5490000000000002e-05 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212042+0000", 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044682250000000001 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212082+0000", 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9906e-05 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212112+0000", 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9703e-05 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32369 9.3 9:ca7ef0fc:::benchmark_data_smithi154_291684_object32368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.377688+0000", 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.730016602000006, 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83452293799999999, 2024-09-13T15:06:34.438 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32369 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377688+0000", 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377688+0000", 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377690+0000", 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8610000000000001e-06 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377714+0000", 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4012999999999999e-05 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377732+0000", 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8692999999999999e-05 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377741+0000", 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0350000000000007e-06 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377807+0000", 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5314000000000005e-05 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.377819+0000", 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2234e-05 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.203547+0000", 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.82572820599999996 2024-09-13T15:06:34.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.203684+0000", 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013663299999999999 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.203826+0000", 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014198800000000001 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.207603+0000", 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003777533 2024-09-13T15:06:34.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.211856+0000", 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042523040000000002 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212144+0000", 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028872899999999999 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212183+0000", 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8318999999999997e-05 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212211+0000", 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8076e-05 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:32372 9.3 9:c97723ed:::benchmark_data_smithi154_291684_object32371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-09-13T15:06:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:29.383123+0000", 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.724580983999999, 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.83301933100000003, 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 32372 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383123+0000", 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383123+0000", 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383124+0000", 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.091e-06 2024-09-13T15:06:34.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383153+0000", 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.862e-05 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383167+0000", 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3981e-05 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:06:34.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383175+0000", 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7999999999999999e-06 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383238+0000", 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2967999999999997e-05 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:29.383249+0000", 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1195e-05 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.207629+0000", 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.82437974000000003 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.207747+0000", 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118078 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.208142+0000", 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039529099999999998 2024-09-13T15:06:34.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212010+0000", 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038675670000000001 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.212244+0000", 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000234171 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:34.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.216000+0000", 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037556590000000002 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.216102+0000", 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102607 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:30.216143+0000", 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0562999999999998e-05 2024-09-13T15:06:34.452 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.453 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:06:34.453 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:06:34.453 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:06:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:34.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:34.470 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 84 16 94986 94970 70.6404 50.8125 0.0145717 0.0141506 2024-09-13T15:06:34.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:34.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:34.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:34.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:34.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:34.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:34.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:34.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:34.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:35.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:35.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:35.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:35.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:35.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:35.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:35.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:35 smithi154 ceph-mon[102314]: pgmap v1290: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-13T15:06:35.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:35 smithi154 ceph-mon[107038]: pgmap v1290: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-13T15:06:35.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 85 16 95971 95955 70.5335 61.5625 0.0182279 0.0141721 2024-09-13T15:06:35.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:35 smithi172 ceph-mon[100696]: pgmap v1290: 89 pgs: 89 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-13T15:06:35.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:35.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:35.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:35.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:35.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:35.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:35.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:35.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:35.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:36.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:36.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:36.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:36.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:36.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:36.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:36.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:36.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:36.470 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 86 16 96897 96881 70.3863 57.875 0.0117097 0.0142013 2024-09-13T15:06:36.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:36.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:36.679 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:36.703 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:36.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:36.828 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:36.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:36.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:36.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:37.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:37.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:37.110 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:06:37.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:37.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:37.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:06:37.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:37.356 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:37 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2624293025' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:37.356 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:37 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:37.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:37.420 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:37 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2624293025' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:37.420 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:37 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:37.421 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:37 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2624293025' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:37.421 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:37 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:06:37.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 87 16 97818 97802 70.2389 57.5625 0.0118545 0.0142314 2024-09-13T15:06:37.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:37.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:37.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:37.620 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:37.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:37.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:37.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:37.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:37.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:37.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:37.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:37.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:38.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:38.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:06:38.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:38.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:38.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:38.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:38.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:38.440 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:38 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:38.440 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:38 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:38.441 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:38 smithi172 ceph-mon[100696]: osdmap e403: 8 total, 8 up, 7 in 2024-09-13T15:06:38.441 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:38 smithi172 ceph-mon[100696]: pgmap v1291: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T15:06:38.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[102314]: osdmap e403: 8 total, 8 up, 7 in 2024-09-13T15:06:38.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[102314]: pgmap v1291: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T15:06:38.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 88 16 98719 98703 70.0806 56.3125 0.00718124 0.0142634 2024-09-13T15:06:38.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:06:38.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:06:38.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[107038]: osdmap e403: 8 total, 8 up, 7 in 2024-09-13T15:06:38.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:38 smithi154 ceph-mon[107038]: pgmap v1291: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-13T15:06:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:38.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:38.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:38.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:38.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:38.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:38.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:38.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:38.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:38.992 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:38.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:39.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:39.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:39.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:39.275 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:39.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:39.408 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:39.451 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:39 smithi154 ceph-mon[102314]: pgmap v1293: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-13T15:06:39.452 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:39 smithi154 ceph-mon[107038]: pgmap v1293: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-13T15:06:39.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 89 16 99599 99583 69.9112 55 0.00762206 0.0142981 2024-09-13T15:06:39.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:39 smithi172 ceph-mon[100696]: pgmap v1293: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-13T15:06:39.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:39.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:39.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:39.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:39.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:39.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:39.943 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:06:39.943 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:06:39.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.943 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:100020 9.e 9:764ff9eb:::benchmark_data_smithi154_291684_object100019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-09-13T15:06:39.943 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:39.933238+0000", 2024-09-13T15:06:39.943 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0078757310000000004, 2024-09-13T15:06:39.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079011900000000006, 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 100020 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933238+0000", 2024-09-13T15:06:39.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933238+0000", 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933236+0000", 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:39.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933275+0000", 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.858e-05 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933289+0000", 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4007e-05 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933297+0000", 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8869999999999992e-06 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933355+0000", 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8121000000000001e-05 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933507+0000", 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015219800000000001 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.933646+0000", 2024-09-13T15:06:39.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000138727 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.936647+0000", 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030013560000000002 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:100022 9.1 e403/390 9:87c942c1:::benchmark_data_smithi154_291684_object100021:head v 403'6318, mlcod=403'6316)", 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:39.938505+0000", 2024-09-13T15:06:39.948 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0026088119999999998, 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002684723, 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938505+0000", 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938505+0000", 2024-09-13T15:06:39.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938504+0000", 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938561+0000", 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7021999999999997e-05 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:39.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938571+0000", 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7750000000000004e-06 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938576+0000", 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9849999999999997e-06 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938620+0000", 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4258999999999998e-05 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.938642+0000", 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2357999999999998e-05 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:100024 9.4 e403/374 9:2ca2602b:::benchmark_data_smithi154_291684_object100023:head v 403'6364, mlcod=403'6359)", 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:39.939910+0000", 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.001203963, 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013118800000000001, 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:39.952 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.939910+0000", 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.939910+0000", 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.939909+0000", 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.939957+0000", 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7901000000000001e-05 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.939971+0000", 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4037e-05 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.939977+0000", 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9170000000000003e-06 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.940028+0000", 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0992000000000002e-05 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:39.940051+0000", 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3300999999999999e-05 2024-09-13T15:06:39.955 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:39.956 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:39.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:39.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:39.956 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T15:06:39.956 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 3 2024-09-13T15:06:39.956 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:06:39.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:39.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:39.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:40.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:40.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:40.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:40.251 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:40.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:40.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:40.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total time run: 90.0093 2024-09-13T15:06:40.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total writes made: 100458 2024-09-13T15:06:40.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Write size: 65536 2024-09-13T15:06:40.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object size: 65536 2024-09-13T15:06:40.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Bandwidth (MB/sec): 69.7553 2024-09-13T15:06:40.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Bandwidth: 40.3931 2024-09-13T15:06:40.471 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max bandwidth (MB/sec): 245.125 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min bandwidth (MB/sec): 0 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average IOPS: 1116 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev IOPS: 646.297 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max IOPS: 3922 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min IOPS: 0 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average Latency(s): 0.014331 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Latency(s): 0.0450639 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max latency(s): 2.16284 2024-09-13T15:06:40.472 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min latency(s): 0.0012263 2024-09-13T15:06:40.475 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T15:06:40.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:40.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:40.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:40.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.659 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:40.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:40.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:06:40.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:40.820 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:40.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:40.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:40.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:40.964 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:41.024 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:41.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:41.094 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:41.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:41.235 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:41.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:41.364 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:41.449 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:41 smithi172 ceph-mon[100696]: pgmap v1294: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 56 MiB/s wr, 911 op/s 2024-09-13T15:06:41.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:41.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:41 smithi154 ceph-mon[102314]: pgmap v1294: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 56 MiB/s wr, 911 op/s 2024-09-13T15:06:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:41 smithi154 ceph-mon[107038]: pgmap v1294: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 56 MiB/s wr, 911 op/s 2024-09-13T15:06:41.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:41.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-13T15:06:41.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:41.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:41.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:41.801 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:41.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:41.937 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:41.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:42.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:42.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:42.067 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:06:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:06:41] "GET /metrics HTTP/1.1" 200 36611 "" "Prometheus/2.51.0" 2024-09-13T15:06:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:42.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:42.195 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:42.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:42.322 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:42.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:42.450 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:42.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:06:42.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:42.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:42.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:42.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:42.814 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:42.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:42.935 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:42.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:43.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:43.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:43.106 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:06:43.107 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:06:43.107 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:06:43.107 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:06:43.107 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:06:43.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:43.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:43.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:43.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:43.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:43.392 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:43.392 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:43.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:43.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:43.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:43.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:43.746 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:06:43.747 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:43.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:43.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:06:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:43.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:43.796 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:06:43.796 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:43.841 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:06:43.842 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:43.882 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:06:43.882 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:43.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:43.911 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:06:43.911 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:43.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:43.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:43.948 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:06:43.948 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:43.979 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:06:43.980 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:44.010 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:06:44.010 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:44.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.050 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:44.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:44.067 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:44 smithi172 ceph-mon[100696]: pgmap v1295: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 55 MiB/s wr, 910 op/s 2024-09-13T15:06:44.067 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/249413397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:44.090 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:44.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:44.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:44.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:44 smithi154 ceph-mon[102314]: pgmap v1295: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 55 MiB/s wr, 910 op/s 2024-09-13T15:06:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/249413397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:44 smithi154 ceph-mon[107038]: pgmap v1295: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 55 MiB/s wr, 910 op/s 2024-09-13T15:06:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/249413397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:44.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:44.325 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:44.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:44.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:44.463 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:44.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:44.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:44.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:44.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:44.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:44.863 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:44.965 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:44.988 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:44.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:45.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:45.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:45.229 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:45.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:45.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:45.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:45.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:45 smithi154 ceph-mon[102314]: pgmap v1296: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 49 MiB/s wr, 914 op/s 2024-09-13T15:06:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:45 smithi154 ceph-mon[107038]: pgmap v1296: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 49 MiB/s wr, 914 op/s 2024-09-13T15:06:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:45 smithi172 ceph-mon[100696]: pgmap v1296: 89 pgs: 89 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 49 MiB/s wr, 914 op/s 2024-09-13T15:06:45.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:45.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:45.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:45.658 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:45.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:45.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:45.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:45.925 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:45.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:46.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.070 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:46.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:46.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:46.199 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:46.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:46.345 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:46.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:46.473 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:46.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:46.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:46.629 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:46.629 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:46.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:46.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:46.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:46.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:46.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:46.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:06:46.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:47.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:47.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:47.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:47.219 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:47 smithi154 ceph-mon[107038]: pgmap v1297: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 29 MiB/s wr, 920 op/s 2024-09-13T15:06:47.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:47.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:47.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:06:47.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:47.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:47 smithi154 ceph-mon[102314]: pgmap v1297: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 29 MiB/s wr, 920 op/s 2024-09-13T15:06:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:47 smithi172 ceph-mon[100696]: pgmap v1297: 89 pgs: 89 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 29 MiB/s wr, 920 op/s 2024-09-13T15:06:47.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:47.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:47.610 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:47.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:47.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:47.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:47.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:47.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:47.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-13T15:06:47.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:48.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:48.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:48.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:48.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.271 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:48.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:48.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:48.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:48.421 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:48.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:48.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:48.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:48.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:48.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:48.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:48.880 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:06:48.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:48.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:48.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:49.036 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/347540363' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:49.036 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:48 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:49.037 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/347540363' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:49.037 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:48 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:49.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:49.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:49.082 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/347540363' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:49.082 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:48 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:06:49.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:49.235 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:49.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:49.358 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:49.358 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:49.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:49.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.497 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:49.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:49.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:49.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:49.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:49.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:49.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:49.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:49.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:49.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:50.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:50.045 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:50.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:06:50.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.173 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:50.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:50.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[102314]: osdmap e404: 8 total, 8 up, 7 in 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[102314]: pgmap v1299: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 20 MiB/s wr, 923 op/s 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:50.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[107038]: osdmap e404: 8 total, 8 up, 7 in 2024-09-13T15:06:50.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:49 smithi154 ceph-mon[107038]: pgmap v1299: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 20 MiB/s wr, 923 op/s 2024-09-13T15:06:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:49 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:06:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:49 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:06:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:49 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:06:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:49 smithi172 ceph-mon[100696]: osdmap e404: 8 total, 8 up, 7 in 2024-09-13T15:06:50.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:49 smithi172 ceph-mon[100696]: pgmap v1299: 89 pgs: 89 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 102 B/s rd, 20 MiB/s wr, 923 op/s 2024-09-13T15:06:50.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:50.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:50.342 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:50.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:50.442 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:50.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:50.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:50.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:50.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:50.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:50.840 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:50.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:50.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:50.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:51.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:51.093 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:51.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:51.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:51.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:51.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:51.370 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:51.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:51 smithi154 ceph-mon[102314]: pgmap v1300: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 7.4 MiB/s wr, 907 op/s 2024-09-13T15:06:51.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:51 smithi154 ceph-mon[107038]: pgmap v1300: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 7.4 MiB/s wr, 907 op/s 2024-09-13T15:06:51.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:51 smithi172 ceph-mon[100696]: pgmap v1300: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 7.4 MiB/s wr, 907 op/s 2024-09-13T15:06:51.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:51.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:51.636 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:51.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:51.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:51.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:51.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:51.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:51.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:06:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:06:51] "GET /metrics HTTP/1.1" 200 36603 "" "Prometheus/2.51.0" 2024-09-13T15:06:52.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:52.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:52.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:52.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:52.221 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:06:52.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.349 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:52.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:52.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:52.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:52.590 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:06:52.591 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:06:52.611 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:52.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:52.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:52.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:52.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:52.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:52.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:52.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.020 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:53.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:53.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:53.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:53.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:53.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:53.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:06:53.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.428 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:53.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:53.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:53 smithi154 ceph-mon[102314]: pgmap v1301: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 2.8 MiB/s wr, 897 op/s 2024-09-13T15:06:53.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:53 smithi154 ceph-mon[107038]: pgmap v1301: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 2.8 MiB/s wr, 897 op/s 2024-09-13T15:06:53.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:53 smithi172 ceph-mon[100696]: pgmap v1301: 89 pgs: 89 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 2.8 MiB/s wr, 897 op/s 2024-09-13T15:06:53.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:53.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:53.579 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:53.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:53.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:06:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:53.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:53.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:53.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:53.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:53.973 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:53.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:54.069 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:06:54.070 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:06:54.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:54.087 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:06:54.087 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:06:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:54.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:54.118 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:06:54.118 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:06:54.151 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:06:54.151 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:06:54.185 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:06:54.185 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:06:54.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:54.215 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:06:54.215 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:06:54.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2855037287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:54.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:54.250 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2855037287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:54.250 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:06:54.250 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:06:54.251 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2855037287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:06:54.285 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:06:54.285 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:06:54.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:54.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:54.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:54.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:54.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:54.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:54.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:54.751 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:54.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:54.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:54.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:54.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:55.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:55.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:55.141 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:55.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:55.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:55.345 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:55 smithi172 ceph-mon[100696]: pgmap v1302: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-13T15:06:55.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:55.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:55.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:55.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:55.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:55 smithi154 ceph-mon[102314]: pgmap v1302: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-13T15:06:55.601 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:55 smithi154 ceph-mon[107038]: pgmap v1302: 89 pgs: 89 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-13T15:06:55.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:55.671 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:55.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:55.794 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:55.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:55.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:55.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:56.013 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:06:56.013 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:06:56.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:06:56.013 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:06:56.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.013 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213593 8.9 8:958b8aa3:::benchmark_data_smithi154_261841_object97554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.761769+0000", 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 171.24153447899999, 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.682830848, 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213593 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761769+0000", 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761769+0000", 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761768+0000", 2024-09-13T15:06:56.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761771+0000", 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.118e-06 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761778+0000", 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5479999999999998e-06 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761783+0000", 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7339999999999996e-06 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761854+0000", 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0456000000000004e-05 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.761865+0000", 2024-09-13T15:06:56.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1121000000000001e-05 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.004140+0000", 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.242275346 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.004151+0000", 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0601000000000001e-05 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.424796+0000", 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.420645438 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.424909+0000", 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011265900000000001 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3,6", 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.425010+0000", 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100466 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.443708+0000", 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018698359000000001 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.443747+0000", 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8828999999999997e-05 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444004+0000", 2024-09-13T15:06:56.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025691100000000002 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444437+0000", 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043300700000000001 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444488+0000", 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1255999999999999e-05 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444599+0000", 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111507 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213598 8.9 8:9620ca2b:::benchmark_data_smithi154_261841_object97559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.770505+0000", 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 171.232798411, 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.68277707, 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.022 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213598 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770505+0000", 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770505+0000", 2024-09-13T15:06:56.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770503+0000", 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770509+0000", 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6300000000000003e-06 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770520+0000", 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1083e-05 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770525+0000", 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.169e-06 2024-09-13T15:06:56.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770571+0000", 2024-09-13T15:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5538999999999999e-05 2024-09-13T15:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.770581+0000", 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0491e-05 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.004192+0000", 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.23361090400000001 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.004200+0000", 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3529999999999995e-06 2024-09-13T15:06:56.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.430483+0000", 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.42628244 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.430598+0000", 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115567 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3,6", 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.430716+0000", 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011807399999999999 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.443776+0000", 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013059256999999999 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444034+0000", 2024-09-13T15:06:56.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025865099999999999 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444614+0000", 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057939899999999997 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.453176+0000", 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085623569999999996 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.453214+0000", 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7459000000000002e-05 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.453282+0000", 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8177999999999997e-05 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.033 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213610 8.9 8:981d5987:::benchmark_data_smithi154_261841_object97571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.796744+0000", 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 171.206559497, 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.657161334, 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213610 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796744+0000", 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796744+0000", 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796742+0000", 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796747+0000", 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0400000000000003e-06 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796754+0000", 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8189999999999992e-06 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796759+0000", 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5569999999999997e-06 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796800+0000", 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1381000000000002e-05 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.796807+0000", 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3049999999999998e-06 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.004225+0000", 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.20741817600000001 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.004232+0000", 2024-09-13T15:06:56.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6159999999999996e-06 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.435698+0000", 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4314656690000001 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.435826+0000", 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127744 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3,6", 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.435948+0000", 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122633 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444426+0000", 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084780289999999998 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.444619+0000", 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000192685 2024-09-13T15:06:56.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.447051+0000", 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002431613 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.453816+0000", 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0067653569999999996 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.453848+0000", 2024-09-13T15:06:56.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2341000000000001e-05 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.453905+0000", 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6453999999999999e-05 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18356 9.a 9:5be79c94:::benchmark_data_smithi154_291684_object18355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.140893+0000", 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 99.862409989, 2024-09-13T15:06:56.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1469738469999999, 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18356 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.140893+0000", 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.140893+0000", 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.140892+0000", 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.140956+0000", 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3440000000000002e-05 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.140965+0000", 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8370000000000003e-06 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.140971+0000", 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.993e-06 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.142122+0000", 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011515659999999999 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.142245+0000", 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012303499999999999 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,7", 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.142382+0000", 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136904 2024-09-13T15:06:56.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.147161+0000", 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047788539999999999 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.149073+0000", 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00191219 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.157330+0000", 2024-09-13T15:06:56.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082565920000000001 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.157342+0000", 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1761e-05 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.148849+0000", 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99150758500000002 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.148867+0000", 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8301999999999999e-05 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.287791+0000", 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1389234450000001 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.287867+0000", 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6043000000000004e-05 2024-09-13T15:06:56.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18380 9.a 9:565a984c:::benchmark_data_smithi154_291684_object18379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.174077+0000", 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 99.829226481000006, 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1262367229999999, 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.050 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18380 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174077+0000", 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174077+0000", 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174075+0000", 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174133+0000", 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.753e-05 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174146+0000", 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3030999999999999e-05 2024-09-13T15:06:56.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174153+0000", 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4599999999999997e-06 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174221+0000", 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8178999999999999e-05 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.174234+0000", 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2945999999999999e-05 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.148909+0000", 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97467435099999999 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.148919+0000", 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0227000000000001e-05 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.288270+0000", 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1393508020000001 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.288361+0000", 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1209e-05 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,3", 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.288451+0000", 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9781e-05 2024-09-13T15:06:56.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.290943+0000", 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024923229999999998 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.297949+0000", 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007005687 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.299580+0000", 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001631672 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.299616+0000", 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5791999999999999e-05 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.300313+0000", 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00069726100000000002 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29662 9.7 9:e3b3739c:::benchmark_data_smithi154_291684_object29661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.150214+0000", 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.853088924999994, 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0233651260000001, 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29662 2024-09-13T15:06:56.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150214+0000", 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150214+0000", 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150212+0000", 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150246+0000", 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3698000000000002e-05 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150257+0000", 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1156e-05 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150264+0000", 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.153e-06 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.152131+0000", 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018665120000000001 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.152142+0000", 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.102e-05 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.168857+0000", 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.016715008 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.168950+0000", 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3035000000000006e-05 2024-09-13T15:06:56.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.169041+0000", 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1551999999999994e-05 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.171372+0000", 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023303669999999999 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.171405+0000", 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3361999999999997e-05 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173532+0000", 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021266800000000002 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173558+0000", 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.633e-05 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173579+0000", 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1087e-05 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29668 9.7 9:e750a330:::benchmark_data_smithi154_291684_object29667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e382)", 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.150539+0000", 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.852764191999995, 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0231497549999999, 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.067 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29668 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150539+0000", 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150539+0000", 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150538+0000", 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150584+0000", 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6309e-05 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150593+0000", 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7229999999999993e-06 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.150625+0000", 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2032000000000001e-05 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.152170+0000", 2024-09-13T15:06:56.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00154451 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.152178+0000", 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8230000000000001e-06 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.170068+0000", 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.017889652 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.170139+0000", 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1278000000000004e-05 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.170221+0000", 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1590999999999999e-05 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.172100+0000", 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018790219999999999 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173505+0000", 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014050709999999999 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173621+0000", 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011594700000000001 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173658+0000", 2024-09-13T15:06:56.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7447000000000002e-05 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173689+0000", 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0275000000000001e-05 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29674 9.7 9:ebabd04c:::benchmark_data_smithi154_291684_object29673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.157666+0000", 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.845637015999998, 2024-09-13T15:06:56.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.016128484, 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29674 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157666+0000", 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157666+0000", 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157665+0000", 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157741+0000", 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5920000000000005e-05 2024-09-13T15:06:56.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157754+0000", 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3237e-05 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157761+0000", 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7390000000000002e-06 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157812+0000", 2024-09-13T15:06:56.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0478e-05 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.157822+0000", 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0491e-05 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.170704+0000", 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0128818040000001 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.170770+0000", 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6217999999999995e-05 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.170846+0000", 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5809000000000006e-05 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173475+0000", 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026292939999999999 2024-09-13T15:06:56.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173720+0000", 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000244711 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173746+0000", 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6383000000000001e-05 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173776+0000", 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9365000000000001e-05 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173795+0000", 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8842e-05 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29715 9.7 9:e1ec0fcb:::benchmark_data_smithi154_291684_object29714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.081 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.185929+0000", 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.817373821000004, 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98796868999999998, 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29715 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.185929+0000", 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.185929+0000", 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.185929+0000", 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.186015+0000", 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6683999999999999e-05 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.186034+0000", 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8933999999999999e-05 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.186043+0000", 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8410000000000006e-06 2024-09-13T15:06:56.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.186106+0000", 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2662000000000004e-05 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.186117+0000", 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0924000000000001e-05 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.171430+0000", 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98531307099999998 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.171499+0000", 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9047999999999993e-05 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.171574+0000", 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5667999999999994e-05 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173589+0000", 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020147889999999999 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173825+0000", 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023562999999999999 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173854+0000", 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9352999999999998e-05 2024-09-13T15:06:56.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173878+0000", 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3906000000000001e-05 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173898+0000", 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9925e-05 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29739 9.7 9:e934c2e5:::benchmark_data_smithi154_291684_object29738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.201294+0000", 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.802008607999994, 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97522979700000001, 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29739 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.089 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201294+0000", 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201294+0000", 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201294+0000", 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201317+0000", 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3142e-05 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201324+0000", 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4379999999999998e-06 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201330+0000", 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7130000000000002e-06 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.202961+0000", 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016308399999999999 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.202971+0000", 2024-09-13T15:06:56.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0054000000000001e-05 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.172126+0000", 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96915520799999999 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.172218+0000", 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1708999999999999e-05 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.172300+0000", 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2509000000000006e-05 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173596+0000", 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001295956 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173935+0000", 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033814799999999998 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.176472+0000", 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025377289999999999 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.176502+0000", 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0188999999999998e-05 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.176524+0000", 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1803000000000001e-05 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.095 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29741 9.7 9:e3e15612:::benchmark_data_smithi154_291684_object29740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.201508+0000", 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.801795454000001, 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.979228196, 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29741 2024-09-13T15:06:56.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201508+0000", 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201508+0000", 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201507+0000", 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201561+0000", 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4450000000000002e-05 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201572+0000", 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0392e-05 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.201578+0000", 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.297e-06 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203080+0000", 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001502102 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203090+0000", 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7920000000000004e-06 2024-09-13T15:06:56.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.172808+0000", 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.969717787 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.172884+0000", 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6155000000000005e-05 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.172986+0000", 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102243 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173907+0000", 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092062399999999999 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177828+0000", 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039211990000000002 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180669+0000", 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028413169999999999 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180714+0000", 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4781999999999999e-05 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180736+0000", 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1781999999999999e-05 2024-09-13T15:06:56.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29744 9.7 9:e68dc3e7:::benchmark_data_smithi154_291684_object29743:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.203028+0000", 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.800275013999993, 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97784523499999998, 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29744 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203028+0000", 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203028+0000", 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203027+0000", 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203117+0000", 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0012000000000005e-05 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203124+0000", 2024-09-13T15:06:56.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8029999999999999e-06 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203128+0000", 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9689999999999996e-06 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203161+0000", 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2888000000000001e-05 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.203170+0000", 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6030000000000005e-06 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.173957+0000", 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97078753799999995 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.174021+0000", 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3623999999999999e-05 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.174096+0000", 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4653000000000005e-05 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177151+0000", 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030558780000000002 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177858+0000", 2024-09-13T15:06:56.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070686499999999999 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180786+0000", 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002927444 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180834+0000", 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8285000000000003e-05 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180873+0000", 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9310999999999997e-05 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29757 9.7 9:e7c32bed:::benchmark_data_smithi154_291684_object29756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.211081+0000", 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.792222061999993, 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97531090600000003, 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.110 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29757 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211081+0000", 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211081+0000", 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211080+0000", 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211139+0000", 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9837999999999999e-05 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211150+0000", 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1102e-05 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211156+0000", 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1329999999999998e-06 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211200+0000", 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4403999999999997e-05 2024-09-13T15:06:56.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.211209+0000", 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0019999999999995e-06 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.174758+0000", 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96354929099999997 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.174862+0000", 2024-09-13T15:06:56.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104178 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.174970+0000", 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107742 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177797+0000", 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028265249999999999 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180631+0000", 2024-09-13T15:06:56.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002834086 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186329+0000", 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056979659999999996 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186365+0000", 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6239000000000003e-05 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186392+0000", 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6863999999999999e-05 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29782 9.7 9:e2205ca5:::benchmark_data_smithi154_291684_object29781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.226208+0000", 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.777095204000005, 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96034244599999996, 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.117 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29782 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226208+0000", 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226208+0000", 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226207+0000", 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226265+0000", 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8545000000000003e-05 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226274+0000", 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0620000000000001e-06 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226278+0000", 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1309999999999999e-06 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226350+0000", 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2305000000000007e-05 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.226363+0000", 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2913e-05 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.175743+0000", 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94938019799999995 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.175865+0000", 2024-09-13T15:06:56.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121666 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.175955+0000", 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0127e-05 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177863+0000", 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001908477 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.182395+0000", 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004532009 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186439+0000", 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0040432760000000002 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186510+0000", 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1093999999999994e-05 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186550+0000", 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0608999999999997e-05 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29784 9.7 9:ec6c8611:::benchmark_data_smithi154_291684_object29783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.227076+0000", 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.776227539999994, 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.959591572, 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.124 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29784 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227076+0000", 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227076+0000", 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227075+0000", 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227138+0000", 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3077000000000006e-05 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227149+0000", 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.114e-05 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227156+0000", 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8299999999999998e-06 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227211+0000", 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4968e-05 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.227222+0000", 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1572e-05 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.176555+0000", 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94933289799999998 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.176625+0000", 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9665999999999993e-05 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.176715+0000", 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0651999999999999e-05 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177867+0000", 2024-09-13T15:06:56.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011519259999999999 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.182427+0000", 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045592130000000003 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186617+0000", 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041905199999999997 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186645+0000", 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8167999999999999e-05 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186667+0000", 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1861e-05 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29794 9.7 9:e636155a:::benchmark_data_smithi154_291684_object29793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.131 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.236930+0000", 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.766373118000004, 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94988450499999999, 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29794 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.236930+0000", 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.236930+0000", 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.236928+0000", 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.236973+0000", 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4870999999999998e-05 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.236986+0000", 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2394e-05 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.236994+0000", 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7250000000000003e-06 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.237054+0000", 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0142000000000001e-05 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.237069+0000", 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4961e-05 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177168+0000", 2024-09-13T15:06:56.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.94009853300000001 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177243+0000", 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4739000000000004e-05 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177327+0000", 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4047000000000004e-05 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180641+0000", 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003313884 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.182461+0000", 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001820459 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186736+0000", 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042748100000000004 2024-09-13T15:06:56.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186781+0000", 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5513999999999998e-05 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186814+0000", 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3011e-05 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29819 9.7 9:e4c669a1:::benchmark_data_smithi154_291684_object29818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.255963+0000", 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.747340210999994, 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.930924847, 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29819 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.255963+0000", 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.255963+0000", 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.255962+0000", 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:06:56.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.256015+0000", 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3341000000000003e-05 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.256024+0000", 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9169999999999995e-06 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.256030+0000", 2024-09-13T15:06:56.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4859999999999997e-06 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.256069+0000", 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9072999999999997e-05 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.256078+0000", 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6200000000000005e-06 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177888+0000", 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92181062000000002 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.177962+0000", 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3495e-05 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.178039+0000", 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6925e-05 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180647+0000", 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026081979999999999 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.184983+0000", 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043358880000000004 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186844+0000", 2024-09-13T15:06:56.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001861068 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186868+0000", 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4032e-05 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.186888+0000", 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9890000000000001e-05 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29820 9.7 9:e17f4e2b:::benchmark_data_smithi154_291684_object29819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.257158+0000", 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.746145510000005, 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.93192671400000004, 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29820 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257158+0000", 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257158+0000", 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257156+0000", 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257214+0000", 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7287999999999998e-05 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257223+0000", 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4930000000000004e-06 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257230+0000", 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7719999999999997e-06 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257283+0000", 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3156999999999999e-05 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.257295+0000", 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2310000000000001e-05 2024-09-13T15:06:56.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.178551+0000", 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.92125553500000001 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.178681+0000", 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013014100000000001 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.178816+0000", 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013467000000000001 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180886+0000", 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020697179999999999 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.185029+0000", 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041437940000000001 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.189024+0000", 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039942780000000004 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.189052+0000", 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7966e-05 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.189084+0000", 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2688999999999998e-05 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29843 9.7 9:ee5845fc:::benchmark_data_smithi154_291684_object29842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.332651+0000", 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.670651784, 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.85653082199999997, 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29843 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332651+0000", 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332651+0000", 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332651+0000", 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332759+0000", 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010825800000000001 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332768+0000", 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0669999999999998e-06 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332773+0000", 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.853e-06 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332823+0000", 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0510000000000003e-05 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.332837+0000", 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3665999999999999e-05 2024-09-13T15:06:56.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.179373+0000", 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84653608499999999 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.179479+0000", 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010569399999999999 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.179684+0000", 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020514599999999999 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.182350+0000", 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002665878 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.185073+0000", 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027236199999999999 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.189124+0000", 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004050395 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.189162+0000", 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8704000000000001e-05 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.189182+0000", 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9695e-05 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.159 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:29845 9.7 9:ee89520c:::benchmark_data_smithi154_291684_object29844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:26.335909+0000", 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.667393906000001, 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.85639443199999998, 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:06:56.160 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29845 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.335909+0000", 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.335909+0000", 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.335908+0000", 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.335974+0000", 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5591000000000003e-05 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.335985+0000", 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1639e-05 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.335992+0000", 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.122e-06 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.336026+0000", 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4649e-05 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:26.336034+0000", 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0919999999999998e-06 2024-09-13T15:06:56.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180899+0000", 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84486489799999998 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.180970+0000", 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0695e-05 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:06:56.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.181044+0000", 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4136999999999998e-05 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.182431+0000", 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00138704 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.185114+0000", 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026828709999999999 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.192258+0000", 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071437219999999999 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.192284+0000", 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6361000000000001e-05 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:27.192304+0000", 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9587999999999999e-05 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.166 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:56.167 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:06:56.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:56.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:56.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:56.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:56.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:56.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:56.606 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:56.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:56.746 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:56.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:56.871 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:56.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:56.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:06:57.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:06:57.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:57.120 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:06:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.135 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:57.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:57.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:57.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:57.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:57.393 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:06:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:57.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:57.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:57 smithi154 ceph-mon[102314]: pgmap v1303: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-13T15:06:57.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:57 smithi154 ceph-mon[107038]: pgmap v1303: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-13T15:06:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:57 smithi172 ceph-mon[100696]: pgmap v1303: 89 pgs: 89 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-13T15:06:57.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:57.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:57.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:57.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:57.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:57.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:57.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:57.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:57.920 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:57.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:58.022 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:58.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:58.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:58.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:58.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:58.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:58.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:06:58.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:06:58.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:58.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:58.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.732 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:58.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:58.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:06:58.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:06:58.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:58.982 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:58.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:59.083 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.111 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:59.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:59.177 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:59.177 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:06:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:59.195 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:116878 9.b e404/374 9:d2ce3c51:::benchmark_data_smithi154_291684_object16417:head v 404'7239, mlcod=404'7237)", 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:59.174460+0000", 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0011681230000000001, 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001197416, 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:06:59.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174460+0000", 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174460+0000", 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174459+0000", 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174468+0000", 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9209999999999998e-06 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174483+0000", 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5515e-05 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174490+0000", 2024-09-13T15:06:59.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2710000000000003e-06 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174533+0000", 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2330999999999998e-05 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:59.174548+0000", 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5688999999999998e-05 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:59.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:06:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T15:06:59.200 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-09-13T15:06:59.200 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:06:59.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:06:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:06:59.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:06:59.373 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:06:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:06:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:59.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:06:59.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:06:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:06:59.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:06:59.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:06:59.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:59.626 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:06:59.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:06:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.773 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:06:59.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:06:59.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:06:59.901 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:06:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:06:59.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:00.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:00.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:00.165 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:00 smithi154 ceph-mon[102314]: pgmap v1304: 89 pgs: 89 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 842 op/s 2024-09-13T15:07:00.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:00 smithi154 ceph-mon[107038]: pgmap v1304: 89 pgs: 89 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 842 op/s 2024-09-13T15:07:00.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:00 smithi172 ceph-mon[100696]: pgmap v1304: 89 pgs: 89 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 842 op/s 2024-09-13T15:07:00.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:00.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:07:00.365 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:00.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:00.444 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:00.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:00.580 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.650 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:00.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:00.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:00.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:00.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:00.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:00.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:00.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:01.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:01.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:01.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:01.246 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:01.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:01.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:01 smithi154 ceph-mon[102314]: pgmap v1305: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-13T15:07:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:01 smithi154 ceph-mon[107038]: pgmap v1305: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-13T15:07:01.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:01 smithi172 ceph-mon[100696]: pgmap v1305: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-13T15:07:01.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:01.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:01.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:01.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:01.796 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:01.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:01.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:07:01.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:01.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:01.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:02.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:02.073 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:02.078 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:07:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:07:01] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-13T15:07:02.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:02.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:02.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:02.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:02.415 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:02.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:02.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:02.528 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:07:02.529 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:07:02.529 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:07:02.529 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:07:02.529 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:07:02.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:02.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:02.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:02.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:02.804 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:07:02.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:02.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:02.928 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:02.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:03.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:03.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:03.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:03.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:03.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:03.212 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2949680762' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:03.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2949680762' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:03.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2949680762' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:07:03.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:07:03.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:03.331 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:03.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.467 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:03.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:03.499 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:03.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:03.591 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:03.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:03.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:03.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:03.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:03.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:03.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:03 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:03.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2949680762' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:07:03.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:03 smithi172 ceph-mon[100696]: osdmap e405: 8 total, 8 up, 7 in 2024-09-13T15:07:03.959 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:03 smithi172 ceph-mon[100696]: pgmap v1307: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 858 op/s 2024-09-13T15:07:03.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:03.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:04.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:04.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:04.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:04.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2949680762' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:07:04.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[102314]: osdmap e405: 8 total, 8 up, 7 in 2024-09-13T15:07:04.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[102314]: pgmap v1307: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 858 op/s 2024-09-13T15:07:04.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:04.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2949680762' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:07:04.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[107038]: osdmap e405: 8 total, 8 up, 7 in 2024-09-13T15:07:04.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:03 smithi154 ceph-mon[107038]: pgmap v1307: 89 pgs: 89 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 858 op/s 2024-09-13T15:07:04.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:04.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:04.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:04.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:04.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:04.518 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:04.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:04.655 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:04.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:04.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:04.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:04.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:04.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:05.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:05.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:05.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:05.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:05.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:05.330 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:05 smithi154 ceph-mon[102314]: pgmap v1308: 89 pgs: 89 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 848 op/s 2024-09-13T15:07:05.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:05 smithi154 ceph-mon[107038]: pgmap v1308: 89 pgs: 89 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 848 op/s 2024-09-13T15:07:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.332 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:05.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:05.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:07:05.363 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:07:05.364 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-13T15:07:05.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:07:05.386 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:05 smithi172 ceph-mon[100696]: pgmap v1308: 89 pgs: 89 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 848 op/s 2024-09-13T15:07:05.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:05.486 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150525 7.c 7:3a1defd9:::benchmark_data_smithi154_225924_object40261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.271468+0000", 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 574.24372947999996, 2024-09-13T15:07:05.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.038483329, 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150525 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271468+0000", 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271468+0000", 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271467+0000", 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271474+0000", 2024-09-13T15:07:05.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.374e-06 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271489+0000", 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4942e-05 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271497+0000", 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4499999999999998e-06 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271547+0000", 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9669000000000002e-05 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271709+0000", 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016256800000000001 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2,3", 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.271846+0000", 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013647600000000001 2024-09-13T15:07:05.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.276339+0000", 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044934969999999999 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.278783+0000", 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024441889999999998 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290520+0000", 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011737081 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.290532+0000", 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1182999999999999e-05 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.309898+0000", 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.019366231 2024-09-13T15:07:05.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.309952+0000", 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4027999999999998e-05 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:150538 7.c 7:32d10010:::benchmark_data_smithi154_225924_object40274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:57:31.311705+0000", 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 574.20349325100005, 2024-09-13T15:07:05.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0082904450000001, 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 150538 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311705+0000", 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311705+0000", 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311703+0000", 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311710+0000", 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7000000000000002e-06 2024-09-13T15:07:05.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311721+0000", 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1728e-05 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311727+0000", 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4070000000000001e-06 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311778+0000", 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1570999999999998e-05 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:31.311789+0000", 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0328e-05 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310250+0000", 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99846129900000002 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310383+0000", 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013281299999999999 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3", 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.310500+0000", 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117446 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.318391+0000", 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078911080000000008 2024-09-13T15:07:05.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319857+0000", 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014651569999999999 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319909+0000", 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2748999999999999e-05 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:57:32.319995+0000", 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5818999999999999e-05 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198815 7.d 7:bb1741ae:::benchmark_data_smithi154_225924_object88551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:07:05.532 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 515.090442478, 2024-09-13T15:07:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.080673513000000002, 2024-09-13T15:07:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198815 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424755+0000", 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424758+0000", 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8930000000000001e-06 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424765+0000", 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8210000000000004e-06 2024-09-13T15:07:05.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.424769+0000", 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7369999999999997e-06 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428701+0000", 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039317220000000003 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428826+0000", 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012472699999999999 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.428948+0000", 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012246300000000001 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.432869+0000", 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003920626 2024-09-13T15:07:05.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.466811+0000", 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033941885999999998 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505293+0000", 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038482159000000002 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505389+0000", 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6051000000000006e-05 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.505429+0000", 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198852 7.7 7:e4156631:::benchmark_data_smithi154_225924_object88588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.510600+0000", 2024-09-13T15:07:05.539 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 515.00459834699996, 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.098632827000000006, 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198852 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510600+0000", 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510600+0000", 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510599+0000", 2024-09-13T15:07:05.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510604+0000", 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2839999999999997e-06 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510615+0000", 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1566000000000001e-05 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510622+0000", 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.793e-06 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510665+0000", 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2363000000000001e-05 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510782+0000", 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117868 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.510914+0000", 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013160799999999999 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.515165+0000", 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042514379999999997 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.518048+0000", 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002882201 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609141+0000", 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.091092993999999997 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609191+0000", 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0241999999999998e-05 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.609232+0000", 2024-09-13T15:07:05.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1523999999999997e-05 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198899 7.7 7:e8acc7c2:::benchmark_data_smithi154_225924_object88635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.610904+0000", 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 514.90429409000001, 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.078514312000000003, 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.546 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198899 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610904+0000", 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610904+0000", 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610897+0000", 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999928 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610911+0000", 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4121000000000001e-05 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610925+0000", 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4239e-05 2024-09-13T15:07:05.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610932+0000", 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2400000000000001e-06 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.610979+0000", 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7040999999999999e-05 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.611109+0000", 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013000300000000001 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.611226+0000", 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011720800000000001 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.615610+0000", 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043831440000000003 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.617396+0000", 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017859589999999999 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689331+0000", 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.071935456999999994 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689380+0000", 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8606999999999997e-05 2024-09-13T15:07:05.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.689418+0000", 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8600999999999999e-05 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198901 7.c 7:30e20b28:::benchmark_data_smithi154_225924_object88637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.614143+0000", 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 514.90105515000005, 2024-09-13T15:07:05.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.088790589000000003, 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198901 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614143+0000", 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614143+0000", 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614141+0000", 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614147+0000", 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9830000000000001e-06 2024-09-13T15:07:05.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614159+0000", 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.168e-05 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614166+0000", 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6159999999999998e-06 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614211+0000", 2024-09-13T15:07:05.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5210000000000003e-05 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614339+0000", 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012781500000000001 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.614459+0000", 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120282 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.618850+0000", 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043914069999999999 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.619032+0000", 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018138900000000001 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702829+0000", 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.083797778000000003 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702886+0000", 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6388999999999998e-05 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.702933+0000", 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7550999999999999e-05 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.558 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198956 7.c 7:37128aaa:::benchmark_data_smithi154_225924_object88692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.743414+0000", 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 514.77178345599998, 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.105403712, 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198956 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743414+0000", 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743414+0000", 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743413+0000", 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743419+0000", 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1510000000000001e-06 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743432+0000", 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2309e-05 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.743440+0000", 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5660000000000007e-06 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750304+0000", 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068643200000000001 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750419+0000", 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114232 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 3,7", 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750550+0000", 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130949 2024-09-13T15:07:05.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.754768+0000", 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042181919999999999 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.754801+0000", 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3374999999999998e-05 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848517+0000", 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.093715883999999999 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848778+0000", 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026113500000000001 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.848818+0000", 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9990000000000002e-05 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198960 7.d 7:b59b0bff:::benchmark_data_smithi154_225924_object88696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.748323+0000", 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 514.76687499000002, 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.084968948000000002, 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198960 2024-09-13T15:07:05.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748323+0000", 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748323+0000", 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748322+0000", 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748326+0000", 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7859999999999996e-06 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748334+0000", 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9619999999999993e-06 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748339+0000", 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9570000000000001e-06 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748557+0000", 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000217216 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748697+0000", 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014085599999999999 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.748819+0000", 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012206600000000001 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758177+0000", 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093571000000000001 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.765093+0000", 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006916447 2024-09-13T15:07:05.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833219+0000", 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.068125854999999999 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833252+0000", 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2867999999999998e-05 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833292+0000", 2024-09-13T15:07:05.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0201999999999999e-05 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198961 7.d 7:be6e4db7:::benchmark_data_smithi154_225924_object88697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.750495+0000", 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 514.76470297499998, 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.082969548000000004, 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198961 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750495+0000", 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750495+0000", 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750494+0000", 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750500+0000", 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2380000000000002e-06 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750513+0000", 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2578999999999999e-05 2024-09-13T15:07:05.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.750520+0000", 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9500000000000004e-06 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753665+0000", 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031451449999999998 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753794+0000", 2024-09-13T15:07:05.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012921499999999999 2024-09-13T15:07:05.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:07:05.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.753911+0000", 2024-09-13T15:07:05.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000117296 2024-09-13T15:07:05.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758467+0000", 2024-09-13T15:07:05.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045562709999999998 2024-09-13T15:07:05.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.765159+0000", 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0066919529999999996 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833387+0000", 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.068227963000000003 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833429+0000", 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1740999999999999e-05 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.833465+0000", 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5429999999999998e-05 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45383.0:198966 7.7 7:e3c5df65:::benchmark_data_smithi154_225924_object88702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:58:30.758920+0000", 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 514.75627760400005, 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.093099978999999999, 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.578 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45383", 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/1802242582", 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 198966 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758920+0000", 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758920+0000", 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758919+0000", 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758927+0000", 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2209999999999997e-06 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758939+0000", 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1528999999999999e-05 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758947+0000", 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6519999999999995e-06 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.758999+0000", 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1884e-05 2024-09-13T15:07:05.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.759133+0000", 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000134346 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.759259+0000", 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000125463 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.762880+0000", 2024-09-13T15:07:05.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036212050000000002 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.836898+0000", 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.074017828999999993 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.851938+0000", 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015040359 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.851994+0000", 2024-09-13T15:07:05.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5305e-05 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:58:30.852020+0000", 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6689000000000001e-05 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:207211 7.c e323/307 7:39464d69:::benchmark_data_smithi154_225924_object96947:head v 323'12711, mlcod=323'12708)", 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:44.382153+0000", 2024-09-13T15:07:05.584 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 441.13304494499999, 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020006836, 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382153+0000", 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382153+0000", 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382152+0000", 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382163+0000", 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0951999999999999e-05 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382182+0000", 2024-09-13T15:07:05.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9122000000000001e-05 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382189+0000", 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5240000000000003e-06 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382256+0000", 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6303999999999994e-05 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.382287+0000", 2024-09-13T15:07:05.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1106999999999999e-05 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.402061+0000", 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019774040999999999 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:44.402160+0000", 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8954000000000002e-05 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45383.0:212080 7.b e325/307 7:d9f119b8:::benchmark_data_smithi154_225924_object101816:head v 325'13046, mlcod=325'13043)", 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T14:59:50.067041+0000", 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 435.44815646900003, 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016940786999999999, 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067041+0000", 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067041+0000", 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067040+0000", 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067048+0000", 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.571e-06 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067063+0000", 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5033e-05 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067070+0000", 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4719999999999998e-06 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067132+0000", 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1681999999999997e-05 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.067166+0000", 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4097000000000002e-05 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.083935+0000", 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016768314999999999 2024-09-13T15:07:05.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T14:59:50.083982+0000", 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7728000000000001e-05 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.35787.0:229140 8.5 e370/368 8:a586db2b:::benchmark_data_smithi154_261841_object113101:head v 370'14272, mlcod=370'14269)", 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:29.322821+0000", 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 156.192377096, 2024-09-13T15:07:05.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019172475000000001, 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.322821+0000", 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.322821+0000", 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.322820+0000", 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.322825+0000", 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9019999999999998e-06 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.322833+0000", 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3749999999999997e-06 2024-09-13T15:07:05.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.322837+0000", 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2860000000000001e-06 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.325830+0000", 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029930930000000001 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.325852+0000", 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1933000000000002e-05 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.341919+0000", 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016067201 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.341993+0000", 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4064000000000001e-05 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.35787.0:229142 8.5 e370/368 8:a9d05259:::benchmark_data_smithi154_261841_object113103:head v 370'14273, mlcod=370'14270)", 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:29.329126+0000", 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 156.18607237000001, 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019691134999999999, 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.329126+0000", 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.329126+0000", 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.329125+0000", 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.329132+0000", 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4619999999999999e-06 2024-09-13T15:07:05.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.329145+0000", 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3061000000000001e-05 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.329152+0000", 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5799999999999997e-06 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.338558+0000", 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094060419999999999 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.338579+0000", 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506e-05 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.348782+0000", 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010202258000000001 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:29.348817+0000", 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4975999999999999e-05 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:21999 9.0 e382/374 9:0fb08482:::benchmark_data_smithi154_291684_object21998:head v 382'1364, mlcod=382'1361)", 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:20.468296+0000", 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.04690214199999, 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020825903999999999, 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.602 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.468296+0000", 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.468296+0000", 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.468294+0000", 2024-09-13T15:07:05.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.487876+0000", 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019581086000000001 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.487889+0000", 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3044000000000001e-05 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.487896+0000", 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5789999999999998e-06 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.488031+0000", 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000134738 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.488047+0000", 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6560999999999999e-05 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.489091+0000", 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043541 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.489122+0000", 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0688000000000002e-05 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.606 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:22008 9.0 e382/374 9:0c79b753:::benchmark_data_smithi154_291684_object22007:head v 382'1365, mlcod=382'1362)", 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:20.488684+0000", 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.02651356299999, 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020394804999999998, 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.488684+0000", 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.488684+0000", 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.488684+0000", 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.507186+0000", 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018501933000000002 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.507212+0000", 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6389000000000001e-05 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.507219+0000", 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.985e-06 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.507269+0000", 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9969999999999998e-05 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.507288+0000", 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9491999999999999e-05 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.509048+0000", 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017590220000000001 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:20.509079+0000", 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1674999999999997e-05 2024-09-13T15:07:05.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:34765 9.b e386/374 9:d34176a6:::benchmark_data_smithi154_291684_object34764:head v 386'2115, mlcod=386'2113)", 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:31.952219+0000", 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.562978826000005, 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018158339999999999, 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.952219+0000", 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.952219+0000", 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.952218+0000", 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.968685+0000", 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016466963000000001 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.968731+0000", 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5924999999999998e-05 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.968738+0000", 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9099999999999999e-06 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.968865+0000", 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012689999999999999 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.968890+0000", 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4876999999999999e-05 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.970313+0000", 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001423625 2024-09-13T15:07:05.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:31.970377+0000", 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4182000000000003e-05 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:63070 9.0 e392/374 9:09061f55:::benchmark_data_smithi154_291684_object63069:head v 392'3919, mlcod=392'3916)", 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:58.674089+0000", 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.841108661000007, 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022883791000000001, 2024-09-13T15:07:05.615 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.674089+0000", 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.674089+0000", 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.674089+0000", 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.693173+0000", 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019084533000000001 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.693189+0000", 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5639000000000001e-05 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.693196+0000", 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3030000000000002e-06 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.693237+0000", 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0614000000000001e-05 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.693265+0000", 2024-09-13T15:07:05.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8323e-05 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.696938+0000", 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036726020000000001 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:58.696973+0000", 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5204999999999999e-05 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:96846 9.0 e402/374 9:0e6f87a6:::benchmark_data_smithi154_291684_object96845:head v 402'6023, mlcod=402'6020)", 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:36.398788+0000", 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.116409535999999, 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025717956, 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.398788+0000", 2024-09-13T15:07:05.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.398788+0000", 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.398787+0000", 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.418706+0000", 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019918480999999998 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.418725+0000", 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9295000000000001e-05 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.418734+0000", 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8540000000000003e-06 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.418873+0000", 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013868299999999999 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.418899+0000", 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6662e-05 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.424466+0000", 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055663199999999996 2024-09-13T15:07:05.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:36.424506+0000", 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0605999999999997e-05 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:100109 9.b e403/374 9:db106a5a:::benchmark_data_smithi154_291684_object100108:head v 403'6201, mlcod=403'6199)", 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:40.050771+0000", 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.464426517, 2024-09-13T15:07:05.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022575636, 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.050771+0000", 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.050771+0000", 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.050771+0000", 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.069616+0000", 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01884541 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.069632+0000", 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5724000000000001e-05 2024-09-13T15:07:05.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.069639+0000", 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3549999999999999e-06 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.069724+0000", 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5092000000000005e-05 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.069743+0000", 2024-09-13T15:07:05.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8856999999999999e-05 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.073274+0000", 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035307120000000001 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:40.073347+0000", 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3306999999999995e-05 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:07:05.629 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:07:05.629 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:07:05.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:05.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:05.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:05.779 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:05.881 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:05.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:05.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:06.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:06.086 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:06.158 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:06.160 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:07:06.160 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:07:06.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:06.202 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:07:06.202 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:07:06.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:06.235 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:07:06.235 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:07:06.264 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:07:06.265 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:07:06.298 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:07:06.299 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:07:06.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:06.335 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:06.336 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:07:06.336 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:07:06.368 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:07:06.368 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:07:06.404 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:07:06.405 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:07:06.430 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1795190904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:06.430 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1795190904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:06.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:06.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1795190904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:06.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:06.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:06.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:07:06.720 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:06.771 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:06.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:06.905 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:06.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:06.949 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:07.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:07.045 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:07.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:07.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:07.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:07.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:07.332 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:07 smithi172 ceph-mon[100696]: pgmap v1309: 89 pgs: 89 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 820 op/s 2024-09-13T15:07:07.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:07.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:07.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:07 smithi154 ceph-mon[102314]: pgmap v1309: 89 pgs: 89 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 820 op/s 2024-09-13T15:07:07.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:07 smithi154 ceph-mon[107038]: pgmap v1309: 89 pgs: 89 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 820 op/s 2024-09-13T15:07:07.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:07.577 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:07.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:07:07.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:07.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:07.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:07.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:07.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:07.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:08.018 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:08.119 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:08.143 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:08.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:08.271 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:08.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:08.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:08.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:08.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:08.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:08.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:08.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:08.738 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:08.739 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:08.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:08.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:08.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:08.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:08.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:09.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:09.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:09.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:09.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:09.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:09.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:09.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:09.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:09 smithi154 ceph-mon[102314]: pgmap v1310: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 822 op/s 2024-09-13T15:07:09.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:09 smithi154 ceph-mon[107038]: pgmap v1310: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 822 op/s 2024-09-13T15:07:09.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:09 smithi172 ceph-mon[100696]: pgmap v1310: 89 pgs: 89 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 822 op/s 2024-09-13T15:07:09.506 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:09.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:09.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:09.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:09.759 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:09.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:09.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:09.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:09.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:07:09.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:10.021 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:10.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:10.155 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:10.192 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:07:10.197 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:07:10.205 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:10.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:10.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:10.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:10.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:10.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:10.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:10.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:10.675 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:10.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:10.801 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:10.903 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:10.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:10.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:11.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:11.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:11.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:11.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:07:11.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:11.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:11.327 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:11.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:11.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:11 smithi172 ceph-mon[100696]: from='client.36153 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:07:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:11 smithi172 ceph-mon[100696]: pgmap v1311: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:07:11.554 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:11 smithi154 ceph-mon[102314]: from='client.36153 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:07:11.554 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:11 smithi154 ceph-mon[102314]: pgmap v1311: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:07:11.555 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:11 smithi154 ceph-mon[107038]: from='client.36153 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:07:11.555 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:11 smithi154 ceph-mon[107038]: pgmap v1311: 89 pgs: 89 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:07:11.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:11.608 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:11.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:11.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:11.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:11.902 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1310,"stamp":"2024-09-13T15:07:09.122498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5091891197,"num_objects":77652,"num_object_clones":0,"num_object_copies":232956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77652,"num_whiteouts":0,"num_read":6320,"num_read_kb":13777,"num_write":124417,"num_write_kb":6444354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10505,"num_bytes_recovered":674662783,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114576,"ondisk_log_size":114576,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18801160,"kb_used_data":14971024,"kb_used_omap":39,"kb_used_meta":3829976,"kb_avail":637414904,"statfs":{"total":671965249536,"available":652712861696,"internally_reserved":0,"allocated":15330328576,"data_stored":15314864877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40279,"internal_metadata":3921896105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,4,5,11,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-539492352,"num_objects":-8232,"num_object_clones":0,"num_object_copies":-24696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003254"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":64,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.830781+0000","last_change":"2024-09-13T15:05:17.171354+0000","last_active":"2024-09-13T15:07:02.830781+0000","last_peered":"2024-09-13T15:07:02.830781+0000","last_clean":"2024-09-13T15:07:02.830781+0000","last_became_active":"2024-09-13T15:05:17.169948+0000","last_became_peered":"2024-09-13T15:05:17.169948+0000","last_unstale":"2024-09-13T15:07:02.830781+0000","last_undegraded":"2024-09-13T15:07:02.830781+0000","last_fullsized":"2024-09-13T15:07:02.830781+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:45:38.476807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":201,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873108+0000","last_change":"2024-09-13T15:05:17.168763+0000","last_active":"2024-09-13T15:06:48.873108+0000","last_peered":"2024-09-13T15:06:48.873108+0000","last_clean":"2024-09-13T15:06:48.873108+0000","last_became_active":"2024-09-13T15:05:17.160223+0000","last_became_peered":"2024-09-13T15:05:17.160223+0000","last_unstale":"2024-09-13T15:06:48.873108+0000","last_undegraded":"2024-09-13T15:06:48.873108+0000","last_fullsized":"2024-09-13T15:06:48.873108+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:44.201545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":264,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893739+0000","last_change":"2024-09-13T15:05:17.161668+0000","last_active":"2024-09-13T15:06:48.893739+0000","last_peered":"2024-09-13T15:06:48.893739+0000","last_clean":"2024-09-13T15:06:48.893739+0000","last_became_active":"2024-09-13T15:05:17.161258+0000","last_became_peered":"2024-09-13T15:05:17.161258+0000","last_unstale":"2024-09-13T15:06:48.893739+0000","last_undegraded":"2024-09-13T15:06:48.893739+0000","last_fullsized":"2024-09-13T15:06:48.893739+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:21.646012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":641,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.835995+0000","last_change":"2024-09-13T15:05:17.169489+0000","last_active":"2024-09-13T15:07:02.835995+0000","last_peered":"2024-09-13T15:07:02.835995+0000","last_clean":"2024-09-13T15:07:02.835995+0000","last_became_active":"2024-09-13T15:05:17.169178+0000","last_became_peered":"2024-09-13T15:05:17.169178+0000","last_unstale":"2024-09-13T15:07:02.835995+0000","last_undegraded":"2024-09-13T15:07:02.835995+0000","last_fullsized":"2024-09-13T15:07:02.835995+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:32.374777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":645,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.836819+0000","last_change":"2024-09-13T15:05:17.167709+0000","last_active":"2024-09-13T15:07:02.836819+0000","last_peered":"2024-09-13T15:07:02.836819+0000","last_clean":"2024-09-13T15:07:02.836819+0000","last_became_active":"2024-09-13T15:05:17.166796+0000","last_became_peered":"2024-09-13T15:05:17.166796+0000","last_unstale":"2024-09-13T15:07:02.836819+0000","last_undegraded":"2024-09-13T15:07:02.836819+0000","last_fullsized":"2024-09-13T15:07:02.836819+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:28.384317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":259,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.872806+0000","last_change":"2024-09-13T15:05:17.167443+0000","last_active":"2024-09-13T15:06:48.872806+0000","last_peered":"2024-09-13T15:06:48.872806+0000","last_clean":"2024-09-13T15:06:48.872806+0000","last_became_active":"2024-09-13T15:05:17.167117+0000","last_became_peered":"2024-09-13T15:05:17.167117+0000","last_unstale":"2024-09-13T15:06:48.872806+0000","last_undegraded":"2024-09-13T15:06:48.872806+0000","last_fullsized":"2024-09-13T15:06:48.872806+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:19.536344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":145,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.858078+0000","last_change":"2024-09-13T15:05:17.173494+0000","last_active":"2024-09-13T15:07:02.858078+0000","last_peered":"2024-09-13T15:07:02.858078+0000","last_clean":"2024-09-13T15:07:02.858078+0000","last_became_active":"2024-09-13T15:05:17.172387+0000","last_became_peered":"2024-09-13T15:05:17.172387+0000","last_unstale":"2024-09-13T15:07:02.858078+0000","last_undegraded":"2024-09-13T15:07:02.858078+0000","last_fullsized":"2024-09-13T15:07:02.858078+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:16.546250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":258,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.831636+0000","last_change":"2024-09-13T15:04:05.021292+0000","last_active":"2024-09-13T15:07:02.831636+0000","last_peered":"2024-09-13T15:07:02.831636+0000","last_clean":"2024-09-13T15:07:02.831636+0000","last_became_active":"2024-09-13T15:04:05.011146+0000","last_became_peered":"2024-09-13T15:04:05.011146+0000","last_unstale":"2024-09-13T15:07:02.831636+0000","last_undegraded":"2024-09-13T15:07:02.831636+0000","last_fullsized":"2024-09-13T15:07:02.831636+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:00.137694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"405'7602","reported_seq":10017,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:06.280584+0000","last_change":"2024-09-13T15:05:53.238583+0000","last_active":"2024-09-13T15:07:06.280584+0000","last_peered":"2024-09-13T15:07:06.280584+0000","last_clean":"2024-09-13T15:07:06.280584+0000","last_became_active":"2024-09-13T15:05:53.232926+0000","last_became_peered":"2024-09-13T15:05:53.232926+0000","last_unstale":"2024-09-13T15:07:06.280584+0000","last_undegraded":"2024-09-13T15:07:06.280584+0000","last_fullsized":"2024-09-13T15:07:06.280584+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:56.757310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311296000,"num_objects":4750,"num_object_clones":0,"num_object_copies":14250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7602,"num_write_kb":395264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1166,"num_bytes_recovered":76414976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":372,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.833504+0000","last_change":"2024-09-13T15:05:17.176480+0000","last_active":"2024-09-13T15:07:02.833504+0000","last_peered":"2024-09-13T15:07:02.833504+0000","last_clean":"2024-09-13T15:07:02.833504+0000","last_became_active":"2024-09-13T15:05:17.166060+0000","last_became_peered":"2024-09-13T15:05:17.166060+0000","last_unstale":"2024-09-13T15:07:02.833504+0000","last_undegraded":"2024-09-13T15:07:02.833504+0000","last_fullsized":"2024-09-13T15:07:02.833504+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:56.196550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":310,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893422+0000","last_change":"2024-09-13T15:05:17.160469+0000","last_active":"2024-09-13T15:06:48.893422+0000","last_peered":"2024-09-13T15:06:48.893422+0000","last_clean":"2024-09-13T15:06:48.893422+0000","last_became_active":"2024-09-13T15:05:17.160119+0000","last_became_peered":"2024-09-13T15:05:17.160119+0000","last_unstale":"2024-09-13T15:06:48.893422+0000","last_undegraded":"2024-09-13T15:06:48.893422+0000","last_fullsized":"2024-09-13T15:06:48.893422+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:34.393376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"405'7755","reported_seq":7829,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:06.113664+0000","last_change":"2024-09-13T15:05:00.141413+0000","last_active":"2024-09-13T15:07:06.113664+0000","last_peered":"2024-09-13T15:07:06.113664+0000","last_clean":"2024-09-13T15:07:06.113664+0000","last_became_active":"2024-09-13T15:05:00.141090+0000","last_became_peered":"2024-09-13T15:05:00.141090+0000","last_unstale":"2024-09-13T15:07:06.113664+0000","last_undegraded":"2024-09-13T15:07:06.113664+0000","last_fullsized":"2024-09-13T15:07:06.113664+0000","mapping_epoch":374,"log_start":"382'1600","ondisk_log_start":"382'1600","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6155,"log_dups_size":0,"ondisk_log_size":6155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:37.515185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329449472,"num_objects":5027,"num_object_clones":0,"num_object_copies":15081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7755,"num_write_kb":409024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":191,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873491+0000","last_change":"2024-09-13T15:05:17.182361+0000","last_active":"2024-09-13T15:06:48.873491+0000","last_peered":"2024-09-13T15:06:48.873491+0000","last_clean":"2024-09-13T15:06:48.873491+0000","last_became_active":"2024-09-13T15:05:17.166521+0000","last_became_peered":"2024-09-13T15:05:17.166521+0000","last_unstale":"2024-09-13T15:06:48.873491+0000","last_undegraded":"2024-09-13T15:06:48.873491+0000","last_fullsized":"2024-09-13T15:06:48.873491+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:47.412279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":469,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.872997+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:06:48.872997+0000","last_peered":"2024-09-13T15:06:48.872997+0000","last_clean":"2024-09-13T15:06:48.872997+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:06:48.872997+0000","last_undegraded":"2024-09-13T15:06:48.872997+0000","last_fullsized":"2024-09-13T15:06:48.872997+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"405'7714","reported_seq":10033,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:08.843903+0000","last_change":"2024-09-13T15:05:27.170000+0000","last_active":"2024-09-13T15:07:08.843903+0000","last_peered":"2024-09-13T15:07:08.843903+0000","last_clean":"2024-09-13T15:07:08.843903+0000","last_became_active":"2024-09-13T15:05:27.167567+0000","last_became_peered":"2024-09-13T15:05:27.167567+0000","last_unstale":"2024-09-13T15:07:08.843903+0000","last_undegraded":"2024-09-13T15:07:08.843903+0000","last_fullsized":"2024-09-13T15:07:08.843903+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7714,"log_dups_size":0,"ondisk_log_size":7714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:54.280593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308805632,"num_objects":4712,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7714,"num_write_kb":397632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":74055680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":167,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893823+0000","last_change":"2024-09-13T15:05:17.179727+0000","last_active":"2024-09-13T15:06:48.893823+0000","last_peered":"2024-09-13T15:06:48.893823+0000","last_clean":"2024-09-13T15:06:48.893823+0000","last_became_active":"2024-09-13T15:05:17.161731+0000","last_became_peered":"2024-09-13T15:05:17.161731+0000","last_unstale":"2024-09-13T15:06:48.893823+0000","last_undegraded":"2024-09-13T15:06:48.893823+0000","last_fullsized":"2024-09-13T15:06:48.893823+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:03.974208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":380,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.820601+0000","last_change":"2024-09-13T15:05:17.160712+0000","last_active":"2024-09-13T15:07:02.820601+0000","last_peered":"2024-09-13T15:07:02.820601+0000","last_clean":"2024-09-13T15:07:02.820601+0000","last_became_active":"2024-09-13T15:05:17.160361+0000","last_became_peered":"2024-09-13T15:05:17.160361+0000","last_unstale":"2024-09-13T15:07:02.820601+0000","last_undegraded":"2024-09-13T15:07:02.820601+0000","last_fullsized":"2024-09-13T15:07:02.820601+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:58.499951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"405'7846","reported_seq":7886,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:08.747230+0000","last_change":"2024-09-13T15:05:00.294922+0000","last_active":"2024-09-13T15:07:08.747230+0000","last_peered":"2024-09-13T15:07:08.747230+0000","last_clean":"2024-09-13T15:07:08.747230+0000","last_became_active":"2024-09-13T15:05:00.294640+0000","last_became_peered":"2024-09-13T15:05:00.294640+0000","last_unstale":"2024-09-13T15:07:08.747230+0000","last_undegraded":"2024-09-13T15:07:08.747230+0000","last_fullsized":"2024-09-13T15:07:08.747230+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7846,"log_dups_size":0,"ondisk_log_size":7846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:17:27.248373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310116352,"num_objects":4732,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7846,"num_write_kb":402496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":281,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893603+0000","last_change":"2024-09-13T15:04:05.013208+0000","last_active":"2024-09-13T15:06:48.893603+0000","last_peered":"2024-09-13T15:06:48.893603+0000","last_clean":"2024-09-13T15:06:48.893603+0000","last_became_active":"2024-09-13T15:04:05.012896+0000","last_became_peered":"2024-09-13T15:04:05.012896+0000","last_unstale":"2024-09-13T15:06:48.893603+0000","last_undegraded":"2024-09-13T15:06:48.893603+0000","last_fullsized":"2024-09-13T15:06:48.893603+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:45.434822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":560,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.840414+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:07:02.840414+0000","last_peered":"2024-09-13T15:07:02.840414+0000","last_clean":"2024-09-13T15:07:02.840414+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:07:02.840414+0000","last_undegraded":"2024-09-13T15:07:02.840414+0000","last_fullsized":"2024-09-13T15:07:02.840414+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"405'7699","reported_seq":10080,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:05.886378+0000","last_change":"2024-09-13T15:05:52.168407+0000","last_active":"2024-09-13T15:07:05.886378+0000","last_peered":"2024-09-13T15:07:05.886378+0000","last_clean":"2024-09-13T15:07:05.886378+0000","last_became_active":"2024-09-13T15:05:52.164452+0000","last_became_peered":"2024-09-13T15:05:52.164452+0000","last_unstale":"2024-09-13T15:07:05.886378+0000","last_undegraded":"2024-09-13T15:07:05.886378+0000","last_fullsized":"2024-09-13T15:07:05.886378+0000","mapping_epoch":390,"log_start":"377'1100","ondisk_log_start":"377'1100","created":374,"last_epoch_clean":391,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:59.354716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326696960,"num_objects":4985,"num_object_clones":0,"num_object_copies":14955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7699,"num_write_kb":405888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1148,"num_bytes_recovered":75235328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":686,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.829520+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:07:02.829520+0000","last_peered":"2024-09-13T15:07:02.829520+0000","last_clean":"2024-09-13T15:07:02.829520+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:07:02.829520+0000","last_undegraded":"2024-09-13T15:07:02.829520+0000","last_fullsized":"2024-09-13T15:07:02.829520+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":412,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873001+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:06:48.873001+0000","last_peered":"2024-09-13T15:06:48.873001+0000","last_clean":"2024-09-13T15:06:48.873001+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:06:48.873001+0000","last_undegraded":"2024-09-13T15:06:48.873001+0000","last_fullsized":"2024-09-13T15:06:48.873001+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"405'7602","reported_seq":7676,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:06.080229+0000","last_change":"2024-09-13T15:05:00.295183+0000","last_active":"2024-09-13T15:07:06.080229+0000","last_peered":"2024-09-13T15:07:06.080229+0000","last_clean":"2024-09-13T15:07:06.080229+0000","last_became_active":"2024-09-13T15:05:00.294632+0000","last_became_peered":"2024-09-13T15:05:00.294632+0000","last_unstale":"2024-09-13T15:07:06.080229+0000","last_undegraded":"2024-09-13T15:07:06.080229+0000","last_fullsized":"2024-09-13T15:07:06.080229+0000","mapping_epoch":374,"log_start":"382'1400","ondisk_log_start":"382'1400","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:52.843009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318636032,"num_objects":4862,"num_object_clones":0,"num_object_copies":14586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7602,"num_write_kb":398848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":209,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.872678+0000","last_change":"2024-09-13T15:04:05.013503+0000","last_active":"2024-09-13T15:06:48.872678+0000","last_peered":"2024-09-13T15:06:48.872678+0000","last_clean":"2024-09-13T15:06:48.872678+0000","last_became_active":"2024-09-13T15:04:05.013204+0000","last_became_peered":"2024-09-13T15:04:05.013204+0000","last_unstale":"2024-09-13T15:06:48.872678+0000","last_undegraded":"2024-09-13T15:06:48.872678+0000","last_fullsized":"2024-09-13T15:06:48.872678+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:17.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":406,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873470+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:06:48.873470+0000","last_peered":"2024-09-13T15:06:48.873470+0000","last_clean":"2024-09-13T15:06:48.873470+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:06:48.873470+0000","last_undegraded":"2024-09-13T15:06:48.873470+0000","last_fullsized":"2024-09-13T15:06:48.873470+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"405'7713","reported_seq":10041,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:05.842651+0000","last_change":"2024-09-13T15:05:30.181240+0000","last_active":"2024-09-13T15:07:05.842651+0000","last_peered":"2024-09-13T15:07:05.842651+0000","last_clean":"2024-09-13T15:07:05.842651+0000","last_became_active":"2024-09-13T15:05:30.177725+0000","last_became_peered":"2024-09-13T15:05:30.177725+0000","last_unstale":"2024-09-13T15:07:05.842651+0000","last_undegraded":"2024-09-13T15:07:05.842651+0000","last_fullsized":"2024-09-13T15:07:05.842651+0000","mapping_epoch":385,"log_start":"377'1100","ondisk_log_start":"377'1100","created":374,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6613,"log_dups_size":0,"ondisk_log_size":6613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:35.297203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326696960,"num_objects":4985,"num_object_clones":0,"num_object_copies":14955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7713,"num_write_kb":406336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":73662464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":421,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.872928+0000","last_change":"2024-09-13T15:05:17.166900+0000","last_active":"2024-09-13T15:06:48.872928+0000","last_peered":"2024-09-13T15:06:48.872928+0000","last_clean":"2024-09-13T15:06:48.872928+0000","last_became_active":"2024-09-13T15:05:17.166511+0000","last_became_peered":"2024-09-13T15:05:17.166511+0000","last_unstale":"2024-09-13T15:06:48.872928+0000","last_undegraded":"2024-09-13T15:06:48.872928+0000","last_fullsized":"2024-09-13T15:06:48.872928+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:38.150397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":439,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873504+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:06:48.873504+0000","last_peered":"2024-09-13T15:06:48.873504+0000","last_clean":"2024-09-13T15:06:48.873504+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:06:48.873504+0000","last_undegraded":"2024-09-13T15:06:48.873504+0000","last_fullsized":"2024-09-13T15:06:48.873504+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"405'7662","reported_seq":7734,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:05.884500+0000","last_change":"2024-09-13T15:05:00.294995+0000","last_active":"2024-09-13T15:07:05.884500+0000","last_peered":"2024-09-13T15:07:05.884500+0000","last_clean":"2024-09-13T15:07:05.884500+0000","last_became_active":"2024-09-13T15:05:00.294641+0000","last_became_peered":"2024-09-13T15:05:00.294641+0000","last_unstale":"2024-09-13T15:07:05.884500+0000","last_undegraded":"2024-09-13T15:07:05.884500+0000","last_fullsized":"2024-09-13T15:07:05.884500+0000","mapping_epoch":374,"log_start":"377'1100","ondisk_log_start":"377'1100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:57.613762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320864256,"num_objects":4896,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7662,"num_write_kb":401856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":145,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.829815+0000","last_change":"2024-09-13T15:05:17.168897+0000","last_active":"2024-09-13T15:07:02.829815+0000","last_peered":"2024-09-13T15:07:02.829815+0000","last_clean":"2024-09-13T15:07:02.829815+0000","last_became_active":"2024-09-13T15:05:17.168557+0000","last_became_peered":"2024-09-13T15:05:17.168557+0000","last_unstale":"2024-09-13T15:07:02.829815+0000","last_undegraded":"2024-09-13T15:07:02.829815+0000","last_fullsized":"2024-09-13T15:07:02.829815+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:43.776873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":432,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.833805+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:07:02.833805+0000","last_peered":"2024-09-13T15:07:02.833805+0000","last_clean":"2024-09-13T15:07:02.833805+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:07:02.833805+0000","last_undegraded":"2024-09-13T15:07:02.833805+0000","last_fullsized":"2024-09-13T15:07:02.833805+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"405'7647","reported_seq":7719,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:05.858834+0000","last_change":"2024-09-13T15:05:00.139111+0000","last_active":"2024-09-13T15:07:05.858834+0000","last_peered":"2024-09-13T15:07:05.858834+0000","last_clean":"2024-09-13T15:07:05.858834+0000","last_became_active":"2024-09-13T15:05:00.138827+0000","last_became_peered":"2024-09-13T15:05:00.138827+0000","last_unstale":"2024-09-13T15:07:05.858834+0000","last_undegraded":"2024-09-13T15:07:05.858834+0000","last_fullsized":"2024-09-13T15:07:05.858834+0000","mapping_epoch":374,"log_start":"377'1100","ondisk_log_start":"377'1100","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:05.115688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319881216,"num_objects":4881,"num_object_clones":0,"num_object_copies":14643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":400896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":710,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.831891+0000","last_change":"2024-09-13T15:05:20.215619+0000","last_active":"2024-09-13T15:07:02.831891+0000","last_peered":"2024-09-13T15:07:02.831891+0000","last_clean":"2024-09-13T15:07:02.831891+0000","last_became_active":"2024-09-13T15:05:20.215201+0000","last_became_peered":"2024-09-13T15:05:20.215201+0000","last_unstale":"2024-09-13T15:07:02.831891+0000","last_undegraded":"2024-09-13T15:07:02.831891+0000","last_fullsized":"2024-09-13T15:07:02.831891+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:35.730765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1060,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893644+0000","last_change":"2024-09-13T15:05:20.221461+0000","last_active":"2024-09-13T15:06:48.893644+0000","last_peered":"2024-09-13T15:06:48.893644+0000","last_clean":"2024-09-13T15:06:48.893644+0000","last_became_active":"2024-09-13T15:05:20.221044+0000","last_became_peered":"2024-09-13T15:05:20.221044+0000","last_unstale":"2024-09-13T15:06:48.893644+0000","last_undegraded":"2024-09-13T15:06:48.893644+0000","last_fullsized":"2024-09-13T15:06:48.893644+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:36:06.133611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":648,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.830431+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:07:02.830431+0000","last_peered":"2024-09-13T15:07:02.830431+0000","last_clean":"2024-09-13T15:07:02.830431+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:07:02.830431+0000","last_undegraded":"2024-09-13T15:07:02.830431+0000","last_fullsized":"2024-09-13T15:07:02.830431+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":249,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:58.711044+0000","last_change":"2024-09-13T15:05:17.174768+0000","last_active":"2024-09-13T15:06:58.711044+0000","last_peered":"2024-09-13T15:06:58.711044+0000","last_clean":"2024-09-13T15:06:58.711044+0000","last_became_active":"2024-09-13T15:05:17.174352+0000","last_became_peered":"2024-09-13T15:05:17.174352+0000","last_unstale":"2024-09-13T15:06:58.711044+0000","last_undegraded":"2024-09-13T15:06:58.711044+0000","last_fullsized":"2024-09-13T15:06:58.711044+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:15:37.462653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"405'7615","reported_seq":7688,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:06.296901+0000","last_change":"2024-09-13T15:05:00.140878+0000","last_active":"2024-09-13T15:07:06.296901+0000","last_peered":"2024-09-13T15:07:06.296901+0000","last_clean":"2024-09-13T15:07:06.296901+0000","last_became_active":"2024-09-13T15:05:00.139767+0000","last_became_peered":"2024-09-13T15:05:00.139767+0000","last_unstale":"2024-09-13T15:07:06.296901+0000","last_undegraded":"2024-09-13T15:07:06.296901+0000","last_fullsized":"2024-09-13T15:07:06.296901+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7615,"log_dups_size":0,"ondisk_log_size":7615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:58.764080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316080128,"num_objects":4823,"num_object_clones":0,"num_object_copies":14469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7615,"num_write_kb":398016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":550,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.821385+0000","last_change":"2024-09-13T15:05:17.189624+0000","last_active":"2024-09-13T15:07:02.821385+0000","last_peered":"2024-09-13T15:07:02.821385+0000","last_clean":"2024-09-13T15:07:02.821385+0000","last_became_active":"2024-09-13T15:05:17.165622+0000","last_became_peered":"2024-09-13T15:05:17.165622+0000","last_unstale":"2024-09-13T15:07:02.821385+0000","last_undegraded":"2024-09-13T15:07:02.821385+0000","last_fullsized":"2024-09-13T15:07:02.821385+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:03.225679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":634,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.835990+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:07:02.835990+0000","last_peered":"2024-09-13T15:07:02.835990+0000","last_clean":"2024-09-13T15:07:02.835990+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:07:02.835990+0000","last_undegraded":"2024-09-13T15:07:02.835990+0000","last_fullsized":"2024-09-13T15:07:02.835990+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":629,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.836570+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:07:02.836570+0000","last_peered":"2024-09-13T15:07:02.836570+0000","last_clean":"2024-09-13T15:07:02.836570+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:07:02.836570+0000","last_undegraded":"2024-09-13T15:07:02.836570+0000","last_fullsized":"2024-09-13T15:07:02.836570+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"405'7796","reported_seq":10180,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:08.869844+0000","last_change":"2024-09-13T15:06:19.412652+0000","last_active":"2024-09-13T15:07:08.869844+0000","last_peered":"2024-09-13T15:07:08.869844+0000","last_clean":"2024-09-13T15:07:08.869844+0000","last_became_active":"2024-09-13T15:06:19.406388+0000","last_became_peered":"2024-09-13T15:06:19.406388+0000","last_unstale":"2024-09-13T15:07:08.869844+0000","last_undegraded":"2024-09-13T15:07:08.869844+0000","last_fullsized":"2024-09-13T15:07:08.869844+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:21.244358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308281344,"num_objects":4704,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":400000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":76283904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1069,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.833489+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:07:02.833489+0000","last_peered":"2024-09-13T15:07:02.833489+0000","last_clean":"2024-09-13T15:07:02.833489+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:07:02.833489+0000","last_undegraded":"2024-09-13T15:07:02.833489+0000","last_fullsized":"2024-09-13T15:07:02.833489+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":698,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.821616+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:07:02.821616+0000","last_peered":"2024-09-13T15:07:02.821616+0000","last_clean":"2024-09-13T15:07:02.821616+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:07:02.821616+0000","last_undegraded":"2024-09-13T15:07:02.821616+0000","last_fullsized":"2024-09-13T15:07:02.821616+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":435,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.818869+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:07:02.818869+0000","last_peered":"2024-09-13T15:07:02.818869+0000","last_clean":"2024-09-13T15:07:02.818869+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:07:02.818869+0000","last_undegraded":"2024-09-13T15:07:02.818869+0000","last_fullsized":"2024-09-13T15:07:02.818869+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"405'7626","reported_seq":9993,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:06.292309+0000","last_change":"2024-09-13T15:06:06.083813+0000","last_active":"2024-09-13T15:07:06.292309+0000","last_peered":"2024-09-13T15:07:06.292309+0000","last_clean":"2024-09-13T15:07:06.292309+0000","last_became_active":"2024-09-13T15:06:06.080389+0000","last_became_peered":"2024-09-13T15:06:06.080389+0000","last_unstale":"2024-09-13T15:07:06.292309+0000","last_undegraded":"2024-09-13T15:07:06.292309+0000","last_fullsized":"2024-09-13T15:07:06.292309+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7626,"log_dups_size":0,"ondisk_log_size":7626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:03.965346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317849600,"num_objects":4850,"num_object_clones":0,"num_object_copies":14550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7626,"num_write_kb":399232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":74842112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":331,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.832102+0000","last_change":"2024-09-13T15:04:05.035515+0000","last_active":"2024-09-13T15:07:02.832102+0000","last_peered":"2024-09-13T15:07:02.832102+0000","last_clean":"2024-09-13T15:07:02.832102+0000","last_became_active":"2024-09-13T15:04:05.017008+0000","last_became_peered":"2024-09-13T15:04:05.017008+0000","last_unstale":"2024-09-13T15:07:02.832102+0000","last_undegraded":"2024-09-13T15:07:02.832102+0000","last_fullsized":"2024-09-13T15:07:02.832102+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:44.511395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":431,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873296+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:06:48.873296+0000","last_peered":"2024-09-13T15:06:48.873296+0000","last_clean":"2024-09-13T15:06:48.873296+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:06:48.873296+0000","last_undegraded":"2024-09-13T15:06:48.873296+0000","last_fullsized":"2024-09-13T15:06:48.873296+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":190,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893486+0000","last_change":"2024-09-13T15:00:14.949848+0000","last_active":"2024-09-13T15:06:48.893486+0000","last_peered":"2024-09-13T15:06:48.893486+0000","last_clean":"2024-09-13T15:06:48.893486+0000","last_became_active":"2024-09-13T15:00:14.949644+0000","last_became_peered":"2024-09-13T15:00:14.949644+0000","last_unstale":"2024-09-13T15:06:48.893486+0000","last_undegraded":"2024-09-13T15:06:48.893486+0000","last_fullsized":"2024-09-13T15:06:48.893486+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:17.762542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"405'7966","reported_seq":8006,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:08.851352+0000","last_change":"2024-09-13T15:05:00.143221+0000","last_active":"2024-09-13T15:07:08.851352+0000","last_peered":"2024-09-13T15:07:08.851352+0000","last_clean":"2024-09-13T15:07:08.851352+0000","last_became_active":"2024-09-13T15:05:00.142922+0000","last_became_peered":"2024-09-13T15:05:00.142922+0000","last_unstale":"2024-09-13T15:07:08.851352+0000","last_undegraded":"2024-09-13T15:07:08.851352+0000","last_fullsized":"2024-09-13T15:07:08.851352+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7966,"log_dups_size":0,"ondisk_log_size":7966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:26.225827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310640640,"num_objects":4740,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7966,"num_write_kb":406592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":280,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893588+0000","last_change":"2024-09-13T15:00:14.948015+0000","last_active":"2024-09-13T15:06:48.893588+0000","last_peered":"2024-09-13T15:06:48.893588+0000","last_clean":"2024-09-13T15:06:48.893588+0000","last_became_active":"2024-09-13T15:00:14.947799+0000","last_became_peered":"2024-09-13T15:00:14.947799+0000","last_unstale":"2024-09-13T15:06:48.893588+0000","last_undegraded":"2024-09-13T15:06:48.893588+0000","last_fullsized":"2024-09-13T15:06:48.893588+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:08:37.576480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1313,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.819493+0000","last_change":"2024-09-13T15:05:20.213214+0000","last_active":"2024-09-13T15:07:02.819493+0000","last_peered":"2024-09-13T15:07:02.819493+0000","last_clean":"2024-09-13T15:07:02.819493+0000","last_became_active":"2024-09-13T15:05:20.212685+0000","last_became_peered":"2024-09-13T15:05:20.212685+0000","last_unstale":"2024-09-13T15:07:02.819493+0000","last_undegraded":"2024-09-13T15:07:02.819493+0000","last_fullsized":"2024-09-13T15:07:02.819493+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:34.890174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":279,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.835842+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:07:02.835842+0000","last_peered":"2024-09-13T15:07:02.835842+0000","last_clean":"2024-09-13T15:07:02.835842+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:07:02.835842+0000","last_undegraded":"2024-09-13T15:07:02.835842+0000","last_fullsized":"2024-09-13T15:07:02.835842+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"405'7734","reported_seq":10023,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:08.675923+0000","last_change":"2024-09-13T15:06:03.045512+0000","last_active":"2024-09-13T15:07:08.675923+0000","last_peered":"2024-09-13T15:07:08.675923+0000","last_clean":"2024-09-13T15:07:08.675923+0000","last_became_active":"2024-09-13T15:06:03.039480+0000","last_became_peered":"2024-09-13T15:06:03.039480+0000","last_unstale":"2024-09-13T15:07:08.675923+0000","last_undegraded":"2024-09-13T15:07:08.675923+0000","last_fullsized":"2024-09-13T15:07:08.675923+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":374,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:27.441865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310509568,"num_objects":4738,"num_object_clones":0,"num_object_copies":14214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7734,"num_write_kb":399104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1116,"num_bytes_recovered":73138176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":689,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.819383+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:07:02.819383+0000","last_peered":"2024-09-13T15:07:02.819383+0000","last_clean":"2024-09-13T15:07:02.819383+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:07:02.819383+0000","last_undegraded":"2024-09-13T15:07:02.819383+0000","last_fullsized":"2024-09-13T15:07:02.819383+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":742,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.842173+0000","last_change":"2024-09-13T15:05:17.190336+0000","last_active":"2024-09-13T15:07:02.842173+0000","last_peered":"2024-09-13T15:07:02.842173+0000","last_clean":"2024-09-13T15:07:02.842173+0000","last_became_active":"2024-09-13T15:05:17.173941+0000","last_became_peered":"2024-09-13T15:05:17.173941+0000","last_unstale":"2024-09-13T15:07:02.842173+0000","last_undegraded":"2024-09-13T15:07:02.842173+0000","last_fullsized":"2024-09-13T15:07:02.842173+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:31:05.533600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":631,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.840534+0000","last_change":"2024-09-13T15:05:17.171354+0000","last_active":"2024-09-13T15:07:02.840534+0000","last_peered":"2024-09-13T15:07:02.840534+0000","last_clean":"2024-09-13T15:07:02.840534+0000","last_became_active":"2024-09-13T15:05:17.166185+0000","last_became_peered":"2024-09-13T15:05:17.166185+0000","last_unstale":"2024-09-13T15:07:02.840534+0000","last_undegraded":"2024-09-13T15:07:02.840534+0000","last_fullsized":"2024-09-13T15:07:02.840534+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:01:46.045855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"405'7597","reported_seq":9930,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:06.086471+0000","last_change":"2024-09-13T15:05:40.158205+0000","last_active":"2024-09-13T15:07:06.086471+0000","last_peered":"2024-09-13T15:07:06.086471+0000","last_clean":"2024-09-13T15:07:06.086471+0000","last_became_active":"2024-09-13T15:05:40.154150+0000","last_became_peered":"2024-09-13T15:05:40.154150+0000","last_unstale":"2024-09-13T15:07:06.086471+0000","last_undegraded":"2024-09-13T15:07:06.086471+0000","last_fullsized":"2024-09-13T15:07:06.086471+0000","mapping_epoch":387,"log_start":"382'1400","ondisk_log_start":"382'1400","created":374,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:18.178381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320471064,"num_objects":4891,"num_object_clones":0,"num_object_copies":14673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4891,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7597,"num_write_kb":399553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1126,"num_bytes_recovered":73793536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":730,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.858298+0000","last_change":"2024-09-13T15:05:17.179784+0000","last_active":"2024-09-13T15:07:02.858298+0000","last_peered":"2024-09-13T15:07:02.858298+0000","last_clean":"2024-09-13T15:07:02.858298+0000","last_became_active":"2024-09-13T15:05:17.173165+0000","last_became_peered":"2024-09-13T15:05:17.173165+0000","last_unstale":"2024-09-13T15:07:02.858298+0000","last_undegraded":"2024-09-13T15:07:02.858298+0000","last_fullsized":"2024-09-13T15:07:02.858298+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:34.990856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1176,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.822919+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:07:02.822919+0000","last_peered":"2024-09-13T15:07:02.822919+0000","last_clean":"2024-09-13T15:07:02.822919+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:07:02.822919+0000","last_undegraded":"2024-09-13T15:07:02.822919+0000","last_fullsized":"2024-09-13T15:07:02.822919+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":439,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.829933+0000","last_change":"2024-09-13T15:05:17.171537+0000","last_active":"2024-09-13T15:07:02.829933+0000","last_peered":"2024-09-13T15:07:02.829933+0000","last_clean":"2024-09-13T15:07:02.829933+0000","last_became_active":"2024-09-13T15:05:17.171212+0000","last_became_peered":"2024-09-13T15:05:17.171212+0000","last_unstale":"2024-09-13T15:07:02.829933+0000","last_undegraded":"2024-09-13T15:07:02.829933+0000","last_fullsized":"2024-09-13T15:07:02.829933+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:09.747166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"405'7720","reported_seq":10053,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:05.818791+0000","last_change":"2024-09-13T15:06:17.388289+0000","last_active":"2024-09-13T15:07:05.818791+0000","last_peered":"2024-09-13T15:07:05.818791+0000","last_clean":"2024-09-13T15:07:05.818791+0000","last_became_active":"2024-09-13T15:06:17.381406+0000","last_became_peered":"2024-09-13T15:06:17.381406+0000","last_unstale":"2024-09-13T15:07:05.818791+0000","last_undegraded":"2024-09-13T15:07:05.818791+0000","last_fullsized":"2024-09-13T15:07:05.818791+0000","mapping_epoch":398,"log_start":"377'1100","ondisk_log_start":"377'1100","created":374,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:04:59.123724+0000","last_clean_scrub_stamp":"2024-09-13T15:04:59.123724+0000","objects_scrubbed":0,"log_size":6620,"log_dups_size":0,"ondisk_log_size":6620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:43:00.890496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330825728,"num_objects":5048,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":73662464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":565,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.836929+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:07:02.836929+0000","last_peered":"2024-09-13T15:07:02.836929+0000","last_clean":"2024-09-13T15:07:02.836929+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:07:02.836929+0000","last_undegraded":"2024-09-13T15:07:02.836929+0000","last_fullsized":"2024-09-13T15:07:02.836929+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":800,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.833703+0000","last_change":"2024-09-13T15:05:20.212166+0000","last_active":"2024-09-13T15:07:02.833703+0000","last_peered":"2024-09-13T15:07:02.833703+0000","last_clean":"2024-09-13T15:07:02.833703+0000","last_became_active":"2024-09-13T15:05:20.210761+0000","last_became_peered":"2024-09-13T15:05:20.210761+0000","last_unstale":"2024-09-13T15:07:02.833703+0000","last_undegraded":"2024-09-13T15:07:02.833703+0000","last_fullsized":"2024-09-13T15:07:02.833703+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:03.869537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":427,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.872835+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:06:48.872835+0000","last_peered":"2024-09-13T15:06:48.872835+0000","last_clean":"2024-09-13T15:06:48.872835+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:06:48.872835+0000","last_undegraded":"2024-09-13T15:06:48.872835+0000","last_fullsized":"2024-09-13T15:06:48.872835+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":411,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873327+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:06:48.873327+0000","last_peered":"2024-09-13T15:06:48.873327+0000","last_clean":"2024-09-13T15:06:48.873327+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:06:48.873327+0000","last_undegraded":"2024-09-13T15:06:48.873327+0000","last_fullsized":"2024-09-13T15:06:48.873327+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":524,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873196+0000","last_change":"2024-09-13T15:05:17.177141+0000","last_active":"2024-09-13T15:06:48.873196+0000","last_peered":"2024-09-13T15:06:48.873196+0000","last_clean":"2024-09-13T15:06:48.873196+0000","last_became_active":"2024-09-13T15:05:17.164585+0000","last_became_peered":"2024-09-13T15:05:17.164585+0000","last_unstale":"2024-09-13T15:06:48.873196+0000","last_undegraded":"2024-09-13T15:06:48.873196+0000","last_fullsized":"2024-09-13T15:06:48.873196+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:10.575447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":28536,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":701,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.820728+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:07:02.820728+0000","last_peered":"2024-09-13T15:07:02.820728+0000","last_clean":"2024-09-13T15:07:02.820728+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:07:02.820728+0000","last_undegraded":"2024-09-13T15:07:02.820728+0000","last_fullsized":"2024-09-13T15:07:02.820728+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":341,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.828010+0000","last_change":"2024-09-13T15:05:17.186042+0000","last_active":"2024-09-13T15:07:02.828010+0000","last_peered":"2024-09-13T15:07:02.828010+0000","last_clean":"2024-09-13T15:07:02.828010+0000","last_became_active":"2024-09-13T15:05:17.173556+0000","last_became_peered":"2024-09-13T15:05:17.173556+0000","last_unstale":"2024-09-13T15:07:02.828010+0000","last_undegraded":"2024-09-13T15:07:02.828010+0000","last_fullsized":"2024-09-13T15:07:02.828010+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:54:09.073402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":291,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893865+0000","last_change":"2024-09-13T15:05:17.166273+0000","last_active":"2024-09-13T15:06:48.893865+0000","last_peered":"2024-09-13T15:06:48.893865+0000","last_clean":"2024-09-13T15:06:48.893865+0000","last_became_active":"2024-09-13T15:05:17.165866+0000","last_became_peered":"2024-09-13T15:05:17.165866+0000","last_unstale":"2024-09-13T15:06:48.893865+0000","last_undegraded":"2024-09-13T15:06:48.893865+0000","last_fullsized":"2024-09-13T15:06:48.893865+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:45.954602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":681,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.839812+0000","last_change":"2024-09-13T15:05:17.168553+0000","last_active":"2024-09-13T15:07:02.839812+0000","last_peered":"2024-09-13T15:07:02.839812+0000","last_clean":"2024-09-13T15:07:02.839812+0000","last_became_active":"2024-09-13T15:05:17.168165+0000","last_became_peered":"2024-09-13T15:05:17.168165+0000","last_unstale":"2024-09-13T15:07:02.839812+0000","last_undegraded":"2024-09-13T15:07:02.839812+0000","last_fullsized":"2024-09-13T15:07:02.839812+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:22.447491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":657,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.836409+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:07:02.836409+0000","last_peered":"2024-09-13T15:07:02.836409+0000","last_clean":"2024-09-13T15:07:02.836409+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:07:02.836409+0000","last_undegraded":"2024-09-13T15:07:02.836409+0000","last_fullsized":"2024-09-13T15:07:02.836409+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":718,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.830011+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:07:02.830011+0000","last_peered":"2024-09-13T15:07:02.830011+0000","last_clean":"2024-09-13T15:07:02.830011+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:07:02.830011+0000","last_undegraded":"2024-09-13T15:07:02.830011+0000","last_fullsized":"2024-09-13T15:07:02.830011+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":210,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:53.873030+0000","last_change":"2024-09-13T15:04:05.011358+0000","last_active":"2024-09-13T15:06:53.873030+0000","last_peered":"2024-09-13T15:06:53.873030+0000","last_clean":"2024-09-13T15:06:53.873030+0000","last_became_active":"2024-09-13T15:04:05.011035+0000","last_became_peered":"2024-09-13T15:04:05.011035+0000","last_unstale":"2024-09-13T15:06:53.873030+0000","last_undegraded":"2024-09-13T15:06:53.873030+0000","last_fullsized":"2024-09-13T15:06:53.873030+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:52.740914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":682,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.830178+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:07:02.830178+0000","last_peered":"2024-09-13T15:07:02.830178+0000","last_clean":"2024-09-13T15:07:02.830178+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:07:02.830178+0000","last_undegraded":"2024-09-13T15:07:02.830178+0000","last_fullsized":"2024-09-13T15:07:02.830178+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":646,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.836716+0000","last_change":"2024-09-13T15:05:17.168020+0000","last_active":"2024-09-13T15:07:02.836716+0000","last_peered":"2024-09-13T15:07:02.836716+0000","last_clean":"2024-09-13T15:07:02.836716+0000","last_became_active":"2024-09-13T15:05:17.167277+0000","last_became_peered":"2024-09-13T15:05:17.167277+0000","last_unstale":"2024-09-13T15:07:02.836716+0000","last_undegraded":"2024-09-13T15:07:02.836716+0000","last_fullsized":"2024-09-13T15:07:02.836716+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:41.746548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":655,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.816533+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:07:02.816533+0000","last_peered":"2024-09-13T15:07:02.816533+0000","last_clean":"2024-09-13T15:07:02.816533+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:07:02.816533+0000","last_undegraded":"2024-09-13T15:07:02.816533+0000","last_fullsized":"2024-09-13T15:07:02.816533+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":520,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.827857+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:07:02.827857+0000","last_peered":"2024-09-13T15:07:02.827857+0000","last_clean":"2024-09-13T15:07:02.827857+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:07:02.827857+0000","last_undegraded":"2024-09-13T15:07:02.827857+0000","last_fullsized":"2024-09-13T15:07:02.827857+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":406,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873101+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:06:48.873101+0000","last_peered":"2024-09-13T15:06:48.873101+0000","last_clean":"2024-09-13T15:06:48.873101+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:06:48.873101+0000","last_undegraded":"2024-09-13T15:06:48.873101+0000","last_fullsized":"2024-09-13T15:06:48.873101+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":271,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.834570+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:07:02.834570+0000","last_peered":"2024-09-13T15:07:02.834570+0000","last_clean":"2024-09-13T15:07:02.834570+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:07:02.834570+0000","last_undegraded":"2024-09-13T15:07:02.834570+0000","last_fullsized":"2024-09-13T15:07:02.834570+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":634,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.840611+0000","last_change":"2024-09-13T15:05:17.171724+0000","last_active":"2024-09-13T15:07:02.840611+0000","last_peered":"2024-09-13T15:07:02.840611+0000","last_clean":"2024-09-13T15:07:02.840611+0000","last_became_active":"2024-09-13T15:05:17.171338+0000","last_became_peered":"2024-09-13T15:05:17.171338+0000","last_unstale":"2024-09-13T15:07:02.840611+0000","last_undegraded":"2024-09-13T15:07:02.840611+0000","last_fullsized":"2024-09-13T15:07:02.840611+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:22.296363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":416,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.873178+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:06:48.873178+0000","last_peered":"2024-09-13T15:06:48.873178+0000","last_clean":"2024-09-13T15:06:48.873178+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:06:48.873178+0000","last_undegraded":"2024-09-13T15:06:48.873178+0000","last_fullsized":"2024-09-13T15:06:48.873178+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":248,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893610+0000","last_change":"2024-09-13T14:48:21.151287+0000","last_active":"2024-09-13T15:06:48.893610+0000","last_peered":"2024-09-13T15:06:48.893610+0000","last_clean":"2024-09-13T15:06:48.893610+0000","last_became_active":"2024-09-13T14:48:21.098497+0000","last_became_peered":"2024-09-13T14:48:21.098497+0000","last_unstale":"2024-09-13T15:06:48.893610+0000","last_undegraded":"2024-09-13T15:06:48.893610+0000","last_fullsized":"2024-09-13T15:06:48.893610+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.734846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":404,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.872685+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:06:48.872685+0000","last_peered":"2024-09-13T15:06:48.872685+0000","last_clean":"2024-09-13T15:06:48.872685+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:06:48.872685+0000","last_undegraded":"2024-09-13T15:06:48.872685+0000","last_fullsized":"2024-09-13T15:06:48.872685+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":276,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:48.893806+0000","last_change":"2024-09-13T15:04:05.027223+0000","last_active":"2024-09-13T15:06:48.893806+0000","last_peered":"2024-09-13T15:06:48.893806+0000","last_clean":"2024-09-13T15:06:48.893806+0000","last_became_active":"2024-09-13T15:04:05.014803+0000","last_became_peered":"2024-09-13T15:04:05.014803+0000","last_unstale":"2024-09-13T15:06:48.893806+0000","last_undegraded":"2024-09-13T15:06:48.893806+0000","last_fullsized":"2024-09-13T15:06:48.893806+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:23.094302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":692,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.840193+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:07:02.840193+0000","last_peered":"2024-09-13T15:07:02.840193+0000","last_clean":"2024-09-13T15:07:02.840193+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:07:02.840193+0000","last_undegraded":"2024-09-13T15:07:02.840193+0000","last_fullsized":"2024-09-13T15:07:02.840193+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":715,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.828243+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:07:02.828243+0000","last_peered":"2024-09-13T15:07:02.828243+0000","last_clean":"2024-09-13T15:07:02.828243+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:07:02.828243+0000","last_undegraded":"2024-09-13T15:07:02.828243+0000","last_fullsized":"2024-09-13T15:07:02.828243+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":438,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-09-13T15:06:58.710954+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:06:58.710954+0000","last_peered":"2024-09-13T15:06:58.710954+0000","last_clean":"2024-09-13T15:06:58.710954+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:06:58.710954+0000","last_undegraded":"2024-09-13T15:06:58.710954+0000","last_fullsized":"2024-09-13T15:06:58.710954+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":681,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-09-13T15:07:02.836143+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:07:02.836143+0000","last_peered":"2024-09-13T15:07:02.836143+0000","last_clean":"2024-09-13T15:07:02.836143+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:07:02.836143+0000","last_undegraded":"2024-09-13T15:07:02.836143+0000","last_fullsized":"2024-09-13T15:07:02.836143+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5087100952,"num_objects":77624,"num_object_clones":0,"num_object_copies":232872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123294,"num_write_kb":6429313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10240,"num_bytes_recovered":671088640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15188439040,"data_stored":15188426824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113394,"ondisk_log_size":113394,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":31878,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":21634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":0,"seq":231928234477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496113,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1431364,"kb_used_data":926908,"kb_used_omap":1,"kb_used_meta":504446,"kb_avail":92313788,"statfs":{"total":95995035648,"available":94529318912,"internally_reserved":0,"allocated":949153792,"data_stored":946944072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":516553159},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757748,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3681468,"kb_used_data":3040820,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":90063684,"statfs":{"total":95995035648,"available":92225212416,"internally_reserved":0,"allocated":3113799680,"data_stored":3111593755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":656013767},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019388,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2168072,"kb_used_data":1551232,"kb_used_omap":1,"kb_used_meta":616830,"kb_avail":91577080,"statfs":{"total":95995035648,"available":93774929920,"internally_reserved":0,"allocated":1588461568,"data_stored":1586247837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":631634374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468272,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3962232,"kb_used_data":3182320,"kb_used_omap":4,"kb_used_meta":779899,"kb_avail":89782920,"statfs":{"total":95995035648,"available":91937710080,"internally_reserved":0,"allocated":3258695680,"data_stored":3256493601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":798616740},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273172,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511496,"kb_used_data":1233280,"kb_used_omap":12,"kb_used_meta":278131,"kb_avail":92233656,"statfs":{"total":95995035648,"available":94447263744,"internally_reserved":0,"allocated":1262878720,"data_stored":1260665491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":284807040},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795324,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3847232,"kb_used_data":3139448,"kb_used_omap":7,"kb_used_meta":707768,"kb_avail":89897920,"statfs":{"total":95995035648,"available":92055470080,"internally_reserved":0,"allocated":3214794752,"data_stored":3212584930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":724755058},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502106,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2199296,"kb_used_data":1897016,"kb_used_omap":10,"kb_used_meta":302261,"kb_avail":91545856,"statfs":{"total":95995035648,"available":93742956544,"internally_reserved":0,"allocated":1942544384,"data_stored":1940335191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":309515967},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1912602624,"data_stored":1912602624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3203010560,"data_stored":3203006488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1239547904,"data_stored":1239547904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3233087488,"data_stored":3233087488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577062400,"data_stored":1577058328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3089633280,"data_stored":3089629208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":933494784,"data_stored":933494784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:07:11.906 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T15:07:11.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-13T15:07:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:11.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:11.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:11.998 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:11.998 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:12.024 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:12.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:12.154 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:12.162 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:07:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:07:11] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-13T15:07:12.179 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:12.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:12.306 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:12.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:12.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:12.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:12.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:12.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:12.719 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:12.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:07:12.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:12.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:12.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:12.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:12.987 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1896354322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:13.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:13.088 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:07:13.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:13.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1896354322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1896354322' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:13.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:13.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:13.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:13.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:13.530 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:13.554 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:13.570 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:07:13.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:13.677 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:13.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:13.804 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:13.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:13.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:13.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:14.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:14.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:14.064 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:14.138 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:13 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:14.138 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1896354322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:14.139 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:13 smithi172 ceph-mon[100696]: osdmap e406: 8 total, 8 up, 7 in 2024-09-13T15:07:14.139 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:14.139 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:14.139 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:13 smithi172 ceph-mon[100696]: pgmap v1313: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:07:14.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:14.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:14.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1896354322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:14.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[102314]: osdmap e406: 8 total, 8 up, 7 in 2024-09-13T15:07:14.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:14.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:14.183 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[102314]: pgmap v1313: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:07:14.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:14.183 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:14.184 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1896354322' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:14.184 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[107038]: osdmap e406: 8 total, 8 up, 7 in 2024-09-13T15:07:14.184 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:14.184 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:14.184 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:13 smithi154 ceph-mon[107038]: pgmap v1313: 89 pgs: 89 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 833 op/s 2024-09-13T15:07:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:14.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:14.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:14.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:14.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:14.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:14.591 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:14.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:14.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:14.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:14.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:14.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:14.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:15.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:15.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:15.132 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:15.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:15.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:15.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:15.397 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:15.411 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:15 smithi172 ceph-mon[100696]: pgmap v1314: 89 pgs: 89 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:07:15.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.543 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:15.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:15.561 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:15 smithi154 ceph-mon[107038]: pgmap v1314: 89 pgs: 89 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:07:15.563 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:15 smithi154 ceph-mon[102314]: pgmap v1314: 89 pgs: 89 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:07:15.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:15.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:15.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:15.802 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:15.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:15.920 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:15.920 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:15.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:15.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:16.031 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:16.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:16.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:16.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:16.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:16.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.344 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:16.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:16.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:16.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:16.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:16.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:16.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:16.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:07:16.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:07:16.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.764 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:16.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:16.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:16.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:16.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:16.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:17.012 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:17.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:17.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:17.254 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:17 smithi154 ceph-mon[107038]: pgmap v1315: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:07:17.254 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:17 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1183848320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:17.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:17 smithi154 ceph-mon[102314]: pgmap v1315: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:07:17.254 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:17 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1183848320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:17.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:17.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:17.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:17.424 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:17.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:17 smithi172 ceph-mon[100696]: pgmap v1315: 89 pgs: 89 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 824 op/s 2024-09-13T15:07:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:17 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1183848320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:17.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:07:17.531 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:07:17.532 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:07:17.532 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:07:17.532 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:07:17.532 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:07:17.532 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:07:17.532 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:07:17.532 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:07:17.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:17.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:17.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:17.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:17.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:17.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:17.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:17.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:18.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:18.087 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:18.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:18.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:18.330 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:18.357 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:18.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:18.485 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:18.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:18.611 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:18.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:18.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:18.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:18.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:18.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:19.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.036 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:19.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:19.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:19.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:19.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:19.287 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:19.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:19.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:19.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:19 smithi172 ceph-mon[100696]: pgmap v1316: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:07:19.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:19.537 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:19 smithi154 ceph-mon[102314]: pgmap v1316: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:07:19.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:19 smithi154 ceph-mon[107038]: pgmap v1316: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-09-13T15:07:19.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:19.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:19.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:19.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:19.821 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:19.821 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:19.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:19.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:19.991 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:19.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:20.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:20.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:20.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:20.235 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 9 pg_num to 26 2024-09-13T15:07:20.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:20.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:20.393 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:20.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1130368168' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:07:20.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1130368168' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:07:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1130368168' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-13T15:07:20.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:20.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:20.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:20.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:20.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:20.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:20.928 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:20.949 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:20.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:21.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:21.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:21.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:21.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:21.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:07:21.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:21.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:21.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:21.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.615 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:21.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:21.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1130368168' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T15:07:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[102314]: osdmap e407: 8 total, 8 up, 7 in 2024-09-13T15:07:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[102314]: pgmap v1318: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 799 op/s 2024-09-13T15:07:21.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T15:07:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1130368168' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T15:07:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[107038]: osdmap e407: 8 total, 8 up, 7 in 2024-09-13T15:07:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[107038]: pgmap v1318: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 799 op/s 2024-09-13T15:07:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T15:07:21.737 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1130368168' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-13T15:07:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:21 smithi172 ceph-mon[100696]: osdmap e407: 8 total, 8 up, 7 in 2024-09-13T15:07:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:21 smithi172 ceph-mon[100696]: pgmap v1318: 89 pgs: 89 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 799 op/s 2024-09-13T15:07:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-13T15:07:21.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:21.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:21.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:21.984 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:21.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:22.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:22.108 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:22.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:22.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:07:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:07:21] "GET /metrics HTTP/1.1" 200 36601 "" "Prometheus/2.51.0" 2024-09-13T15:07:22.232 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:22.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:22.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:22.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:22.497 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:22.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:22.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:22 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T15:07:22.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:22 smithi154 ceph-mon[102314]: osdmap e408: 8 total, 8 up, 7 in 2024-09-13T15:07:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:22 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T15:07:22.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:22 smithi154 ceph-mon[107038]: osdmap e408: 8 total, 8 up, 7 in 2024-09-13T15:07:22.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:22 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-13T15:07:22.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:22 smithi172 ceph-mon[100696]: osdmap e408: 8 total, 8 up, 7 in 2024-09-13T15:07:22.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:22.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:22.902 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:22.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:23.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:23.027 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:23.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:23.154 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:23.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:23.280 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:23.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:23.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:23.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[102314]: osdmap e409: 8 total, 8 up, 7 in 2024-09-13T15:07:23.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[102314]: 9.3 scrub starts 2024-09-13T15:07:23.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[102314]: 9.0 scrub starts 2024-09-13T15:07:23.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[102314]: pgmap v1321: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail 2024-09-13T15:07:23.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[107038]: osdmap e409: 8 total, 8 up, 7 in 2024-09-13T15:07:23.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[107038]: 9.3 scrub starts 2024-09-13T15:07:23.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[107038]: 9.0 scrub starts 2024-09-13T15:07:23.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:23 smithi154 ceph-mon[107038]: pgmap v1321: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail 2024-09-13T15:07:23.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:23.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:23.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:23.697 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:23 smithi172 ceph-mon[100696]: osdmap e409: 8 total, 8 up, 7 in 2024-09-13T15:07:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:23 smithi172 ceph-mon[100696]: 9.3 scrub starts 2024-09-13T15:07:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:23 smithi172 ceph-mon[100696]: 9.0 scrub starts 2024-09-13T15:07:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:23 smithi172 ceph-mon[100696]: pgmap v1321: 99 pgs: 2 peering, 10 unknown, 87 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail 2024-09-13T15:07:23.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:23.824 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:23.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:23.950 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:23.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:24.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:24.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:24.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.228 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:24.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:24.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:24.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:24.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:24 smithi154 ceph-mon[102314]: 9.5 scrub starts 2024-09-13T15:07:24.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:24 smithi154 ceph-mon[102314]: 9.8 scrub starts 2024-09-13T15:07:24.397 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:24 smithi154 ceph-mon[102314]: 9.14 scrub starts 2024-09-13T15:07:24.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:24 smithi154 ceph-mon[107038]: 9.5 scrub starts 2024-09-13T15:07:24.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:24 smithi154 ceph-mon[107038]: 9.8 scrub starts 2024-09-13T15:07:24.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:24 smithi154 ceph-mon[107038]: 9.14 scrub starts 2024-09-13T15:07:24.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:24 smithi172 ceph-mon[100696]: 9.5 scrub starts 2024-09-13T15:07:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:24 smithi172 ceph-mon[100696]: 9.8 scrub starts 2024-09-13T15:07:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:24 smithi172 ceph-mon[100696]: 9.14 scrub starts 2024-09-13T15:07:24.492 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:24.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:24.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:24.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:24.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:24.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:24.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:24.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:25.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:25.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:25.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:25.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:25.284 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:07:25.298 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:25.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:25.339 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[102314]: 9.9 scrub starts 2024-09-13T15:07:25.339 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[102314]: 9.18 scrub starts 2024-09-13T15:07:25.339 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2030500788' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[102314]: 9.4 scrub starts 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[102314]: pgmap v1322: 99 pgs: 2 peering, 8 unknown, 89 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[107038]: 9.9 scrub starts 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[107038]: 9.18 scrub starts 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2030500788' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[107038]: 9.4 scrub starts 2024-09-13T15:07:25.340 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:25 smithi154 ceph-mon[107038]: pgmap v1322: 99 pgs: 2 peering, 8 unknown, 89 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail 2024-09-13T15:07:25.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:25.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:25.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:25.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:25.629 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:25 smithi172 ceph-mon[100696]: 9.9 scrub starts 2024-09-13T15:07:25.629 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:25 smithi172 ceph-mon[100696]: 9.18 scrub starts 2024-09-13T15:07:25.629 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:25 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2030500788' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:07:25.629 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:07:25.630 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:25 smithi172 ceph-mon[100696]: 9.4 scrub starts 2024-09-13T15:07:25.630 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:25 smithi172 ceph-mon[100696]: pgmap v1322: 99 pgs: 2 peering, 8 unknown, 89 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail 2024-09-13T15:07:25.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:25.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:25.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:25.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:25.893 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:25.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:25.995 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:26.020 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:26.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:26.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:26.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:26.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:26.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:26.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:26.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:26.576 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:26.622 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:26 smithi172 ceph-mon[100696]: 9.15 scrub starts 2024-09-13T15:07:26.623 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:26 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:26.623 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2030500788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:26.623 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:26 smithi172 ceph-mon[100696]: osdmap e410: 8 total, 8 up, 7 in 2024-09-13T15:07:26.623 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:26 smithi172 ceph-mon[100696]: 9.13 scrub starts 2024-09-13T15:07:26.623 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:26 smithi172 ceph-mon[100696]: 9.10 scrub starts 2024-09-13T15:07:26.623 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[102314]: 9.15 scrub starts 2024-09-13T15:07:26.623 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2030500788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[102314]: osdmap e410: 8 total, 8 up, 7 in 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[102314]: 9.13 scrub starts 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[102314]: 9.10 scrub starts 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[107038]: 9.15 scrub starts 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2030500788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[107038]: osdmap e410: 8 total, 8 up, 7 in 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[107038]: 9.13 scrub starts 2024-09-13T15:07:26.624 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:26 smithi154 ceph-mon[107038]: 9.10 scrub starts 2024-09-13T15:07:26.662 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:26.671 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:07:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:07:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:140054 9.15 9:a8524f21:::benchmark_data_smithi154_291684_object39593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:26.650420+0000", 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.017026289999999999, 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017057803999999999, 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:07:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 140054 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650420+0000", 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650420+0000", 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650421+0000", 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0449999999999999e-06 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650426+0000", 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6890000000000003e-06 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:07:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650438+0000", 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2648e-05 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650445+0000", 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3330000000000002e-06 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650493+0000", 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7895999999999997e-05 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650611+0000", 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011842599999999999 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.650920+0000", 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030907500000000002 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.652744+0000", 2024-09-13T15:07:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018238849999999999 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.663739+0000", 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010995153000000001 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:140066 9.9 9:9552e5fc:::benchmark_data_smithi154_291684_object39605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:26.654907+0000", 2024-09-13T15:07:26.677 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.012539428, 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012687281, 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 140066 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:07:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.654907+0000", 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.654907+0000", 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.654907+0000", 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1900000000000003e-07 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.654910+0000", 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8899999999999999e-06 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.654918+0000", 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7619999999999995e-06 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.654924+0000", 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.049e-06 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.657353+0000", 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024283740000000001 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.657468+0000", 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115443 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.657831+0000", 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00036252700000000002 2024-09-13T15:07:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.663714+0000", 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058838149999999997 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664680+0000", 2024-09-13T15:07:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096517300000000005 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:140070 9.15 9:aec54721:::benchmark_data_smithi154_291684_object39609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:26.664513+0000", 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.002933228, 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003151282, 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:07:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 140070 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664513+0000", 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:07:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664513+0000", 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664515+0000", 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.517e-06 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664519+0000", 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7859999999999996e-06 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664530+0000", 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1005999999999999e-05 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664538+0000", 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0330000000000003e-06 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664787+0000", 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024950699999999999 2024-09-13T15:07:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.664998+0000", 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021050299999999999 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.665311+0000", 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031383799999999999 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.45983.0:140070 9.15 e410/408)", 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:26.666654+0000", 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.00079195999999999997, 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001063998, 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.666654+0000", 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.666654+0000", 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.666655+0000", 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7400000000000002e-07 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.666659+0000", 2024-09-13T15:07:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6550000000000002e-06 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.666672+0000", 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3455000000000001e-05 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:26.666681+0000", 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7180000000000002e-06 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:07:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T15:07:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 4 2024-09-13T15:07:26.691 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:07:26.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:26.752 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:26.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:26.883 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:26.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:26.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:27.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:27.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:27.164 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:27.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:27.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:27.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:27.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: 9.19 scrub starts 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: 9.19 scrub ok 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: 9.13 scrub ok 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: 9.11 scrub starts 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: 9.10 scrub ok 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: 9.7 scrub starts 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: pgmap v1324: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 267 op/s 2024-09-13T15:07:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: 9.19 scrub starts 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: 9.19 scrub ok 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: 9.13 scrub ok 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: 9.11 scrub starts 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: 9.10 scrub ok 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: 9.7 scrub starts 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: pgmap v1324: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 267 op/s 2024-09-13T15:07:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T15:07:27.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:27.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:27.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:27.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: 9.19 scrub starts 2024-09-13T15:07:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: 9.19 scrub ok 2024-09-13T15:07:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: 9.13 scrub ok 2024-09-13T15:07:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: 9.11 scrub starts 2024-09-13T15:07:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: 9.10 scrub ok 2024-09-13T15:07:27.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: 9.7 scrub starts 2024-09-13T15:07:27.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: pgmap v1324: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 267 op/s 2024-09-13T15:07:27.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-13T15:07:27.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:27.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:27.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:27.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:27.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:28.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:28.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:28.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.188 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:28.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:28.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:28.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:28.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:28.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:07:28.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:28.435 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:28.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:28.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:28.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:28.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:28.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: 9.11 scrub ok 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: 9.7 scrub ok 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: osdmap e411: 8 total, 8 up, 7 in 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: 9.18 scrub starts 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: 9.6 scrub starts 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: 9.6 scrub ok 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: 9.18 scrub ok 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: 9.11 scrub ok 2024-09-13T15:07:28.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: 9.7 scrub ok 2024-09-13T15:07:28.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T15:07:28.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: osdmap e411: 8 total, 8 up, 7 in 2024-09-13T15:07:28.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: 9.18 scrub starts 2024-09-13T15:07:28.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: 9.6 scrub starts 2024-09-13T15:07:28.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: 9.6 scrub ok 2024-09-13T15:07:28.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: 9.18 scrub ok 2024-09-13T15:07:28.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: 9.11 scrub ok 2024-09-13T15:07:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: 9.7 scrub ok 2024-09-13T15:07:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-13T15:07:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: osdmap e411: 8 total, 8 up, 7 in 2024-09-13T15:07:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: 9.18 scrub starts 2024-09-13T15:07:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: 9.6 scrub starts 2024-09-13T15:07:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: 9.6 scrub ok 2024-09-13T15:07:28.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: 9.18 scrub ok 2024-09-13T15:07:28.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:28.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:28.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:28.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:28.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:28.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:29.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:29.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:29.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:29.176 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:29.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:29.300 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:29.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:29.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:29.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:29.562 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: 9.9 scrub starts 2024-09-13T15:07:29.562 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: 9.9 scrub ok 2024-09-13T15:07:29.562 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: 9.4 scrub starts 2024-09-13T15:07:29.562 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: 9.4 scrub ok 2024-09-13T15:07:29.563 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: osdmap e412: 8 total, 8 up, 7 in 2024-09-13T15:07:29.563 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: 9.8 scrub starts 2024-09-13T15:07:29.563 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: 9.8 scrub ok 2024-09-13T15:07:29.563 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: 9.14 scrub starts 2024-09-13T15:07:29.563 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: pgmap v1327: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail 2024-09-13T15:07:29.563 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:29 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T15:07:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.564 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:29.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:29.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:29.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: 9.9 scrub starts 2024-09-13T15:07:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: 9.9 scrub ok 2024-09-13T15:07:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: 9.4 scrub starts 2024-09-13T15:07:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: 9.4 scrub ok 2024-09-13T15:07:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: osdmap e412: 8 total, 8 up, 7 in 2024-09-13T15:07:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: 9.8 scrub starts 2024-09-13T15:07:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: 9.8 scrub ok 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: 9.14 scrub starts 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: pgmap v1327: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: 9.9 scrub starts 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: 9.9 scrub ok 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: 9.4 scrub starts 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: 9.4 scrub ok 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: osdmap e412: 8 total, 8 up, 7 in 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: 9.8 scrub starts 2024-09-13T15:07:29.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: 9.8 scrub ok 2024-09-13T15:07:29.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: 9.14 scrub starts 2024-09-13T15:07:29.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: pgmap v1327: 99 pgs: 2 active+clean+scrubbing, 97 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail 2024-09-13T15:07:29.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:29 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-13T15:07:29.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:07:29.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:29.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:29.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:29.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:30.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:30.101 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:07:30.101 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:07:30.102 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T15:07:30.102 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.266353 2024-09-13T15:07:30.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 0.26635313614851097 2024-09-13T15:07:30.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:30.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:30.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:30.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: 9.15 scrub starts 2024-09-13T15:07:30.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: 9.15 scrub ok 2024-09-13T15:07:30.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: 9.14 scrub ok 2024-09-13T15:07:30.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T15:07:30.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 410'8563 2024-09-13T15:07:30.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: osdmap e413: 8 total, 8 up, 7 in 2024-09-13T15:07:30.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 410'8563 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: 9.2 scrub starts 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: 9.0 scrub starts 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[102314]: osdmap e414: 8 total, 8 up, 7 in 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: 9.15 scrub starts 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: 9.15 scrub ok 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: 9.14 scrub ok 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 410'8563 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: osdmap e413: 8 total, 8 up, 7 in 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 410'8563 2024-09-13T15:07:30.298 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: 9.2 scrub starts 2024-09-13T15:07:30.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: 9.0 scrub starts 2024-09-13T15:07:30.299 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:30 smithi154 ceph-mon[107038]: osdmap e414: 8 total, 8 up, 7 in 2024-09-13T15:07:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:30.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:30.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:30.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:30.546 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: 9.15 scrub starts 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: 9.15 scrub ok 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: 9.14 scrub ok 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 410'8563 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: osdmap e413: 8 total, 8 up, 7 in 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 410'8563 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: 9.2 scrub starts 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: 9.0 scrub starts 2024-09-13T15:07:30.547 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:30 smithi172 ceph-mon[100696]: osdmap e414: 8 total, 8 up, 7 in 2024-09-13T15:07:30.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:30.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:30.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:30.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:30.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:30.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:30.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:30.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:30.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:30.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:31.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:31.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:31.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:31.209 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:31.226 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:31.247 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:31.328 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:31.355 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:31.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:31.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:31.550 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.5 scrub starts 2024-09-13T15:07:31.550 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.5 scrub ok 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.2 scrub ok 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.0 scrub ok 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.12 scrub starts 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.16 scrub starts 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.12 scrub ok 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: 9.16 scrub ok 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: osdmap e415: 8 total, 8 up, 7 in 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[102314]: pgmap v1331: 99 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 97 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail 2024-09-13T15:07:31.551 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.5 scrub starts 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.5 scrub ok 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.2 scrub ok 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.0 scrub ok 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.12 scrub starts 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.16 scrub starts 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.12 scrub ok 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: 9.16 scrub ok 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: osdmap e415: 8 total, 8 up, 7 in 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-mon[107038]: pgmap v1331: 99 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 97 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail 2024-09-13T15:07:31.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:31.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.5 scrub starts 2024-09-13T15:07:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.5 scrub ok 2024-09-13T15:07:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.2 scrub ok 2024-09-13T15:07:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.0 scrub ok 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.12 scrub starts 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.16 scrub starts 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.12 scrub ok 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: 9.16 scrub ok 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: osdmap e415: 8 total, 8 up, 7 in 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:31 smithi172 ceph-mon[100696]: pgmap v1331: 99 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 97 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail 2024-09-13T15:07:31.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:31.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:31.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:31.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:31.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:32.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:32.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:32.101 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:07:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:07:31] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-13T15:07:32.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:32.196 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:07:32.197 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:07:32.212 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:32.242 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:07:32.242 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:07:32.282 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:07:32.282 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:07:32.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:32.315 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:07:32.315 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:07:32.343 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:32.347 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:07:32.347 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:07:32.381 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:07:32.381 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[102314]: 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 411'8682 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[102314]: 9.3 scrub starts 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2328946685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[107038]: 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 411'8682 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[107038]: 9.3 scrub starts 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:32.382 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2328946685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:32.413 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:07:32.414 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:07:32.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:32.452 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:07:32.452 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:07:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:32.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:32.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:32.608 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:32 smithi172 ceph-mon[100696]: 9.11 starting backfill to osd.1 from (0'0,0'0] MAX to 411'8682 2024-09-13T15:07:32.609 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:32 smithi172 ceph-mon[100696]: 9.3 scrub starts 2024-09-13T15:07:32.609 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:32.609 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:32 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:32.609 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2328946685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:07:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:32.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:32.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:32.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:32.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:32.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:32.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:33.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:33.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:33.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.198 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:33.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:33.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:33.323 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:33.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:33.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:33 smithi154 ceph-mon[102314]: 9.3 scrub ok 2024-09-13T15:07:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:33 smithi154 ceph-mon[102314]: 9.1 scrub starts 2024-09-13T15:07:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:33 smithi154 ceph-mon[102314]: pgmap v1332: 99 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 97 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2610/177885 objects misplaced (1.467%) 2024-09-13T15:07:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:33 smithi154 ceph-mon[107038]: 9.3 scrub ok 2024-09-13T15:07:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:33 smithi154 ceph-mon[107038]: 9.1 scrub starts 2024-09-13T15:07:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:33 smithi154 ceph-mon[107038]: pgmap v1332: 99 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 97 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2610/177885 objects misplaced (1.467%) 2024-09-13T15:07:33.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:33.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:33.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:33.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:33 smithi172 ceph-mon[100696]: 9.3 scrub ok 2024-09-13T15:07:33.668 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:33 smithi172 ceph-mon[100696]: 9.1 scrub starts 2024-09-13T15:07:33.668 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:33 smithi172 ceph-mon[100696]: pgmap v1332: 99 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 97 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2610/177885 objects misplaced (1.467%) 2024-09-13T15:07:33.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:33.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:33.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:33.901 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:33.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:34.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:34.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:34.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:34.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:34.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:34.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:34.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:34.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:34 smithi154 ceph-mon[102314]: 9.1 scrub ok 2024-09-13T15:07:34.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:34 smithi154 ceph-mon[107038]: 9.1 scrub ok 2024-09-13T15:07:34.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:34.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:34.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:34.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:34 smithi172 ceph-mon[100696]: 9.1 scrub ok 2024-09-13T15:07:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:34.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:34.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:34.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:34.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:34.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:35.009 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:35.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:35.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:35.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:35.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:35.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:35.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:35.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:35 smithi154 ceph-mon[102314]: pgmap v1333: 99 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 97 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2610/175053 objects misplaced (1.491%) 2024-09-13T15:07:35.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:35 smithi154 ceph-mon[107038]: pgmap v1333: 99 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 97 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2610/175053 objects misplaced (1.491%) 2024-09-13T15:07:35.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:35.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:35.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:35.692 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:35 smithi172 ceph-mon[100696]: pgmap v1333: 99 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 97 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 2610/175053 objects misplaced (1.491%) 2024-09-13T15:07:35.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:35.827 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:35.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:35.952 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:35.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:36.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:36.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:36.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:36.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:36.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:36.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:36.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:36.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:36.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:36.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:36.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:36.733 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:36.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:36.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:36.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:36.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:37.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:37.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:37.146 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:37.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:37.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:37.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:37.399 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:37.410 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:07:37.410 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:07:37.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:37.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:37.553 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:37.553 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:37.553 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[102314]: pgmap v1334: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2897/164514 objects misplaced (1.761%) 2024-09-13T15:07:37.553 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T15:07:37.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:37.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:37.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[107038]: pgmap v1334: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2897/164514 objects misplaced (1.761%) 2024-09-13T15:07:37.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T15:07:37.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:37.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:37 smithi172 ceph-mon[100696]: pgmap v1334: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2897/164514 objects misplaced (1.761%) 2024-09-13T15:07:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-13T15:07:37.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:37.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:37.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:37.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:37.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:38.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:38.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:38.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:38.196 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:38.293 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:07:38.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:38.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:38.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:38.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:38.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:38.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:38.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:38.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T15:07:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[102314]: osdmap e416: 8 total, 8 up, 7 in 2024-09-13T15:07:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/916604932' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:38.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T15:07:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[107038]: osdmap e416: 8 total, 8 up, 7 in 2024-09-13T15:07:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/916604932' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:38 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-13T15:07:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:38 smithi172 ceph-mon[100696]: osdmap e416: 8 total, 8 up, 7 in 2024-09-13T15:07:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/916604932' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:38 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:07:38.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:38.826 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:38.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:38.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:38.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:39.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:39.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:39.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:39.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:39.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:39.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:39.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:39.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:39.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:39.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:39.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[102314]: osdmap e417: 8 total, 8 up, 7 in 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[102314]: 9.17 deep-scrub starts 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[102314]: pgmap v1337: 99 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfilling, 96 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2897/160302 objects misplaced (1.807%) 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:07:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:39.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[107038]: osdmap e417: 8 total, 8 up, 7 in 2024-09-13T15:07:39.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[107038]: 9.17 deep-scrub starts 2024-09-13T15:07:39.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[107038]: pgmap v1337: 99 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfilling, 96 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2897/160302 objects misplaced (1.807%) 2024-09-13T15:07:39.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:39 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T15:07:39.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:39 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:07:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:39 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:07:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:39 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:07:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:39 smithi172 ceph-mon[100696]: osdmap e417: 8 total, 8 up, 7 in 2024-09-13T15:07:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:39 smithi172 ceph-mon[100696]: 9.17 deep-scrub starts 2024-09-13T15:07:39.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:39 smithi172 ceph-mon[100696]: pgmap v1337: 99 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfilling, 96 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 2897/160302 objects misplaced (1.807%) 2024-09-13T15:07:39.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:39 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-13T15:07:39.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:39.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:39.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:39.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:39.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:40.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:40.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:40.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:40.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:40.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:40.366 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:40.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:40.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:40.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:40.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T15:07:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[102314]: osdmap e418: 8 total, 8 up, 7 in 2024-09-13T15:07:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[102314]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 415'8855 2024-09-13T15:07:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[102314]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 415'8855 2024-09-13T15:07:40.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[102314]: osdmap e419: 8 total, 8 up, 7 in 2024-09-13T15:07:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T15:07:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[107038]: osdmap e418: 8 total, 8 up, 7 in 2024-09-13T15:07:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[107038]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 415'8855 2024-09-13T15:07:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[107038]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 415'8855 2024-09-13T15:07:40.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:40 smithi154 ceph-mon[107038]: osdmap e419: 8 total, 8 up, 7 in 2024-09-13T15:07:40.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:40 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-13T15:07:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:40 smithi172 ceph-mon[100696]: osdmap e418: 8 total, 8 up, 7 in 2024-09-13T15:07:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:40 smithi172 ceph-mon[100696]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 415'8855 2024-09-13T15:07:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:40 smithi172 ceph-mon[100696]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 415'8855 2024-09-13T15:07:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:40 smithi172 ceph-mon[100696]: osdmap e419: 8 total, 8 up, 7 in 2024-09-13T15:07:40.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:40.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:40.885 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:40.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:40.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:41.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:41.110 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.134 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:41.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:41.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:41.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:41.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.392 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:41.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:41.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:41.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:41.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:41.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:41.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:41.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:41.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:41.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:41.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:42.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:42.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:42.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:42.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:42.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[102314]: 9.13 starting backfill to osd.0 from (0'0,0'0] MAX to 416'8959 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[102314]: osdmap e420: 8 total, 8 up, 7 in 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[102314]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 416'8959 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[102314]: pgmap v1341: 99 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 96 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 529 op/s; 1530/158394 objects misplaced (0.966%); 21 MiB/s, 342 objects/s recovering 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[107038]: 9.13 starting backfill to osd.0 from (0'0,0'0] MAX to 416'8959 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[107038]: osdmap e420: 8 total, 8 up, 7 in 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[107038]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 416'8959 2024-09-13T15:07:42.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[107038]: pgmap v1341: 99 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 96 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 529 op/s; 1530/158394 objects misplaced (0.966%); 21 MiB/s, 342 objects/s recovering 2024-09-13T15:07:42.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:42 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T15:07:42.222 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:07:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:07:41] "GET /metrics HTTP/1.1" 200 36598 "" "Prometheus/2.51.0" 2024-09-13T15:07:42.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:42.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:42.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:42.413 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:42 smithi172 ceph-mon[100696]: 9.13 starting backfill to osd.0 from (0'0,0'0] MAX to 416'8959 2024-09-13T15:07:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:42 smithi172 ceph-mon[100696]: osdmap e420: 8 total, 8 up, 7 in 2024-09-13T15:07:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:42 smithi172 ceph-mon[100696]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 416'8959 2024-09-13T15:07:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:42 smithi172 ceph-mon[100696]: pgmap v1341: 99 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 96 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 529 op/s; 1530/158394 objects misplaced (0.966%); 21 MiB/s, 342 objects/s recovering 2024-09-13T15:07:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:42 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T15:07:42.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:42.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:42.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:42.668 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:42.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:42.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:42.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:42.928 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:42.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:42.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:43.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:43.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:43.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:43.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:43.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:43.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:43.391 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T15:07:43.392 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[102314]: osdmap e421: 8 total, 8 up, 7 in 2024-09-13T15:07:43.392 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:43.392 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:43.392 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T15:07:43.392 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[107038]: osdmap e421: 8 total, 8 up, 7 in 2024-09-13T15:07:43.392 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:43.392 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:43.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:43.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T15:07:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:43 smithi172 ceph-mon[100696]: osdmap e421: 8 total, 8 up, 7 in 2024-09-13T15:07:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:43.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:43.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:43.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:43.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:43.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:43.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:43.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:43.989 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:43.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:44.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:44.118 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:44.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:44.244 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:44.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:44.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:44 smithi172 ceph-mon[100696]: pgmap v1343: 99 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 2 active+remapped+backfilling, 95 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 3806/155850 objects misplaced (2.442%); 29 MiB/s, 457 objects/s recovering 2024-09-13T15:07:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-13T15:07:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:44 smithi154 ceph-mon[102314]: pgmap v1343: 99 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 2 active+remapped+backfilling, 95 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 3806/155850 objects misplaced (2.442%); 29 MiB/s, 457 objects/s recovering 2024-09-13T15:07:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-13T15:07:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:44 smithi154 ceph-mon[107038]: pgmap v1343: 99 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 2 active+remapped+backfilling, 95 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 370 op/s; 3806/155850 objects misplaced (2.442%); 29 MiB/s, 457 objects/s recovering 2024-09-13T15:07:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-13T15:07:44.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:44.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:44.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:44.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:44.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:44.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:44.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:44.891 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:44.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:44.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:45.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:45.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:45.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:45.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:45.264 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:45.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.389 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:45.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-13T15:07:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[102314]: osdmap e422: 8 total, 8 up, 7 in 2024-09-13T15:07:45.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[102314]: 9.17 deep-scrub ok 2024-09-13T15:07:45.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[102314]: osdmap e423: 8 total, 8 up, 7 in 2024-09-13T15:07:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-13T15:07:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[107038]: osdmap e422: 8 total, 8 up, 7 in 2024-09-13T15:07:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[107038]: 9.17 deep-scrub ok 2024-09-13T15:07:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:45 smithi154 ceph-mon[107038]: osdmap e423: 8 total, 8 up, 7 in 2024-09-13T15:07:45.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-13T15:07:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:45 smithi172 ceph-mon[100696]: osdmap e422: 8 total, 8 up, 7 in 2024-09-13T15:07:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:45 smithi172 ceph-mon[100696]: 9.17 deep-scrub ok 2024-09-13T15:07:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:45 smithi172 ceph-mon[100696]: osdmap e423: 8 total, 8 up, 7 in 2024-09-13T15:07:45.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:45.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:45.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:45.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:45.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:45.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:45.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:45.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:45.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:46.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:46.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:46.138 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:46.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:46.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:46.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:46.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[102314]: pgmap v1346: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfilling, 95 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 447 op/s; 5432/152919 objects misplaced (3.552%); 19 MiB/s, 306 objects/s recovering 2024-09-13T15:07:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-13T15:07:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-13T15:07:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[102314]: osdmap e424: 8 total, 8 up, 7 in 2024-09-13T15:07:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[107038]: pgmap v1346: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfilling, 95 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 447 op/s; 5432/152919 objects misplaced (3.552%); 19 MiB/s, 306 objects/s recovering 2024-09-13T15:07:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-13T15:07:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-13T15:07:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:46 smithi154 ceph-mon[107038]: osdmap e424: 8 total, 8 up, 7 in 2024-09-13T15:07:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:46 smithi172 ceph-mon[100696]: pgmap v1346: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 active+remapped+backfilling, 95 active+clean; 3.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 447 op/s; 5432/152919 objects misplaced (3.552%); 19 MiB/s, 306 objects/s recovering 2024-09-13T15:07:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-13T15:07:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-13T15:07:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:46 smithi172 ceph-mon[100696]: osdmap e424: 8 total, 8 up, 7 in 2024-09-13T15:07:46.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:46.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:46.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:46.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:46.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:46.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:46.883 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:46.908 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:46.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:47.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:47.033 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:47.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:47.161 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:47.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:47.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:47.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:47.421 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:47 smithi154 ceph-mon[102314]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 421'8850 2024-09-13T15:07:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:47 smithi154 ceph-mon[102314]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 421'8850 2024-09-13T15:07:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:47 smithi154 ceph-mon[102314]: osdmap e425: 8 total, 8 up, 7 in 2024-09-13T15:07:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:47 smithi154 ceph-mon[107038]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 421'8850 2024-09-13T15:07:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:47 smithi154 ceph-mon[107038]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 421'8850 2024-09-13T15:07:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:47 smithi154 ceph-mon[107038]: osdmap e425: 8 total, 8 up, 7 in 2024-09-13T15:07:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:47 smithi172 ceph-mon[100696]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 421'8850 2024-09-13T15:07:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:47 smithi172 ceph-mon[100696]: 9.15 starting backfill to osd.4 from (0'0,0'0] MAX to 421'8850 2024-09-13T15:07:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:47 smithi172 ceph-mon[100696]: osdmap e425: 8 total, 8 up, 7 in 2024-09-13T15:07:47.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:47.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:47.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:47.676 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:47.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:47.802 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:47.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:47.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:47.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:48.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:48.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:48.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:48.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:48.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:48.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:48.359 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:48 smithi154 ceph-mon[107038]: pgmap v1349: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 95 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 8336/148899 objects misplaced (5.598%); 12 MiB/s, 187 objects/s recovering 2024-09-13T15:07:48.359 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:48 smithi154 ceph-mon[107038]: osdmap e426: 8 total, 8 up, 7 in 2024-09-13T15:07:48.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:48 smithi154 ceph-mon[102314]: pgmap v1349: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 95 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 8336/148899 objects misplaced (5.598%); 12 MiB/s, 187 objects/s recovering 2024-09-13T15:07:48.361 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:48 smithi154 ceph-mon[102314]: osdmap e426: 8 total, 8 up, 7 in 2024-09-13T15:07:48.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:07:48.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:48.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:48 smithi172 ceph-mon[100696]: pgmap v1349: 99 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 95 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 579 op/s; 8336/148899 objects misplaced (5.598%); 12 MiB/s, 187 objects/s recovering 2024-09-13T15:07:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:48 smithi172 ceph-mon[100696]: osdmap e426: 8 total, 8 up, 7 in 2024-09-13T15:07:48.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:48.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:48.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:48.767 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:48.869 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:48.894 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:48.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:48.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:48.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:49.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:49.133 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:07:49.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:49.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:49 smithi154 ceph-mon[102314]: pgmap v1351: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3642/146679 objects misplaced (2.483%); 26 MiB/s, 422 objects/s recovering 2024-09-13T15:07:49.206 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:49 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-13T15:07:49.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:49.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:49.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:49.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:49 smithi154 ceph-mon[107038]: pgmap v1351: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3642/146679 objects misplaced (2.483%); 26 MiB/s, 422 objects/s recovering 2024-09-13T15:07:49.468 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:49 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-13T15:07:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:49 smithi172 ceph-mon[100696]: pgmap v1351: 99 pgs: 2 active+remapped+backfilling, 97 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3642/146679 objects misplaced (2.483%); 26 MiB/s, 422 objects/s recovering 2024-09-13T15:07:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:49 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-13T15:07:49.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:49.521 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:49.623 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:49.648 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:49.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:49.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:07:49.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:49.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:49.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:49.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:49.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:07:50.004 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:50.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:50.067 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.5 primary-affinity to 0.266353 (8174552) 2024-09-13T15:07:50.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:50.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:50.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:50.279 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:50.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:50.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[102314]: osdmap e427: 8 total, 8 up, 7 in 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1018031736' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097} : dispatch 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097} : dispatch 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097}]': finished 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[102314]: osdmap e428: 8 total, 8 up, 7 in 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[107038]: osdmap e427: 8 total, 8 up, 7 in 2024-09-13T15:07:50.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1018031736' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097} : dispatch 2024-09-13T15:07:50.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097} : dispatch 2024-09-13T15:07:50.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097}]': finished 2024-09-13T15:07:50.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:50 smithi154 ceph-mon[107038]: osdmap e428: 8 total, 8 up, 7 in 2024-09-13T15:07:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:50 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-13T15:07:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:50 smithi172 ceph-mon[100696]: osdmap e427: 8 total, 8 up, 7 in 2024-09-13T15:07:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1018031736' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097} : dispatch 2024-09-13T15:07:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:50 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097} : dispatch 2024-09-13T15:07:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:50 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.26635313614851097}]': finished 2024-09-13T15:07:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:50 smithi172 ceph-mon[100696]: osdmap e428: 8 total, 8 up, 7 in 2024-09-13T15:07:50.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:50.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:50.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:50.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:50.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:50.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.776 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:50.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:50.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:50.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:50.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:51.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:51.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:51.124 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.147 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:51.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:51.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:51.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:51.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:51.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:51.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.521 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:51.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:51.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:51.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:51.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:51.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:51.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:51.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:51.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:52.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:52.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:52.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:52.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[102314]: osdmap e429: 8 total, 8 up, 7 in 2024-09-13T15:07:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[102314]: 9.13 continuing backfill to osd.0 from (384'2000,427'9100] 9:cfea006d:::benchmark_data_smithi154_291684_object54421:head to 427'9100 2024-09-13T15:07:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[102314]: 9.13 continuing backfill to osd.6 from (384'2000,427'9100] 9:cfea006d:::benchmark_data_smithi154_291684_object54421:head to 427'9100 2024-09-13T15:07:52.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[102314]: pgmap v1355: 99 pgs: 4 peering, 2 active+remapped+backfilling, 93 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3642/145263 objects misplaced (2.507%); 27 MiB/s, 427 objects/s recovering 2024-09-13T15:07:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[107038]: osdmap e429: 8 total, 8 up, 7 in 2024-09-13T15:07:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[107038]: 9.13 continuing backfill to osd.0 from (384'2000,427'9100] 9:cfea006d:::benchmark_data_smithi154_291684_object54421:head to 427'9100 2024-09-13T15:07:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[107038]: 9.13 continuing backfill to osd.6 from (384'2000,427'9100] 9:cfea006d:::benchmark_data_smithi154_291684_object54421:head to 427'9100 2024-09-13T15:07:52.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:52 smithi154 ceph-mon[107038]: pgmap v1355: 99 pgs: 4 peering, 2 active+remapped+backfilling, 93 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3642/145263 objects misplaced (2.507%); 27 MiB/s, 427 objects/s recovering 2024-09-13T15:07:52.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:07:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:07:51] "GET /metrics HTTP/1.1" 200 36622 "" "Prometheus/2.51.0" 2024-09-13T15:07:52.294 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:52.316 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:52.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:52.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:52 smithi172 ceph-mon[100696]: osdmap e429: 8 total, 8 up, 7 in 2024-09-13T15:07:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:52 smithi172 ceph-mon[100696]: 9.13 continuing backfill to osd.0 from (384'2000,427'9100] 9:cfea006d:::benchmark_data_smithi154_291684_object54421:head to 427'9100 2024-09-13T15:07:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:52 smithi172 ceph-mon[100696]: 9.13 continuing backfill to osd.6 from (384'2000,427'9100] 9:cfea006d:::benchmark_data_smithi154_291684_object54421:head to 427'9100 2024-09-13T15:07:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:52 smithi172 ceph-mon[100696]: pgmap v1355: 99 pgs: 4 peering, 2 active+remapped+backfilling, 93 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 3642/145263 objects misplaced (2.507%); 27 MiB/s, 427 objects/s recovering 2024-09-13T15:07:52.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:52.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:52.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:52.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:52.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:52.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:52.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:52.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:52.971 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:52.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:53.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:53.096 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:53.191 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:53 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:07:53.191 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:53 smithi154 ceph-mon[102314]: osdmap e430: 8 total, 8 up, 7 in 2024-09-13T15:07:53.191 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:53 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:07:53.191 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:53 smithi154 ceph-mon[107038]: osdmap e430: 8 total, 8 up, 7 in 2024-09-13T15:07:53.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:53.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:53.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:53.351 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:53.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:53.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:53 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:07:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:53 smithi172 ceph-mon[100696]: osdmap e430: 8 total, 8 up, 7 in 2024-09-13T15:07:53.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:53.601 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:53.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:53.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:53.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.853 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:53.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:53.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:53.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:53.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:54.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:54.102 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:54.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:54.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:54.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:54.352 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:54.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:54 smithi154 ceph-mon[102314]: osdmap e431: 8 total, 8 up, 7 in 2024-09-13T15:07:54.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:54 smithi154 ceph-mon[102314]: pgmap v1358: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 peering, 1 active+remapped+backfilling, 92 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 19/142947 objects degraded (0.013%); 3154/142947 objects misplaced (2.206%); 18 MiB/s, 283 objects/s recovering 2024-09-13T15:07:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:54 smithi154 ceph-mon[107038]: osdmap e431: 8 total, 8 up, 7 in 2024-09-13T15:07:54.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:54 smithi154 ceph-mon[107038]: pgmap v1358: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 peering, 1 active+remapped+backfilling, 92 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 19/142947 objects degraded (0.013%); 3154/142947 objects misplaced (2.206%); 18 MiB/s, 283 objects/s recovering 2024-09-13T15:07:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:54 smithi172 ceph-mon[100696]: osdmap e431: 8 total, 8 up, 7 in 2024-09-13T15:07:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:54 smithi172 ceph-mon[100696]: pgmap v1358: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 peering, 1 active+remapped+backfilling, 92 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 316 op/s; 19/142947 objects degraded (0.013%); 3154/142947 objects misplaced (2.206%); 18 MiB/s, 283 objects/s recovering 2024-09-13T15:07:54.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:54.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:54.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:54.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:54.755 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:54.857 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:54.879 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:54.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:54.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:55.004 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:55.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:55.129 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:55.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:55.256 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:55.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:07:55.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:07:55.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:55 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 19/142947 objects degraded (0.013%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:07:55.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:55 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 19/142947 objects degraded (0.013%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:07:55.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:55 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 19/142947 objects degraded (0.013%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:07:55.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:55.543 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:55.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:55.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:55.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:55.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:55.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:55.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:55.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:56.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:56.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:56.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:56.217 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:56.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:56 smithi154 ceph-mon[102314]: pgmap v1359: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 peering, 1 active+remapped+backfilling, 91 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 338 op/s; 19/141519 objects degraded (0.013%); 5974/141519 objects misplaced (4.221%); 14 MiB/s, 252 objects/s recovering 2024-09-13T15:07:56.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:56 smithi154 ceph-mon[107038]: pgmap v1359: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 peering, 1 active+remapped+backfilling, 91 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 338 op/s; 19/141519 objects degraded (0.013%); 5974/141519 objects misplaced (4.221%); 14 MiB/s, 252 objects/s recovering 2024-09-13T15:07:56.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:56.344 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:56.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:56.471 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:56 smithi172 ceph-mon[100696]: pgmap v1359: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 peering, 1 active+remapped+backfilling, 91 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 338 op/s; 19/141519 objects degraded (0.013%); 5974/141519 objects misplaced (4.221%); 14 MiB/s, 252 objects/s recovering 2024-09-13T15:07:56.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:56.598 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:56.700 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:56.721 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:56.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:56.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:56.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:56.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:56.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:57.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:57.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:57.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:57.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:57.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:57.402 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:57 smithi172 ceph-mon[100696]: pgmap v1360: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 583 op/s; 19/134616 objects degraded (0.014%); 4752/134616 objects misplaced (3.530%); 20 MiB/s, 337 objects/s recovering 2024-09-13T15:07:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:57 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T15:07:57.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:57.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:57.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:07:57.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:57.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:07:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:57 smithi154 ceph-mon[102314]: pgmap v1360: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 583 op/s; 19/134616 objects degraded (0.014%); 4752/134616 objects misplaced (3.530%); 20 MiB/s, 337 objects/s recovering 2024-09-13T15:07:57.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:57 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T15:07:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:57 smithi154 ceph-mon[107038]: pgmap v1360: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 583 op/s; 19/134616 objects degraded (0.014%); 4752/134616 objects misplaced (3.530%); 20 MiB/s, 337 objects/s recovering 2024-09-13T15:07:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:57 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-09-13T15:07:57.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:57.805 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:57.907 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:07:57.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:57.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:07:57.939 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:07:57.939 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:07:57.939 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-13T15:07:57.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd in 7 2024-09-13T15:07:58.032 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:58.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:58.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:58.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:58.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:58.333 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:58.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:58.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:58.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:58.614 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:58.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:07:58.739 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:07:58.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:07:58.867 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:58.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:07:58.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:59.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:59.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:59.144 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:59.246 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:59.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:59.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:59.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:59.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:07:59.418 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:59.418 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:07:59 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:59.418 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:59.418 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:07:59 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:07:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:07:59 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:07:59.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:59.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:59.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:07:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:59.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:07:59.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:07:59.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:07:59.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:07:59.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:07:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:07:59.957 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:07:59.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:00.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:00.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:00.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:00.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:00.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:00.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:00.434 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:08:00.434 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:08:00.435 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:00 smithi154 ceph-mon[102314]: pgmap v1361: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 543 op/s; 19/132213 objects degraded (0.014%); 4536/132213 objects misplaced (3.431%); 16 MiB/s, 291 objects/s recovering 2024-09-13T15:08:00.435 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:00 smithi154 ceph-mon[107038]: pgmap v1361: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 543 op/s; 19/132213 objects degraded (0.014%); 4536/132213 objects misplaced (3.431%); 16 MiB/s, 291 objects/s recovering 2024-09-13T15:08:00.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:00 smithi172 ceph-mon[100696]: pgmap v1361: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 543 op/s; 19/132213 objects degraded (0.014%); 4536/132213 objects misplaced (3.431%); 16 MiB/s, 291 objects/s recovering 2024-09-13T15:08:00.509 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:00.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:00.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:00.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:00.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:00.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:00.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:00.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:01.017 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:01.041 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:01.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:01.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:01.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:01.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:01.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:01.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:01 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19/132213 objects degraded (0.014%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:01 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19/132213 objects degraded (0.014%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:01.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:01 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19/132213 objects degraded (0.014%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:01.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:01.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:01.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:01.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:01.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:01.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:01.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:02.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:02.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:02.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:02.185 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:02 smithi154 ceph-mon[102314]: pgmap v1362: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 530 op/s; 19/128502 objects degraded (0.015%); 4536/128502 objects misplaced (3.530%); 6.5 MiB/s, 133 objects/s recovering 2024-09-13T15:08:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:02 smithi154 ceph-mon[107038]: pgmap v1362: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 530 op/s; 19/128502 objects degraded (0.015%); 4536/128502 objects misplaced (3.530%); 6.5 MiB/s, 133 objects/s recovering 2024-09-13T15:08:02.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:08:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:08:01] "GET /metrics HTTP/1.1" 200 36624 "" "Prometheus/2.51.0" 2024-09-13T15:08:02.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:02.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:02.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:02.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:02 smithi172 ceph-mon[100696]: pgmap v1362: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 530 op/s; 19/128502 objects degraded (0.015%); 4536/128502 objects misplaced (3.530%); 6.5 MiB/s, 133 objects/s recovering 2024-09-13T15:08:02.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:02.562 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:02.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:02.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:02.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:02.812 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:02.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:02.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:02.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:03.053 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:03.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:03.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:03.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:03.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:03.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:03.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:03.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:03 smithi172 ceph-mon[100696]: pgmap v1363: 99 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 679 op/s; 19/122409 objects degraded (0.016%); 4178/122409 objects misplaced (3.413%); 7.4 MiB/s, 144 objects/s recovering 2024-09-13T15:08:03.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:03.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:03.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:03.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:03 smithi154 ceph-mon[102314]: pgmap v1363: 99 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 679 op/s; 19/122409 objects degraded (0.016%); 4178/122409 objects misplaced (3.413%); 7.4 MiB/s, 144 objects/s recovering 2024-09-13T15:08:03.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:03 smithi154 ceph-mon[107038]: pgmap v1363: 99 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 679 op/s; 19/122409 objects degraded (0.016%); 4178/122409 objects misplaced (3.413%); 7.4 MiB/s, 144 objects/s recovering 2024-09-13T15:08:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:03.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:03.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:03.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:03.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:03.962 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:03.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:03.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:04.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:04.124 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:04.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:04.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:04.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:04.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:04.487 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:04.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:04.572 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:04.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:04.636 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:04.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:04.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:04.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:04.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:04.903 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:04.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:05.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:05.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:05.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:05.175 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:05.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:05.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:05.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:05.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:05.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:05 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19/122409 objects degraded (0.016%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:05.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:05 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19/122409 objects degraded (0.016%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:05 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19/122409 objects degraded (0.016%), 2 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:05.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:05.587 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:05.590 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:05.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:08:05.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:05.714 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:05.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:05.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:05.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:05.964 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:05.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:06.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:06.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:06.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:06.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:06.365 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:06.388 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:06 smithi154 ceph-mon[102314]: pgmap v1364: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 19/119616 objects degraded (0.016%); 3926/119616 objects misplaced (3.282%); 6.8 MiB/s, 146 objects/s recovering 2024-09-13T15:08:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:06 smithi154 ceph-mon[107038]: pgmap v1364: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 19/119616 objects degraded (0.016%); 3926/119616 objects misplaced (3.282%); 6.8 MiB/s, 146 objects/s recovering 2024-09-13T15:08:06.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:06.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:06 smithi172 ceph-mon[100696]: pgmap v1364: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 19/119616 objects degraded (0.016%); 3926/119616 objects misplaced (3.282%); 6.8 MiB/s, 146 objects/s recovering 2024-09-13T15:08:06.514 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:06.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:06.661 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:06.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:06.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:06.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:06.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:06.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:07.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:07.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:07.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:07.209 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:07.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:07.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:07.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:07.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:07.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:07.573 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:07 smithi154 ceph-mon[102314]: pgmap v1365: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 870 op/s; 19/110166 objects degraded (0.017%); 3548/110166 objects misplaced (3.221%); 8.0 MiB/s, 157 objects/s recovering 2024-09-13T15:08:07.573 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:07 smithi154 ceph-mon[107038]: pgmap v1365: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 870 op/s; 19/110166 objects degraded (0.017%); 3548/110166 objects misplaced (3.221%); 8.0 MiB/s, 157 objects/s recovering 2024-09-13T15:08:07.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:07.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.734 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:07.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:07 smithi172 ceph-mon[100696]: pgmap v1365: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 870 op/s; 19/110166 objects degraded (0.017%); 3548/110166 objects misplaced (3.221%); 8.0 MiB/s, 157 objects/s recovering 2024-09-13T15:08:07.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:07.859 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:07.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:07.984 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:07.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:08.086 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:08.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.269 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:08.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:08.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:08.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:08.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:08.510 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:08.514 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:08.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:08.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:08.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:08.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:08.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:08.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:08.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:08.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:09.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.108 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:09.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:09.210 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:09.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:09.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:09.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:09.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:09.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:09.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:09.522 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:09 smithi154 ceph-mon[102314]: pgmap v1366: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 751 op/s; 19/107565 objects degraded (0.018%); 3290/107565 objects misplaced (3.059%); 4.7 MiB/s, 121 objects/s recovering 2024-09-13T15:08:09.522 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:09 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3528216509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:09.523 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:09 smithi154 ceph-mon[107038]: pgmap v1366: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 751 op/s; 19/107565 objects degraded (0.018%); 3290/107565 objects misplaced (3.059%); 4.7 MiB/s, 121 objects/s recovering 2024-09-13T15:08:09.523 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:09 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3528216509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:09.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:09.641 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:08:09.641 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:08:09.641 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:08:09.642 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:08:09.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:08:09.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:08:09.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:08:09.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:08:09.642 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:08:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:08:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:08:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:08:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:08:09.643 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:08:09.643 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:08:09.643 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:08:09.645 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:09 smithi172 ceph-mon[100696]: pgmap v1366: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 751 op/s; 19/107565 objects degraded (0.018%); 3290/107565 objects misplaced (3.059%); 4.7 MiB/s, 121 objects/s recovering 2024-09-13T15:08:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:09 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3528216509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:09.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:09.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:09.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:09.925 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:09.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:10.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:10.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:10.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:10.195 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:10.297 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:10.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:10.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:10.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:10.579 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:10.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:10.664 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:10 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19/107565 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:10.664 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:10 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19/107565 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:10.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:10.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:10 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19/107565 objects degraded (0.018%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:10.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:10.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:10.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:10.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:11.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:11.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:11.130 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:11.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:11.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:11.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:11.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:11.486 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:11.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:11.544 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:11 smithi154 ceph-mon[102314]: pgmap v1367: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 787 op/s; 19/103857 objects degraded (0.018%); 3290/103857 objects misplaced (3.168%); 4.1 MiB/s, 97 objects/s recovering 2024-09-13T15:08:11.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:11 smithi154 ceph-mon[107038]: pgmap v1367: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 787 op/s; 19/103857 objects degraded (0.018%); 3290/103857 objects misplaced (3.168%); 4.1 MiB/s, 97 objects/s recovering 2024-09-13T15:08:11.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:11.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:11.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:11 smithi172 ceph-mon[100696]: pgmap v1367: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 787 op/s; 19/103857 objects degraded (0.018%); 3290/103857 objects misplaced (3.168%); 4.1 MiB/s, 97 objects/s recovering 2024-09-13T15:08:11.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:11.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:11.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:11.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:12.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:12.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:12.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:12.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:08:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:08:11] "GET /metrics HTTP/1.1" 200 36624 "" "Prometheus/2.51.0" 2024-09-13T15:08:12.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:12.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:12.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:12.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:12.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.533 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:12.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:12.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:12.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:12.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:12.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:12.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:12.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:12.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:13.074 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:13.096 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:13.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:13.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:13.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:13.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:13.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:13.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:13.547 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:13.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:13.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:13.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:13.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:13.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:13.957 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:13.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:14.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:14.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:14.198 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:14.222 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:14.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:14.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:14.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:14.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:14 smithi172 ceph-mon[100696]: pgmap v1368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 19/97251 objects degraded (0.020%); 2956/97251 objects misplaced (3.040%); 5.3 MiB/s, 125 objects/s recovering 2024-09-13T15:08:14.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:14 smithi154 ceph-mon[102314]: pgmap v1368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 19/97251 objects degraded (0.020%); 2956/97251 objects misplaced (3.040%); 5.3 MiB/s, 125 objects/s recovering 2024-09-13T15:08:14.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:14 smithi154 ceph-mon[107038]: pgmap v1368: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 867 op/s; 19/97251 objects degraded (0.020%); 2956/97251 objects misplaced (3.040%); 5.3 MiB/s, 125 objects/s recovering 2024-09-13T15:08:14.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:14.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:14.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:14.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:14.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:14.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:14.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:14.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:14.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:15.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:15.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:15.173 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:15 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19/97251 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:15.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:15.296 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:15.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:15.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:15.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:15 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19/97251 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:15.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:15 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19/97251 objects degraded (0.020%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:15.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:15.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:15.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:15.706 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:15.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:15.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:15.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:15.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:15.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:16.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:16.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.104 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:16.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:16.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:16.230 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:16 smithi172 ceph-mon[100696]: pgmap v1369: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 19/94578 objects degraded (0.020%); 2722/94578 objects misplaced (2.878%); 4.7 MiB/s, 130 objects/s recovering 2024-09-13T15:08:16.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:16.354 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:16.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:16.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:16 smithi154 ceph-mon[102314]: pgmap v1369: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 19/94578 objects degraded (0.020%); 2722/94578 objects misplaced (2.878%); 4.7 MiB/s, 130 objects/s recovering 2024-09-13T15:08:16.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:16 smithi154 ceph-mon[107038]: pgmap v1369: 99 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 19/94578 objects degraded (0.020%); 2722/94578 objects misplaced (2.878%); 4.7 MiB/s, 130 objects/s recovering 2024-09-13T15:08:16.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:16.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:16.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:16.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:16.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:16.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:16.876 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:16.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:16.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:17.004 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:17.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:17.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:17.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:17.271 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:17.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.417 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:17.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:17.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:17.524 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:17 smithi154 ceph-mon[107038]: pgmap v1370: 99 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 951 op/s; 19/85341 objects degraded (0.022%); 2622/85341 objects misplaced (3.072%); 4.0 MiB/s, 135 objects/s recovering 2024-09-13T15:08:17.527 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:17 smithi154 ceph-mon[102314]: pgmap v1370: 99 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 951 op/s; 19/85341 objects degraded (0.022%); 2622/85341 objects misplaced (3.072%); 4.0 MiB/s, 135 objects/s recovering 2024-09-13T15:08:17.540 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:17.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:17.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:17 smithi172 ceph-mon[100696]: pgmap v1370: 99 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 951 op/s; 19/85341 objects degraded (0.022%); 2622/85341 objects misplaced (3.072%); 4.0 MiB/s, 135 objects/s recovering 2024-09-13T15:08:17.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:17.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:17.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:17.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:17.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:18.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:18.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:18.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:18.243 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:18.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:18.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:18.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:18.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:18.619 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:18.643 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:18.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:18.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:18.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:18.923 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:18.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:19.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:19.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:19.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:19.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:19.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:19.332 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:19.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:19.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:19.504 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:19 smithi154 ceph-mon[102314]: pgmap v1371: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 19/82740 objects degraded (0.023%); 2386/82740 objects misplaced (2.884%); 3.1 MiB/s, 138 objects/s recovering 2024-09-13T15:08:19.504 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:19 smithi154 ceph-mon[107038]: pgmap v1371: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 19/82740 objects degraded (0.023%); 2386/82740 objects misplaced (2.884%); 3.1 MiB/s, 138 objects/s recovering 2024-09-13T15:08:19.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:19.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:19 smithi172 ceph-mon[100696]: pgmap v1371: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.7 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 19/82740 objects degraded (0.023%); 2386/82740 objects misplaced (2.884%); 3.1 MiB/s, 138 objects/s recovering 2024-09-13T15:08:19.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:19.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:19.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:19.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:19.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:20.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:20.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:20.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:20.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:20.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:20.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:20.320 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:20.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:20.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:20.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:20.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:20.702 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:20.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:08:20.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:20.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:20.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:20.906 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:08:20.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:20.973 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:20.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:21.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:21.099 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:21.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19/82740 objects degraded (0.023%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:21.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1696799749' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:21.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:21.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:21.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:21.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19/82740 objects degraded (0.023%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:21.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1696799749' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:21.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:21.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:21.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:21.224 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:21.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19/82740 objects degraded (0.023%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:21.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1696799749' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:21.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:08:21.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:08:21.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:08:21.243 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:21.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:21.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:21.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.510 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:21.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:21.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:21.637 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:21.738 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.762 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:21.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:21.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:21.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:21.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:22.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:22.044 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:22.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:22.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:08:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[102314]: osdmap e432: 8 total, 8 up, 7 in 2024-09-13T15:08:22.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[102314]: pgmap v1373: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 19/79254 objects degraded (0.024%); 2386/79254 objects misplaced (3.011%); 2.9 MiB/s, 133 objects/s recovering 2024-09-13T15:08:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:08:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[107038]: osdmap e432: 8 total, 8 up, 7 in 2024-09-13T15:08:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-mon[107038]: pgmap v1373: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 19/79254 objects degraded (0.024%); 2386/79254 objects misplaced (3.011%); 2.9 MiB/s, 133 objects/s recovering 2024-09-13T15:08:22.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:08:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:08:21] "GET /metrics HTTP/1.1" 200 36616 "" "Prometheus/2.51.0" 2024-09-13T15:08:22.237 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:08:22.237 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:08:22.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.237 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:176678 9.17 9:edf02d3d:::benchmark_data_smithi154_291684_object76217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:22.237 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:20.809264+0000", 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.101776065, 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.101805113, 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for degraded object", 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 176678 2024-09-13T15:08:22.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.809264+0000", 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.809264+0000", 2024-09-13T15:08:22.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.809265+0000", 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5039999999999999e-06 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.809272+0000", 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7530000000000004e-06 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.809288+0000", 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5548000000000001e-05 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.809297+0000", 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9099999999999994e-06 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.817795+0000", 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084985549999999997 2024-09-13T15:08:22.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.820176+0000", 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023805520000000002 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:176711 9.6 9:65f5981a:::benchmark_data_smithi154_291684_object76250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:20.901517+0000", 2024-09-13T15:08:22.242 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0095225799999999992, 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096154350000000003, 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 176711 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901517+0000", 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901517+0000", 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901519+0000", 2024-09-13T15:08:22.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2529999999999999e-06 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901524+0000", 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8930000000000004e-06 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901535+0000", 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1048e-05 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901543+0000", 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8550000000000008e-06 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901642+0000", 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8964999999999999e-05 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901760+0000", 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117945 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.902330+0000", 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056992299999999996 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.908295+0000", 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005965232 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:22.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.908341+0000", 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5609999999999999e-05 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:176712 9.17 9:ea0ab70c:::benchmark_data_smithi154_291684_object76251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:20.901569+0000", 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0094709949999999994, 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096321010000000006, 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:22.248 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for degraded object", 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 176712 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901569+0000", 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901569+0000", 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901569+0000", 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7599999999999997e-07 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901572+0000", 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.604e-06 2024-09-13T15:08:22.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901579+0000", 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6100000000000002e-06 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901584+0000", 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3279999999999996e-06 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.901647+0000", 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3343999999999995e-05 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.902493+0000", 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000845231 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPushReply(9.17 431/428 [PushReplyOp(9:ea0ab70c:::benchmark_data_smithi154_291684_object76251:head)])", 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:20.907434+0000", 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.003606251, 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003818568, 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.907434+0000", 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.907434+0000", 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.907435+0000", 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.319e-06 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.907443+0000", 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7430000000000002e-06 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.907459+0000", 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6629e-05 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.907470+0000", 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0667e-05 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:176713 9.6 9:6576cbaf:::benchmark_data_smithi154_291684_object76252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:20.910620+0000", 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.00041958300000000001, 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00067412799999999997, 2024-09-13T15:08:22.256 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 176713 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.910620+0000", 2024-09-13T15:08:22.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.910620+0000", 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.910622+0000", 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0059999999999999e-06 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.910629+0000", 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4699999999999999e-06 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.910645+0000", 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6056e-05 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:22.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.910655+0000", 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0073e-05 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.910949+0000", 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029413799999999999 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.911256+0000", 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030656800000000001 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.260 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:176664 9.17 9:ebd3d250:::benchmark_data_smithi154_291684_object76203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:20.782988+0000", 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.12805244199999999, 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.128377031, 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:22.261 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 176664 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.782988+0000", 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:22.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.782988+0000", 2024-09-13T15:08:22.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:22.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:22.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.782990+0000", 2024-09-13T15:08:22.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.029e-06 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.782998+0000", 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9589999999999995e-06 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.783028+0000", 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0840999999999998e-05 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.783038+0000", 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6909999999999992e-06 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.793906+0000", 2024-09-13T15:08:22.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010867691000000001 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.795703+0000", 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017976929999999999 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.812984+0000", 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017280897999999999 2024-09-13T15:08:22.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.813086+0000", 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102086 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.813654+0000", 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056777300000000004 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:22.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.824003+0000", 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010349086 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:20.875143+0000", 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051139786999999999 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T15:08:22.268 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 6 2024-09-13T15:08:22.269 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:08:22.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:21 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:22.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:21 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:08:22.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:21 smithi172 ceph-mon[100696]: osdmap e432: 8 total, 8 up, 7 in 2024-09-13T15:08:22.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:21 smithi172 ceph-mon[100696]: pgmap v1373: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 96 active+clean; 1.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 19/79254 objects degraded (0.024%); 2386/79254 objects misplaced (3.011%); 2.9 MiB/s, 133 objects/s recovering 2024-09-13T15:08:22.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:22.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:22.409 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:22.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:22.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:22.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:22.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:22.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:22.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:22.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:22.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:22.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:22.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:22.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:23.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:23.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:23.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:23.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:23.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:23.369 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:23.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:23.492 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:23.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:23.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:23.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:23 smithi154 ceph-mon[102314]: pgmap v1374: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.5 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 19/72957 objects degraded (0.026%); 2284/72957 objects misplaced (3.131%); 1.5 MiB/s, 146 objects/s recovering 2024-09-13T15:08:23.724 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:23 smithi154 ceph-mon[107038]: pgmap v1374: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.5 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 19/72957 objects degraded (0.026%); 2284/72957 objects misplaced (3.131%); 1.5 MiB/s, 146 objects/s recovering 2024-09-13T15:08:23.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:23 smithi172 ceph-mon[100696]: pgmap v1374: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.5 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 19/72957 objects degraded (0.026%); 2284/72957 objects misplaced (3.131%); 1.5 MiB/s, 146 objects/s recovering 2024-09-13T15:08:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:23.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:23.870 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:23.891 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:23.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:23.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:24.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:24.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:24.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:24.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:24.282 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:24.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:24.407 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:24.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:24.532 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:24.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.658 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:24.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:24.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:24.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:24.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:24.957 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:24.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:25.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:25.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:25.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:25.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:25.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:25.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:25.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:25.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:25 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19/72957 objects degraded (0.026%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:25.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:25 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19/72957 objects degraded (0.026%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:25.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:25 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19/72957 objects degraded (0.026%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:25.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:25.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:25.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:25.730 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:25.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:25.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:25.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:25.992 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:25.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:26.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:26.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:26.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:26 smithi172 ceph-mon[100696]: pgmap v1375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 19/70527 objects degraded (0.027%); 2044/70527 objects misplaced (2.898%); 1.5 MiB/s, 148 objects/s recovering 2024-09-13T15:08:26.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:26.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:26.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:26.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:26 smithi154 ceph-mon[102314]: pgmap v1375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 19/70527 objects degraded (0.027%); 2044/70527 objects misplaced (2.898%); 1.5 MiB/s, 148 objects/s recovering 2024-09-13T15:08:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:26 smithi154 ceph-mon[107038]: pgmap v1375: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 19/70527 objects degraded (0.027%); 2044/70527 objects misplaced (2.898%); 1.5 MiB/s, 148 objects/s recovering 2024-09-13T15:08:26.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:26.520 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:26.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:26.665 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:26.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:26.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:26.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:26.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:26.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:27.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:27.071 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:27.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:27.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:27.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:27.338 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:27.440 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:27.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:27.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:27.590 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:27.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:27.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:27.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:27.844 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:27.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:27.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:27.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:28.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:28.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:28.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[102314]: osdmap e433: 8 total, 8 up, 7 in 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[102314]: pgmap v1377: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.3 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 19/62001 objects degraded (0.031%); 1934/62001 objects misplaced (3.119%); 960 KiB/s, 189 objects/s recovering 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[107038]: osdmap e433: 8 total, 8 up, 7 in 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[107038]: pgmap v1377: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.3 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 19/62001 objects degraded (0.031%); 1934/62001 objects misplaced (3.119%); 960 KiB/s, 189 objects/s recovering 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:28.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:28.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:28.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:28.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:28.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:28 smithi172 ceph-mon[100696]: osdmap e433: 8 total, 8 up, 7 in 2024-09-13T15:08:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:28 smithi172 ceph-mon[100696]: pgmap v1377: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 96 active+clean; 1.3 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 863 op/s; 19/62001 objects degraded (0.031%); 1934/62001 objects misplaced (3.119%); 960 KiB/s, 189 objects/s recovering 2024-09-13T15:08:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:08:28.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:28.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:28.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:28.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:28.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:28.776 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:28.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:28.902 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:28.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:29.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:29.027 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:29.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:08:29.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:29.161 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:29.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:29.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:29.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:29.297 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:29 smithi154 ceph-mon[102314]: osdmap e434: 8 total, 8 up, 7 in 2024-09-13T15:08:29.297 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:29 smithi154 ceph-mon[107038]: osdmap e434: 8 total, 8 up, 7 in 2024-09-13T15:08:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:29.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:29.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:29.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:29 smithi172 ceph-mon[100696]: osdmap e434: 8 total, 8 up, 7 in 2024-09-13T15:08:29.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:29.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:29.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:29.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:29.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:29.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:29.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:29.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:29.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:30.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:30.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T15:08:30.043+0000 7ff2a08a8640 -1 mon.a@0(leader).osd e435 definitely_dead 0 2024-09-13T15:08:30.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[102314]: pgmap v1379: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 97 active+clean; 1.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 1386/59460 objects misplaced (2.331%); 3.1 MiB/s, 241 objects/s recovering 2024-09-13T15:08:30.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-13T15:08:30.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[102314]: osdmap e435: 8 total, 8 up, 7 in 2024-09-13T15:08:30.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4182347481' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T15:08:30.181 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T15:08:30.182 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 19/62001 objects degraded (0.031%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:30.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[107038]: pgmap v1379: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 97 active+clean; 1.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 1386/59460 objects misplaced (2.331%); 3.1 MiB/s, 241 objects/s recovering 2024-09-13T15:08:30.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-13T15:08:30.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[107038]: osdmap e435: 8 total, 8 up, 7 in 2024-09-13T15:08:30.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4182347481' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T15:08:30.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T15:08:30.182 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:30 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 19/62001 objects degraded (0.031%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:30.182 INFO:teuthology.orchestra.run.smithi154.stderr:marked in osd.7. 2024-09-13T15:08:30.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:30.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:30.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:30.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:30.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:30 smithi172 ceph-mon[100696]: pgmap v1379: 99 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 97 active+clean; 1.2 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 1386/59460 objects misplaced (2.331%); 3.1 MiB/s, 241 objects/s recovering 2024-09-13T15:08:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:30 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-13T15:08:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:30 smithi172 ceph-mon[100696]: osdmap e435: 8 total, 8 up, 7 in 2024-09-13T15:08:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:30 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4182347481' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T15:08:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:30 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-13T15:08:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:30 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 19/62001 objects degraded (0.031%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:08:30.578 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:30.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:08:30.599 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:30.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:30.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:30.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:30.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:30.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:30.970 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:30.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:30.990 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-13T15:08:31.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:31.075 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:31.080 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:31.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:31.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:31.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/62001 objects degraded (0.031%), 1 pg degraded) 2024-09-13T15:08:31.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-13T15:08:31.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T15:08:31.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[102314]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T15:08:31.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/62001 objects degraded (0.031%), 1 pg degraded) 2024-09-13T15:08:31.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-13T15:08:31.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T15:08:31.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-mon[107038]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T15:08:31.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:31.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:31.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:31.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:31.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:31.471 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:31.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:31 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/62001 objects degraded (0.031%), 1 pg degraded) 2024-09-13T15:08:31.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:31 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-13T15:08:31.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:31 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-13T15:08:31.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:31 smithi172 ceph-mon[100696]: osdmap e436: 8 total, 8 up, 8 in 2024-09-13T15:08:31.574 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:31.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:31.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:31.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:31.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:31.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:31.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:31.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:31.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:32.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:32.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:32.131 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:08:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:08:31] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-13T15:08:32.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:32.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:32.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:32.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:32.445 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:32 smithi154 ceph-mon[102314]: pgmap v1382: 99 pgs: 7 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 90 active+clean; 1.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 323 op/s; 1386/58101 objects misplaced (2.386%); 4.3 MiB/s, 103 objects/s recovering 2024-09-13T15:08:32.445 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:32 smithi154 ceph-mon[102314]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T15:08:32.446 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:32 smithi154 ceph-mon[107038]: pgmap v1382: 99 pgs: 7 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 90 active+clean; 1.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 323 op/s; 1386/58101 objects misplaced (2.386%); 4.3 MiB/s, 103 objects/s recovering 2024-09-13T15:08:32.446 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:32 smithi154 ceph-mon[107038]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T15:08:32.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:32 smithi172 ceph-mon[100696]: pgmap v1382: 99 pgs: 7 peering, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 90 active+clean; 1.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 323 op/s; 1386/58101 objects misplaced (2.386%); 4.3 MiB/s, 103 objects/s recovering 2024-09-13T15:08:32.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:32 smithi172 ceph-mon[100696]: osdmap e437: 8 total, 8 up, 8 in 2024-09-13T15:08:32.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:32.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:32.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:32.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:32.774 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:32.876 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:32.883 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:08:32.883 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:08:32.883 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:08:32.884 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:08:32.884 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:08:32.884 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:08:32.884 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:08:32.884 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:08:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:32.902 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:32.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:32.916 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:08:32.916 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:08:32.947 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:08:32.947 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:08:32.979 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:08:32.979 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:08:33.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:33.026 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:08:33.026 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:08:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:33.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:33.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:33.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:33.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9881 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9900 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 435'9865 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.15 starting backfill to osd.7 from (0'0,0'0] MAX to 434'9772 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 435'11362 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.13 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10037 2024-09-13T15:08:33.280 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.9 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9885 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9824 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.19 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9919 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 435'11402 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.11 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10041 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.d starting backfill to osd.7 from (0'0,0'0] MAX to 435'11411 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.c starting backfill to osd.7 from (0'0,0'0] MAX to 435'11607 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9977 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.3 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10084 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9865 2024-09-13T15:08:33.281 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/191263526' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9881 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9900 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 435'9865 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.15 starting backfill to osd.7 from (0'0,0'0] MAX to 434'9772 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 435'11362 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.13 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10037 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.9 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9885 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9824 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.19 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9919 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 435'11402 2024-09-13T15:08:33.282 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.11 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10041 2024-09-13T15:08:33.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.d starting backfill to osd.7 from (0'0,0'0] MAX to 435'11411 2024-09-13T15:08:33.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.c starting backfill to osd.7 from (0'0,0'0] MAX to 435'11607 2024-09-13T15:08:33.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9977 2024-09-13T15:08:33.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.3 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10084 2024-09-13T15:08:33.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9865 2024-09-13T15:08:33.283 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/191263526' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:33.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:33.310 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:33.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:33.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9881 2024-09-13T15:08:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9900 2024-09-13T15:08:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 435'9865 2024-09-13T15:08:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.15 starting backfill to osd.7 from (0'0,0'0] MAX to 434'9772 2024-09-13T15:08:33.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 435'11362 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: osdmap e438: 8 total, 8 up, 8 in 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.13 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10037 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.9 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9885 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9824 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.19 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9919 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.a starting backfill to osd.7 from (0'0,0'0] MAX to 435'11402 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.11 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10041 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.d starting backfill to osd.7 from (0'0,0'0] MAX to 435'11411 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.c starting backfill to osd.7 from (0'0,0'0] MAX to 435'11607 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9977 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.3 starting backfill to osd.7 from (0'0,0'0] MAX to 435'10084 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 435'9865 2024-09-13T15:08:33.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/191263526' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:33.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:33.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:33.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:33.711 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:33.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:33.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:33.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:33.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:33.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:34.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:34.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:34.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:34.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:34.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:34.433 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:34 smithi154 ceph-mon[102314]: pgmap v1385: 99 pgs: 13 remapped+peering, 7 peering, 1 active+recovering+undersized+remapped, 78 active+clean; 1.1 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 470 op/s; 836/53811 objects misplaced (1.554%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T15:08:34.433 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:34 smithi154 ceph-mon[107038]: pgmap v1385: 99 pgs: 13 remapped+peering, 7 peering, 1 active+recovering+undersized+remapped, 78 active+clean; 1.1 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 470 op/s; 836/53811 objects misplaced (1.554%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T15:08:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.435 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:34.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:34.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:34 smithi172 ceph-mon[100696]: pgmap v1385: 99 pgs: 13 remapped+peering, 7 peering, 1 active+recovering+undersized+remapped, 78 active+clean; 1.1 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 470 op/s; 836/53811 objects misplaced (1.554%); 4.2 MiB/s, 66 objects/s recovering 2024-09-13T15:08:34.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:34.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:34.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:34.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:34.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:34.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:34.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:34.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:34.980 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:34.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:35.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:35.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:35.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:35.279 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:35.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:35.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:35 smithi172 ceph-mon[100696]: pgmap v1386: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 13 remapped+peering, 7 peering, 74 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 395 op/s; 45/52350 objects degraded (0.086%); 1622/52350 objects misplaced (3.098%); 6.0 MiB/s, 101 objects/s recovering 2024-09-13T15:08:35.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:35 smithi172 ceph-mon[100696]: Health check failed: Degraded data redundancy: 45/52350 objects degraded (0.086%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:35.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:35.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:35 smithi154 ceph-mon[102314]: pgmap v1386: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 13 remapped+peering, 7 peering, 74 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 395 op/s; 45/52350 objects degraded (0.086%); 1622/52350 objects misplaced (3.098%); 6.0 MiB/s, 101 objects/s recovering 2024-09-13T15:08:35.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:35 smithi154 ceph-mon[102314]: Health check failed: Degraded data redundancy: 45/52350 objects degraded (0.086%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:35.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:35 smithi154 ceph-mon[107038]: pgmap v1386: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 13 remapped+peering, 7 peering, 74 active+clean; 1.1 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 395 op/s; 45/52350 objects degraded (0.086%); 1622/52350 objects misplaced (3.098%); 6.0 MiB/s, 101 objects/s recovering 2024-09-13T15:08:35.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:35 smithi154 ceph-mon[107038]: Health check failed: Degraded data redundancy: 45/52350 objects degraded (0.086%), 3 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:35.554 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:35.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:35.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:35.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:35.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:35.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:35.972 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:35.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:35.991 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:08:35.991 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:08:35.991 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T15:08:35.991 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-09-13T15:08:35.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T15:08:36.073 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:36.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:36.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:36.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:36.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:36.362 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:08:36.362 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:08:36.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:36.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:36.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:36.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:36.642 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:36.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:36.771 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:36.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:36.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:36.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:36.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:08:37.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:37.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:37.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:37.185 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:37.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:08:37.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.311 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:37.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:37.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:37.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:37.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:37.553 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:37 smithi154 ceph-mon[107038]: pgmap v1387: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 914 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 59/43761 objects degraded (0.135%); 9524/43761 objects misplaced (21.764%); 5.2 MiB/s, 86 objects/s recovering 2024-09-13T15:08:37.553 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:37 smithi154 ceph-mon[102314]: pgmap v1387: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 914 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 59/43761 objects degraded (0.135%); 9524/43761 objects misplaced (21.764%); 5.2 MiB/s, 86 objects/s recovering 2024-09-13T15:08:37.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:37.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:37.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:37 smithi172 ceph-mon[100696]: pgmap v1387: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 914 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 59/43761 objects degraded (0.135%); 9524/43761 objects misplaced (21.764%); 5.2 MiB/s, 86 objects/s recovering 2024-09-13T15:08:37.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:37.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:37.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:37.849 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:37.940 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:37.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:38.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:38.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:38.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:38.188 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:08:38.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:38.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:38.330 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:08:38.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:38.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:38.435 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1190789790' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:38.436 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1190789790' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:38.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:38.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:38.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:38.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1190789790' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:08:38.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.849 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:38.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:38.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:38.983 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:38.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:39.085 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:39.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:39.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.246 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:39.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:39.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:39.373 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:39.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:39.501 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:39.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:39.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1190789790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[102314]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T15:08:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[102314]: pgmap v1389: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 856 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 132/40974 objects degraded (0.322%); 8952/40974 objects misplaced (21.848%); 5.3 MiB/s, 101 objects/s recovering 2024-09-13T15:08:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1190789790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[107038]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T15:08:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:39 smithi154 ceph-mon[107038]: pgmap v1389: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 856 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 132/40974 objects degraded (0.322%); 8952/40974 objects misplaced (21.848%); 5.3 MiB/s, 101 objects/s recovering 2024-09-13T15:08:39.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:39 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:08:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:39 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1190789790' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:08:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:39 smithi172 ceph-mon[100696]: osdmap e439: 8 total, 8 up, 8 in 2024-09-13T15:08:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:39 smithi172 ceph-mon[100696]: pgmap v1389: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 856 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 132/40974 objects degraded (0.322%); 8952/40974 objects misplaced (21.848%); 5.3 MiB/s, 101 objects/s recovering 2024-09-13T15:08:39.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:39.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:39.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:39.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:39.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:40.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:40.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:40.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:40.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:40.192 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:40.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:40.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:40.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:40.461 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:40.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:40.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:40.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:40.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:40.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:40.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:40.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:40.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:41.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:41.139 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:41.169 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:41.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:41.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:41.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:41.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:41 smithi154 ceph-mon[102314]: pgmap v1390: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 814 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 551 op/s; 132/38949 objects degraded (0.339%); 8525/38949 objects misplaced (21.888%); 2.9 MiB/s, 60 objects/s recovering 2024-09-13T15:08:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:41 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 132/38949 objects degraded (0.339%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:41.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:41 smithi154 ceph-mon[107038]: pgmap v1390: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 814 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 551 op/s; 132/38949 objects degraded (0.339%); 8525/38949 objects misplaced (21.888%); 2.9 MiB/s, 60 objects/s recovering 2024-09-13T15:08:41.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:41 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 132/38949 objects degraded (0.339%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:41.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:41.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:41.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:41.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:41 smithi172 ceph-mon[100696]: pgmap v1390: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 814 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 551 op/s; 132/38949 objects degraded (0.339%); 8525/38949 objects misplaced (21.888%); 2.9 MiB/s, 60 objects/s recovering 2024-09-13T15:08:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:41 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 132/38949 objects degraded (0.339%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:41.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:41.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:41.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:41.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:41.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:42.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:42.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:42.148 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:08:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:08:41] "GET /metrics HTTP/1.1" 200 36613 "" "Prometheus/2.51.0" 2024-09-13T15:08:42.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:42.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:42.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:42.407 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:42.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:42.536 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:42.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:42.672 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:42.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:42.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:42.916 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:42.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:42.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:42.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:43.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:43.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:43.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:43.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:43.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:43.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:43.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:43.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:43.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:43.513 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:43.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:43.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:43.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:43.802 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:43.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:08:43.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:43.908 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:08:43.908 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:08:43.956 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:08:43.957 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:08:43.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:43.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:44.000 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:08:44.000 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:08:44.033 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:08:44.033 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:08:44.067 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:08:44.067 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:08:44.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:44.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.093 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:44.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:44.099 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:08:44.100 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:08:44.131 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:08:44.131 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:08:44.165 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:08:44.166 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:08:44.166 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:44 smithi154 ceph-mon[102314]: pgmap v1391: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 644 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 132/30768 objects degraded (0.429%); 6514/30768 objects misplaced (21.171%); 2.6 MiB/s, 54 objects/s recovering 2024-09-13T15:08:44.166 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2947749655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:44.166 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:44 smithi154 ceph-mon[107038]: pgmap v1391: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 644 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 132/30768 objects degraded (0.429%); 6514/30768 objects misplaced (21.171%); 2.6 MiB/s, 54 objects/s recovering 2024-09-13T15:08:44.166 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2947749655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:44.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:44.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:44.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:44 smithi172 ceph-mon[100696]: pgmap v1391: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 644 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 132/30768 objects degraded (0.429%); 6514/30768 objects misplaced (21.171%); 2.6 MiB/s, 54 objects/s recovering 2024-09-13T15:08:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2947749655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:08:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:44.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:44.505 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:44.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:44.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:44.680 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:44.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:44.806 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:44.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:44.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:44.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:45.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:45.071 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T15:08:45.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:45.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:45.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:45.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:45 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/247859186' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:45 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:45 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/247859186' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:45 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:45.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:45.327 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:45.429 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:45.450 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:45 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/247859186' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:45 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:08:45.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:45.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:45.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:45.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:45.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:45.824 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:45.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:45.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:45.949 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:45.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:46.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.076 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:46.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:46.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:46.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:46.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:46.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:46.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:46.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:46.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:46.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[102314]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[102314]: pgmap v1393: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 580 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 202/27687 objects degraded (0.730%); 6210/27687 objects misplaced (22.429%); 832 KiB/s, 64 objects/s recovering 2024-09-13T15:08:46.473 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[102314]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T15:08:46.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:46.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[107038]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T15:08:46.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[107038]: pgmap v1393: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 580 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 202/27687 objects degraded (0.730%); 6210/27687 objects misplaced (22.429%); 832 KiB/s, 64 objects/s recovering 2024-09-13T15:08:46.473 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:46 smithi154 ceph-mon[107038]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T15:08:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:46 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:08:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:46 smithi172 ceph-mon[100696]: osdmap e440: 8 total, 8 up, 8 in 2024-09-13T15:08:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:46 smithi172 ceph-mon[100696]: pgmap v1393: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 580 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 202/27687 objects degraded (0.730%); 6210/27687 objects misplaced (22.429%); 832 KiB/s, 64 objects/s recovering 2024-09-13T15:08:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:46 smithi172 ceph-mon[100696]: osdmap e441: 8 total, 8 up, 8 in 2024-09-13T15:08:46.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:46.577 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:46.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:46.701 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:46.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:46.784 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:08:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156567 8.4 e352/331 8:2820a897:::benchmark_data_smithi154_261841_object40528:head v 352'9779, mlcod=352'9777)", 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 351.39227096799999, 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.20225130599999999, 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119135+0000", 2024-09-13T15:08:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7099999999999998e-07 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119138+0000", 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9909999999999998e-06 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119143+0000", 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4099999999999999e-06 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119147+0000", 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0779999999999997e-06 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119321+0000", 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000173067 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119339+0000", 2024-09-13T15:08:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8553e-05 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321283+0000", 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.20194368900000001 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321386+0000", 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103247 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156561 8.f e352/331 8:f0bec425:::benchmark_data_smithi154_261841_object40522:head v 352'9612, mlcod=352'9610)", 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 351.39222782000002, 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.19389062200000001, 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119178+0000", 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2099999999999998e-07 2024-09-13T15:08:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119181+0000", 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0920000000000002e-06 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119188+0000", 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7190000000000004e-06 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.119192+0000", 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3100000000000002e-06 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.123961+0000", 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047686250000000003 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.123979+0000", 2024-09-13T15:08:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.312999+0000", 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18902035 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313068+0000", 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9308999999999997e-05 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156569 8.4 e352/331 8:2859ecb2:::benchmark_data_smithi154_261841_object40530:head v 352'9780, mlcod=352'9777)", 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.124429+0000", 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 351.38697635699998, 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.196985044, 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124429+0000", 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124429+0000", 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124430+0000", 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9100000000000002e-07 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124437+0000", 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4540000000000004e-06 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124448+0000", 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0913999999999999e-05 2024-09-13T15:08:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.124454+0000", 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5989999999999998e-06 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313126+0000", 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18867229199999999 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313146+0000", 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9276999999999999e-05 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321390+0000", 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082445820000000003 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.321414+0000", 2024-09-13T15:08:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4034999999999999e-05 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156572 8.a e352/331 8:5a825b83:::benchmark_data_smithi154_261841_object40533:head v 352'9705, mlcod=352'9703)", 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 351.38398714700003, 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18567791, 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127419+0000", 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3900000000000003e-07 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127425+0000", 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0079999999999999e-06 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127436+0000", 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210000000000001e-05 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127442+0000", 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6950000000000003e-06 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127512+0000", 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9210999999999999e-05 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127529+0000", 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7428999999999999e-05 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313056+0000", 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18552676900000001 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313096+0000", 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0649000000000003e-05 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156571 8.7 e352/331 8:e2978e79:::benchmark_data_smithi154_261841_object40532:head v 352'9816, mlcod=352'9814)", 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.127461+0000", 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 351.38394423900002, 2024-09-13T15:08:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18564062200000001, 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127461+0000", 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127461+0000", 2024-09-13T15:08:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127462+0000", 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5699999999999999e-07 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127466+0000", 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2169999999999996e-06 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127472+0000", 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9510000000000003e-06 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127476+0000", 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2889999999999999e-06 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127877+0000", 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000400911 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.127895+0000", 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7652999999999999e-05 2024-09-13T15:08:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313074+0000", 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18517932400000001 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313102+0000", 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8019999999999999e-05 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.35787.0:156568 8.3 e352/331 8:cb5d79f8:::benchmark_data_smithi154_261841_object40529:head v 352'9697, mlcod=352'9693)", 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:02:54.131003+0000", 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 351.38040286799998, 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18894451300000001, 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131003+0000", 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131003+0000", 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131004+0000", 2024-09-13T15:08:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2139999999999999e-06 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.131009+0000", 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4129999999999997e-06 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.312989+0000", 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18197933299999999 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313005+0000", 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5883e-05 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313072+0000", 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7657999999999995e-05 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.313098+0000", 2024-09-13T15:08:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5267999999999999e-05 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.319919+0000", 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068212050000000003 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:02:54.319947+0000", 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8538999999999999e-05 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:31394 9.7 e384/383 9:ec33988d:::benchmark_data_smithi154_291684_object31393:head v 384'1918, mlcod=384'1915)", 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:28.308317+0000", 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 197.203088759, 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.24459751199999999, 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308317+0000", 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308317+0000", 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308318+0000", 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3009999999999999e-06 2024-09-13T15:08:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308400+0000", 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1773000000000005e-05 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308407+0000", 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7449999999999998e-06 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308412+0000", 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9439999999999996e-06 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308483+0000", 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1654000000000002e-05 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.308502+0000", 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8474e-05 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.552857+0000", 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.244355339 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.552914+0000", 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7281999999999998e-05 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:31402 9.6 e384/374 9:6490ea97:::benchmark_data_smithi154_291684_object31401:head v 384'1995, mlcod=384'1993)", 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:28.311166+0000", 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 197.20023973400001, 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.24471507100000001, 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.311166+0000", 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.311166+0000", 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.311167+0000", 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1009999999999999e-06 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.311213+0000", 2024-09-13T15:08:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5979999999999997e-05 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.552946+0000", 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.24173254499999999 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.552956+0000", 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0076e-05 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.553013+0000", 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6913999999999998e-05 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.553030+0000", 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7320999999999999e-05 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.555820+0000", 2024-09-13T15:08:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027903609999999999 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:28.555881+0000", 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0773000000000002e-05 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134979 9.0 e407/374 9:089187ee:::benchmark_data_smithi154_291684_object34518:head v 407'8381, mlcod=407'8379)", 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.229779+0000", 2024-09-13T15:08:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.281626716999995, 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30610319800000002, 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229779+0000", 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229779+0000", 2024-09-13T15:08:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229780+0000", 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1310000000000001e-06 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229788+0000", 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7640000000000005e-06 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229803+0000", 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5176e-05 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229811+0000", 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0209999999999999e-06 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229881+0000", 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0418000000000002e-05 2024-09-13T15:08:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229902+0000", 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0298000000000001e-05 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.535882+0000", 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30598038999999999 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134986 9.0 e407/374 9:0b57539f:::benchmark_data_smithi154_291684_object34525:head v 407'8382, mlcod=407'8379)", 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.231178+0000", 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.280227847000006, 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30471788999999999, 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231178+0000", 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231178+0000", 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231179+0000", 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6199999999999996e-07 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231184+0000", 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7490000000000004e-06 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231195+0000", 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0915e-05 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231202+0000", 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7190000000000004e-06 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.233208+0000", 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002005476 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.233228+0000", 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0911e-05 2024-09-13T15:08:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.535896+0000", 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30266725799999999 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134990 9.7 e407/383)", 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.237147+0000", 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.274258204000006, 2024-09-13T15:08:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42793508299999999, 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237147+0000", 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237147+0000", 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237149+0000", 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.249e-06 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237156+0000", 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.827e-06 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237166+0000", 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0390999999999999e-05 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237173+0000", 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.024e-06 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.665061+0000", 2024-09-13T15:08:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42788764600000001 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.665083+0000", 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1946e-05 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134987 9.2 e407/374)", 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.253004+0000", 2024-09-13T15:08:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.258401922000004, 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.38170005400000001, 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253004+0000", 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253004+0000", 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253005+0000", 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0499999999999999e-06 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253014+0000", 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8419999999999994e-06 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.267268+0000", 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014254271000000001 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.267278+0000", 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8279999999999998e-06 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634685+0000", 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.36740701799999997 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634704+0000", 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9045000000000001e-05 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134997 9.2 e407/374)", 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.272228+0000", 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.239178151000004, 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.36251625199999998, 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272228+0000", 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272228+0000", 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272229+0000", 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2929999999999999e-06 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272236+0000", 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8680000000000001e-06 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272250+0000", 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.451e-05 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272259+0000", 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6570000000000003e-06 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634731+0000", 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.36247178800000002 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634744+0000", 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3135999999999999e-05 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:135001 9.0 e407/374)", 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.239097134000005, 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.25325296200000003, 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7399999999999998e-07 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272314+0000", 2024-09-13T15:08:46.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5159999999999996e-06 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272321+0000", 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6620000000000005e-06 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272326+0000", 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9479999999999999e-06 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.525542+0000", 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.253215992 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.525562+0000", 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9369999999999999e-05 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:169574 9.17 9:eb9696a3:::benchmark_data_smithi154_291684_object69113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:46.850 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:12.039568+0000", 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.471837710999999, 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.20130683199999999, 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 169574 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.039568+0000", 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.039568+0000", 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.039569+0000", 2024-09-13T15:08:46.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.403e-06 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.039575+0000", 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5790000000000003e-06 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.039592+0000", 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7105999999999999e-05 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.039599+0000", 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.418e-06 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.041028+0000", 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014288269999999999 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.044224+0000", 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031961239999999998 2024-09-13T15:08:46.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058255+0000", 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014030828 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058354+0000", 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8257000000000005e-05 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058855+0000", 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00050142299999999997 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.071280+0000", 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012425426999999999 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.071311+0000", 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1108000000000001e-05 2024-09-13T15:08:46.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.238995+0000", 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16768329500000001 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.239603+0000", 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060839399999999995 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.240875+0000", 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001271643 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:169596 9.17 9:ecff2ad8:::benchmark_data_smithi154_291684_object69135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:12.058682+0000", 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.452724105999998, 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.21879607100000001, 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.858 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 169596 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058682+0000", 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058682+0000", 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058683+0000", 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2100000000000001e-06 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058688+0000", 2024-09-13T15:08:46.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9779999999999996e-06 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058698+0000", 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0658e-05 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.058705+0000", 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2430000000000002e-06 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.063903+0000", 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051979119999999998 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.064935+0000", 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010329250000000001 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.243063+0000", 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.178127386 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.243285+0000", 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022189099999999999 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.244036+0000", 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00075073100000000003 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:46.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.256958+0000", 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012922664 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.265953+0000", 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089944389999999999 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.276123+0000", 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010169984999999999 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.276419+0000", 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029676300000000002 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.277478+0000", 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010582860000000001 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:169602 9.17 9:e8bd2f25:::benchmark_data_smithi154_291684_object69141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:12.062670+0000", 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.448735958999997, 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.21580152, 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 169602 2024-09-13T15:08:46.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.062670+0000", 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.062670+0000", 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.062672+0000", 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9599999999999999e-06 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.062682+0000", 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0555e-05 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.062697+0000", 2024-09-13T15:08:46.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4375e-05 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.062704+0000", 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2649999999999999e-06 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.065065+0000", 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023611280000000001 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.065918+0000", 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00085286100000000001 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.250070+0000", 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.184152119 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.250335+0000", 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000265266 2024-09-13T15:08:46.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.253195+0000", 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028594250000000001 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.256983+0000", 2024-09-13T15:08:46.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037886510000000001 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.267580+0000", 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010596230999999999 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.277535+0000", 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099557450000000002 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.277799+0000", 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026386500000000001 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:12.278471+0000", 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00067207400000000004 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:172010 9.17 9:eb5f2b18:::benchmark_data_smithi154_291684_object71549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:15.154838+0000", 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.356567877, 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.179003513, 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172010 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.874 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.154838+0000", 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.154838+0000", 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.154839+0000", 2024-09-13T15:08:46.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4869999999999999e-06 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.154846+0000", 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.579e-06 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.154949+0000", 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103251 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.154960+0000", 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1070000000000001e-05 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.161388+0000", 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064273120000000001 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.162744+0000", 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001356433 2024-09-13T15:08:46.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.196314+0000", 2024-09-13T15:08:46.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033570497999999997 2024-09-13T15:08:46.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.196408+0000", 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3800000000000003e-05 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.196938+0000", 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00052959999999999997 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.209935+0000", 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012997266 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.209951+0000", 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6283999999999999e-05 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.332286+0000", 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.12233493600000001 2024-09-13T15:08:46.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.332574+0000", 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028739800000000002 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.333841+0000", 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012675989999999999 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:172020 9.17 9:eddf8bf1:::benchmark_data_smithi154_291684_object71559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:15.177954+0000", 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.333451882999999, 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.171982676, 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:46.884 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 172020 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.177954+0000", 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.177954+0000", 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.177955+0000", 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6789999999999999e-06 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.177962+0000", 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.973e-06 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.177978+0000", 2024-09-13T15:08:46.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6019999999999999e-05 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.177986+0000", 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6660000000000008e-06 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.178052+0000", 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6172000000000003e-05 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.179072+0000", 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010199829999999999 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.327961+0000", 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14888874899999999 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.328117+0000", 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000155957 2024-09-13T15:08:46.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.328623+0000", 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00050573600000000001 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.342967+0000", 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014344595999999999 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.343010+0000", 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3084000000000003e-05 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.348951+0000", 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005940982 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.349075+0000", 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012375999999999999 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:15.349936+0000", 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00086131900000000004 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:180695 9.17 9:ea18e4ef:::benchmark_data_smithi154_291684_object80234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:26.313924+0000", 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.197482163, 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.16892914000000001, 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:08:46.891 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 180695 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.313924+0000", 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.313924+0000", 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.313925+0000", 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9479999999999998e-06 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.313931+0000", 2024-09-13T15:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4249999999999997e-06 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.313946+0000", 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4919999999999999e-05 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.313958+0000", 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2245e-05 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.317404+0000", 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00344597 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.318221+0000", 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081698999999999997 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.341016+0000", 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022794669 2024-09-13T15:08:46.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.341109+0000", 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3264999999999996e-05 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.341652+0000", 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054284699999999995 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.345997+0000", 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043450559999999999 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.474454+0000", 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.128456983 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.482394+0000", 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079401390000000006 2024-09-13T15:08:46.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.482475+0000", 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1007999999999994e-05 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:26.482853+0000", 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037767500000000001 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:08:46.898 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:08:46.899 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:08:46.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:46.921 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:46.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:47.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:47.043 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:47.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:47.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:47.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:47.302 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:47.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:47.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:47.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:47.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:47.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:47.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:47.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:47.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:47.861 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:47.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:47.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:48.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:48.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:48.146 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:48.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:48.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:48.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:48.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[102314]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T15:08:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[102314]: 9.7 continuing backfill to osd.7 from (377'161,439'10239] MIN to 439'10239 2024-09-13T15:08:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[102314]: 9.a starting backfill to osd.7 from (403'6098,439'12088] MIN to 439'12090 2024-09-13T15:08:48.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[102314]: pgmap v1396: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 439 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 202/20919 objects degraded (0.966%); 4530/20919 objects misplaced (21.655%); 0 B/s, 52 objects/s recovering 2024-09-13T15:08:48.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[107038]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T15:08:48.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[107038]: 9.7 continuing backfill to osd.7 from (377'161,439'10239] MIN to 439'10239 2024-09-13T15:08:48.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[107038]: 9.a starting backfill to osd.7 from (403'6098,439'12088] MIN to 439'12090 2024-09-13T15:08:48.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:48 smithi154 ceph-mon[107038]: pgmap v1396: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 439 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 202/20919 objects degraded (0.966%); 4530/20919 objects misplaced (21.655%); 0 B/s, 52 objects/s recovering 2024-09-13T15:08:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:48 smithi172 ceph-mon[100696]: osdmap e442: 8 total, 8 up, 8 in 2024-09-13T15:08:48.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:48 smithi172 ceph-mon[100696]: 9.7 continuing backfill to osd.7 from (377'161,439'10239] MIN to 439'10239 2024-09-13T15:08:48.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:48 smithi172 ceph-mon[100696]: 9.a starting backfill to osd.7 from (403'6098,439'12088] MIN to 439'12090 2024-09-13T15:08:48.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:48 smithi172 ceph-mon[100696]: pgmap v1396: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 15 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 439 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 202/20919 objects degraded (0.966%); 4530/20919 objects misplaced (21.655%); 0 B/s, 52 objects/s recovering 2024-09-13T15:08:48.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:48.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:48.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:48.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:48.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:48.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:48.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:48.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:48.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:49.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:49.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:49.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:49.244 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:49.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:49.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:49.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:49 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 202/20919 objects degraded (0.966%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:49.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:49 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 202/20919 objects degraded (0.966%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:49.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:49 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 202/20919 objects degraded (0.966%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:49.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:49.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:49.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:49.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:49.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:49.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:49.917 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:49.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:50.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.050 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:50.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.186 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:50.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:50.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:50.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:50.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:50.437 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:50.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:50 smithi154 ceph-mon[102314]: pgmap v1397: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 377 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 241/17985 objects degraded (1.340%); 4113/17985 objects misplaced (22.869%); 0 B/s, 177 objects/s recovering 2024-09-13T15:08:50.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:50 smithi154 ceph-mon[107038]: pgmap v1397: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 377 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 241/17985 objects degraded (1.340%); 4113/17985 objects misplaced (22.869%); 0 B/s, 177 objects/s recovering 2024-09-13T15:08:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:50 smithi172 ceph-mon[100696]: pgmap v1397: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 377 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 241/17985 objects degraded (1.340%); 4113/17985 objects misplaced (22.869%); 0 B/s, 177 objects/s recovering 2024-09-13T15:08:50.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.569 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:50.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:50.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:50.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:50.965 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:50.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:51.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:51.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:51.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:51.223 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:51.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:51.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:51.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:51.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:51.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:51.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:51.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:51.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:51 smithi154 ceph-mon[102314]: pgmap v1398: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 345 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 241/16428 objects degraded (1.467%); 3799/16428 objects misplaced (23.125%); 0 B/s, 122 objects/s recovering 2024-09-13T15:08:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:51 smithi154 ceph-mon[107038]: pgmap v1398: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 345 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 241/16428 objects degraded (1.467%); 3799/16428 objects misplaced (23.125%); 0 B/s, 122 objects/s recovering 2024-09-13T15:08:51.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:51 smithi172 ceph-mon[100696]: pgmap v1398: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 345 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 615 op/s; 241/16428 objects degraded (1.467%); 3799/16428 objects misplaced (23.125%); 0 B/s, 122 objects/s recovering 2024-09-13T15:08:51.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:51.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:51.903 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:51.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:52.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:52.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:52.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:52.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:08:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:08:51] "GET /metrics HTTP/1.1" 200 36623 "" "Prometheus/2.51.0" 2024-09-13T15:08:52.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:52.311 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:52.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:52.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:52.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:52.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:52.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:52.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:08:52.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:08:52.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:52.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:52.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:52.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:52.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:53.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:53.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:53.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:53.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:53.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:53.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:53.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:53.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:53.636 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:53.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:53.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:53 smithi154 ceph-mon[102314]: pgmap v1399: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 219 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 241/10377 objects degraded (2.322%); 2288/10377 objects misplaced (22.049%); 0 B/s, 93 objects/s recovering 2024-09-13T15:08:53.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:53 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 241/10377 objects degraded (2.322%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:53.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:53 smithi154 ceph-mon[107038]: pgmap v1399: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 219 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 241/10377 objects degraded (2.322%); 2288/10377 objects misplaced (22.049%); 0 B/s, 93 objects/s recovering 2024-09-13T15:08:53.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:53 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 241/10377 objects degraded (2.322%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:53 smithi172 ceph-mon[100696]: pgmap v1399: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 219 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 241/10377 objects degraded (2.322%); 2288/10377 objects misplaced (22.049%); 0 B/s, 93 objects/s recovering 2024-09-13T15:08:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:53 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 241/10377 objects degraded (2.322%), 5 pgs degraded (PG_DEGRADED) 2024-09-13T15:08:53.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:53.793 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:53.895 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:53.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:53.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:54.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.055 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:54.156 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.184 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:54.285 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:54.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:54.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:54.444 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:54.546 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:54.582 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:54.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.715 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:54.816 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:54.837 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:54.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:54.976 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:54.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:55.026 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Removed 100458 objects 2024-09-13T15:08:55.026 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Clean up completed and total clean up time :134.55 2024-09-13T15:08:55.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:55.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:55.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:55.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:55.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:55.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:55.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:55.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:55.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:55.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:55.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:55.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:55 smithi154 ceph-mon[102314]: pgmap v1400: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 144 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 304/6780 objects degraded (4.484%); 1774/6780 objects misplaced (26.165%); 148 KiB/s, 130 objects/s recovering 2024-09-13T15:08:55.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:55 smithi154 ceph-mon[107038]: pgmap v1400: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 144 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 304/6780 objects degraded (4.484%); 1774/6780 objects misplaced (26.165%); 148 KiB/s, 130 objects/s recovering 2024-09-13T15:08:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.722 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:55.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:55 smithi172 ceph-mon[100696]: pgmap v1400: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 144 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 304/6780 objects degraded (4.484%); 1774/6780 objects misplaced (26.165%); 148 KiB/s, 130 objects/s recovering 2024-09-13T15:08:55.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:55.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:55.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:55.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:55.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:56.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:56.090 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:56.101 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-13T15:08:56.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-13T15:08:56.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:56.212 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:56.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:56.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:56.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:56.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:56.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:56.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:56.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:56.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:56.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:56.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:56.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:56.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:56.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:57.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:57.077 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:57.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:57.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:57.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:57.325 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:57.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:57.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:57.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:57.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:57.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:57.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:57.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:57 smithi154 ceph-mon[102314]: pgmap v1401: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 24 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 304/1005 objects degraded (30.249%); 386/1005 objects misplaced (38.408%); 133 KiB/s, 117 objects/s recovering 2024-09-13T15:08:57.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:57 smithi154 ceph-mon[107038]: pgmap v1401: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 24 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 304/1005 objects degraded (30.249%); 386/1005 objects misplaced (38.408%); 133 KiB/s, 117 objects/s recovering 2024-09-13T15:08:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:57 smithi172 ceph-mon[100696]: pgmap v1401: 99 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 14 active+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 24 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 304/1005 objects degraded (30.249%); 386/1005 objects misplaced (38.408%); 133 KiB/s, 117 objects/s recovering 2024-09-13T15:08:57.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:57.820 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:57.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:57.943 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:57.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:58.045 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:58.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:58.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:58.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:58.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:58.314 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:58.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:58.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:58.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:58.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:58.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:58.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:58.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:08:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:08:58.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:58.809 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:58.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:58.933 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:58.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:59.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:08:59.057 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:08:59.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:08:59.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:08:59.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:59.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:59.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:08:59.440 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:08:59.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:08:59.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:59.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:08:59.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:59 smithi154 ceph-mon[102314]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T15:08:59.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:08:59 smithi154 ceph-mon[102314]: pgmap v1403: 99 pgs: 12 peering, 5 active+remapped+backfill_wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 12/84 objects degraded (14.286%); 192 KiB/s, 59 objects/s recovering 2024-09-13T15:08:59.649 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:59 smithi154 ceph-mon[107038]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T15:08:59.649 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:08:59 smithi154 ceph-mon[107038]: pgmap v1403: 99 pgs: 12 peering, 5 active+remapped+backfill_wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 12/84 objects degraded (14.286%); 192 KiB/s, 59 objects/s recovering 2024-09-13T15:08:59.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:08:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.688 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:59.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:08:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:59 smithi172 ceph-mon[100696]: osdmap e443: 8 total, 8 up, 8 in 2024-09-13T15:08:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:08:59 smithi172 ceph-mon[100696]: pgmap v1403: 99 pgs: 12 peering, 5 active+remapped+backfill_wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.6 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 596 op/s; 12/84 objects degraded (14.286%); 192 KiB/s, 59 objects/s recovering 2024-09-13T15:08:59.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:08:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:08:59.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:08:59.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:08:59.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:08:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:08:59.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:00.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:00.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:00.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.182 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:00.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:00.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:00.306 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:00.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:00.429 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:00.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:00.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:00.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:00.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:00.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:00 smithi154 ceph-mon[102314]: Health check update: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:09:00.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:00 smithi154 ceph-mon[102314]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T15:09:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:00 smithi154 ceph-mon[107038]: Health check update: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:09:00.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:00 smithi154 ceph-mon[107038]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T15:09:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:00 smithi172 ceph-mon[100696]: Health check update: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded (PG_DEGRADED) 2024-09-13T15:09:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:00 smithi172 ceph-mon[100696]: osdmap e444: 8 total, 8 up, 8 in 2024-09-13T15:09:00.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:00.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:00.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:09:00.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:09:00.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:00.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:00.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:01.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:01.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:01.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:01.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:01.246 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:01.247 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:01.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:01.317 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:01.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:01.445 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:01 smithi154 ceph-mon[102314]: pgmap v1405: 99 pgs: 12 peering, 3 active+remapped+backfill_wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 428 op/s; 12/84 objects degraded (14.286%); 240 KiB/s, 74 objects/s recovering 2024-09-13T15:09:01.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:01 smithi154 ceph-mon[107038]: pgmap v1405: 99 pgs: 12 peering, 3 active+remapped+backfill_wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 428 op/s; 12/84 objects degraded (14.286%); 240 KiB/s, 74 objects/s recovering 2024-09-13T15:09:01.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:01.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:01.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:01.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:01 smithi172 ceph-mon[100696]: pgmap v1405: 99 pgs: 12 peering, 3 active+remapped+backfill_wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 428 op/s; 12/84 objects degraded (14.286%); 240 KiB/s, 74 objects/s recovering 2024-09-13T15:09:01.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:01.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:01.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:01.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:01.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:02.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:02.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:02.090 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:09:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:09:01] "GET /metrics HTTP/1.1" 200 36616 "" "Prometheus/2.51.0" 2024-09-13T15:09:02.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:02.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:02.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:02.331 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:02.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:02.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:02.464 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:02.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:02.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:02.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:02.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:02.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:02.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:02.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:02.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:02.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:03.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:03.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:03.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:03.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:03.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:03.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:03.454 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:03 smithi154 ceph-mon[102314]: pgmap v1406: 99 pgs: 12 peering, 1 creating+peering, 86 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 278 op/s; 72 KiB/s, 21 objects/s recovering 2024-09-13T15:09:03.454 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:03 smithi154 ceph-mon[102314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded) 2024-09-13T15:09:03.454 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:03 smithi154 ceph-mon[107038]: pgmap v1406: 99 pgs: 12 peering, 1 creating+peering, 86 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 278 op/s; 72 KiB/s, 21 objects/s recovering 2024-09-13T15:09:03.454 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:03 smithi154 ceph-mon[107038]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded) 2024-09-13T15:09:03.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:03.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:03.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:03.608 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:03.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:03.735 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:03 smithi172 ceph-mon[100696]: pgmap v1406: 99 pgs: 12 peering, 1 creating+peering, 86 active+clean; 4.6 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 278 op/s; 72 KiB/s, 21 objects/s recovering 2024-09-13T15:09:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:03 smithi172 ceph-mon[100696]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/84 objects degraded (14.286%), 1 pg degraded) 2024-09-13T15:09:03.837 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:03.863 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:03.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:03.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:03.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:04.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:04.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:04.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:04.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:04.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:04.334 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:04.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:04.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:04.364 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:04.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:04.487 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:04.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:04.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:04.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:04.742 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:04.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:04.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:04.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:04.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:04.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:05.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:05.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:05.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:05.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:05.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:05.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:05.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:05.406 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:05 smithi154 ceph-mon[102314]: pgmap v1407: 99 pgs: 99 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 38 op/s; 72 KiB/s, 21 objects/s recovering 2024-09-13T15:09:05.406 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:05 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T15:09:05.407 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:05 smithi154 ceph-mon[107038]: pgmap v1407: 99 pgs: 99 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 38 op/s; 72 KiB/s, 21 objects/s recovering 2024-09-13T15:09:05.407 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:05 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T15:09:05.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:05 smithi172 ceph-mon[100696]: pgmap v1407: 99 pgs: 99 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 38 op/s; 72 KiB/s, 21 objects/s recovering 2024-09-13T15:09:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:05 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-13T15:09:05.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:05.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:05.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:05.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:05.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:05.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:05.873 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:05.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:05.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:06.000 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:06.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:06.128 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:06.228 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_4' removed 2024-09-13T15:09:06.229 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:09:06.229 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:09:06.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:06.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:06.280 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T15:09:06.281 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:06 smithi154 ceph-mon[107038]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T15:09:06.281 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4148404422' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:09:06.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:06.282 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T15:09:06.282 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:06 smithi154 ceph-mon[102314]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T15:09:06.283 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4148404422' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:09:06.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:06.409 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-13T15:09:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:06 smithi172 ceph-mon[100696]: osdmap e445: 8 total, 8 up, 8 in 2024-09-13T15:09:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4148404422' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:09:06.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:06.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:06.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:06.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:06.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:06.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:06.891 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:06.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:06.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:07.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:07.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:07.141 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:07.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:07.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:07.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:07.344 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4148404422' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:09:07.344 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[102314]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T15:09:07.344 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2499958009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:07.344 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[102314]: pgmap v1410: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:07.345 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4148404422' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:09:07.345 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[107038]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T15:09:07.345 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2499958009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:07.345 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:07 smithi154 ceph-mon[107038]: pgmap v1410: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:07.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:07.412 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4148404422' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:09:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:07 smithi172 ceph-mon[100696]: osdmap e446: 8 total, 8 up, 8 in 2024-09-13T15:09:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:07 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2499958009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:07.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:07 smithi172 ceph-mon[100696]: pgmap v1410: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:07.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:07.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:07.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:07.672 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:07.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:07.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:07.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:07.920 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:07.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:08.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:08.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:08.146 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:08.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:08.188 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-13T15:09:08.189 INFO:tasks.radosbench:Beginning radosbench... 2024-09-13T15:09:08.190 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-13T15:09:08.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-09-13T15:09:08.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:08.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:08.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:08.415 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:08.517 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:08.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:08.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:08.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:08.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:08.774 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:09:08.775 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:09:08.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:08.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:08.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:08.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:09.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:09.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:09.136 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:09.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:09.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:09.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:09.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:09.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:09.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:09:09.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:09.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:09 smithi154 ceph-mon[102314]: pgmap v1411: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:09.514 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:09 smithi154 ceph-mon[107038]: pgmap v1411: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:09.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:09.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:09.652 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:09 smithi172 ceph-mon[100696]: pgmap v1411: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:09.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:09.775 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:09:09.797 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:09.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:09.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:09.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:10.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:10.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:10.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:10.145 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:10.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:10.263 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:09:10.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:10.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:10.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:10.395 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:10.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:10.519 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:10.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:10.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:10.649 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:10 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1071680221' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:10 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1071680221' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:10 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1071680221' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:09:10.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:10.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:10.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:10.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:10.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:10.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.017 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:11.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:11.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:11.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:11.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:11.266 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:11.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:11.390 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1071680221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[102314]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[102314]: pgmap v1413: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1071680221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[107038]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T15:09:11.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-mon[107038]: pgmap v1413: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:11.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.515 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:11.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:11.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:11.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:11.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:11 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:11 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1071680221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:09:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:11 smithi172 ceph-mon[100696]: osdmap e447: 8 total, 8 up, 8 in 2024-09-13T15:09:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:11 smithi172 ceph-mon[100696]: pgmap v1413: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:11.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:11.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:11.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:11.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:11.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:12.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:12.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:12.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:09:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:09:11] "GET /metrics HTTP/1.1" 200 36616 "" "Prometheus/2.51.0" 2024-09-13T15:09:12.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:12.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:12.354 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:12.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:12.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:12.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:09:12.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:12.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:12.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:12.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:12.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:12.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:12.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:12.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:12.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:12.997 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:13.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:13.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:13.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:13.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:13.275 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:13.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:13.403 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:13.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:13.505 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:13.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:13.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:13.657 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:13.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:13.784 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:13.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:13.912 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:13.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:14.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:14.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:14.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:14.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:14 smithi154 ceph-mon[107038]: pgmap v1414: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:14 smithi154 ceph-mon[102314]: pgmap v1414: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:14.263 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:14.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:14.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:09:14.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:14.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:14.443 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:14.443 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:14.444 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:14 smithi172 ceph-mon[100696]: pgmap v1414: 73 pgs: 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:14.456 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:09:14.456 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:09:14.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:14.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:14.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:14.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:14.773 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:14.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:14.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:14.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:14.949 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:14.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:15.020 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_5' created 2024-09-13T15:09:15.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:15.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:15.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:15.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:15.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4126936039' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T15:09:15.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4126936039' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T15:09:15.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:15.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:15.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:15.455 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:15.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4126936039' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-13T15:09:15.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:09:15.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.582 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:15.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:15.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:15.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:15.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:15.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:15.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:15.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:15.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:16.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:16.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:16.138 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:16.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:16.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:16.290 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4126936039' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T15:09:16.290 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:16 smithi154 ceph-mon[102314]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T15:09:16.290 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:16 smithi154 ceph-mon[102314]: pgmap v1416: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:16.291 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4126936039' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T15:09:16.291 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:16 smithi154 ceph-mon[107038]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T15:09:16.291 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:16 smithi154 ceph-mon[107038]: pgmap v1416: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:16.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:16.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:16.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:16.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4126936039' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-13T15:09:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:16 smithi172 ceph-mon[100696]: osdmap e448: 8 total, 8 up, 8 in 2024-09-13T15:09:16.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:16 smithi172 ceph-mon[100696]: pgmap v1416: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:16.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:16.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:16.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:16.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:16.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:16.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:16.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:16.951 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:16.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:17.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:17.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:17.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:17.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:17.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:17 smithi154 ceph-mon[102314]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:09:17.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:17 smithi154 ceph-mon[102314]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T15:09:17.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:17 smithi154 ceph-mon[107038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:09:17.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:17 smithi154 ceph-mon[107038]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T15:09:17.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:17.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:17.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:17.452 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:17 smithi172 ceph-mon[100696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:09:17.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:17 smithi172 ceph-mon[100696]: osdmap e449: 8 total, 8 up, 8 in 2024-09-13T15:09:17.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:17.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:17.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:17.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:17.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:17.828 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:17.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-13T15:09:17.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:17.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:17.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:18.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:18.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:18.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:18.207 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:18.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:18 smithi154 ceph-mon[102314]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T15:09:18.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:18 smithi154 ceph-mon[102314]: pgmap v1419: 89 pgs: 8 unknown, 81 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:18 smithi154 ceph-mon[107038]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T15:09:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:18 smithi154 ceph-mon[107038]: pgmap v1419: 89 pgs: 8 unknown, 81 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:18.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:18.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:18.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:18.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:18 smithi172 ceph-mon[100696]: osdmap e450: 8 total, 8 up, 8 in 2024-09-13T15:09:18.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:18 smithi172 ceph-mon[100696]: pgmap v1419: 89 pgs: 8 unknown, 81 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:18.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:18.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:18.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:18.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:18.718 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:18.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:18.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:18.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:18.964 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:18.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:19.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:19.090 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:19.192 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:19.207 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:09:19.216 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:19.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/383096866' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:19.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:19 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/383096866' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:19.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:19 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:19.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:19.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:19.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:19.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:19.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:19.472 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:09:19.472 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:19.490 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:09:19.490 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/383096866' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:19.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:19 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:19.506 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:09:19.506 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:19.522 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:09:19.522 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:19.539 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:09:19.539 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:19.555 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:09:19.555 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:19.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:19.572 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:09:19.572 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:19.588 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:09:19.588 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:19.594 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:19.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:19.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:19.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:19.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:19.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:19.967 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:19.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:20.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:20.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:20.107 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:20.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:20.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:20.315 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:20.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:20.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:20.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:20.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[102314]: pgmap v1420: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:20.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[102314]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1649951626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[107038]: pgmap v1420: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[107038]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T15:09:20.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:20 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1649951626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:20 smithi172 ceph-mon[100696]: pgmap v1420: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:20 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:09:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:20 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:20 smithi172 ceph-mon[100696]: osdmap e451: 8 total, 8 up, 8 in 2024-09-13T15:09:20.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:20 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1649951626' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:20.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:20.592 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:20.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:20.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:20.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:20.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:20.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:20.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:20.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:21.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:21.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:21.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:21.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:21.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:21.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:21.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:21.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:21.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:21.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:21.476 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:09:21.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:21.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:21.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:21.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:21.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:21.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:21.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:21.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:21.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:21.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:09:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:09:21] "GET /metrics HTTP/1.1" 200 34246 "" "Prometheus/2.51.0" 2024-09-13T15:09:22.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:22.083 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:22.185 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:22.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:22.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:22.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:22.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:22.454 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:22.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:22.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:22 smithi154 ceph-mon[102314]: pgmap v1422: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:22.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:22 smithi154 ceph-mon[107038]: pgmap v1422: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:22 smithi172 ceph-mon[100696]: pgmap v1422: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:22.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:22.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:22.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:22.682 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:22.704 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:22.806 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:22.828 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:22.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:09:22.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:22.989 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:22.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:22.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:23.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:23.114 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:23.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:23.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:23.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:23.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:23.364 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:23.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:23.487 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:23.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:23.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:23.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:09:23.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:23.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:23.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:23.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:23.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:23.981 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:23.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213583 8.3 8:c8734954:::benchmark_data_smithi154_261841_object97544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.745266+0000", 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 319.21024330300003, 2024-09-13T15:09:23.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6782159299999999, 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213583 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.745266+0000", 2024-09-13T15:09:23.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.745266+0000", 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.745267+0000", 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2419999999999999e-06 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:23.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.745270+0000", 2024-09-13T15:09:23.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5610000000000001e-06 2024-09-13T15:09:23.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.745278+0000", 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0700000000000007e-06 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.745283+0000", 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2390000000000002e-06 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.747947+0000", 2024-09-13T15:09:23.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026637100000000001 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.748258+0000", 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031145199999999999 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.748651+0000", 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039297699999999997 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.756410+0000", 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077590209999999996 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.771087+0000", 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014677104 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790158+0000", 2024-09-13T15:09:23.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019071022 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790170+0000", 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1977999999999999e-05 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.002603+0000", 2024-09-13T15:09:24.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.21243278700000001 2024-09-13T15:09:24.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.002617+0000", 2024-09-13T15:09:24.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3774999999999999e-05 2024-09-13T15:09:24.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.414868+0000", 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4122510369999999 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.415151+0000", 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028311599999999998 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.423398+0000", 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082473000000000008 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.423482+0000", 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3539000000000001e-05 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213604 8.7 8:e76edc7f:::benchmark_data_smithi154_261841_object97565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.790069+0000", 2024-09-13T15:09:24.003 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 319.165440418, 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6291619310000001, 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213604 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790069+0000", 2024-09-13T15:09:24.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790069+0000", 2024-09-13T15:09:24.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790071+0000", 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7859999999999999e-06 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790078+0000", 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9970000000000001e-06 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790093+0000", 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4929999999999999e-05 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790101+0000", 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1070000000000005e-06 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790442+0000", 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000341589 2024-09-13T15:09:24.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.790466+0000", 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3295999999999999e-05 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.007094+0000", 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.216628717 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.007104+0000", 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9010000000000006e-06 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.406205+0000", 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3991011760000001 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.406349+0000", 2024-09-13T15:09:24.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000144146 2024-09-13T15:09:24.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.010 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3,6", 2024-09-13T15:09:24.010 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.406517+0000", 2024-09-13T15:09:24.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016765500000000001 2024-09-13T15:09:24.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.410604+0000", 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040871780000000003 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.415580+0000", 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049759230000000002 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.417693+0000", 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021125979999999998 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419060+0000", 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013671620000000001 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419111+0000", 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1437999999999998e-05 2024-09-13T15:09:24.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419231+0000", 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119332 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213621 8.7 8:e1de72c7:::benchmark_data_smithi154_261841_object97582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.808994+0000", 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 319.14651544100002, 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6104004750000001, 2024-09-13T15:09:24.013 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213621 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.014 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.015 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.808994+0000", 2024-09-13T15:09:24.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.808994+0000", 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.808994+0000", 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3799999999999997e-07 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.808997+0000", 2024-09-13T15:09:24.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7760000000000002e-06 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.809008+0000", 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0536999999999999e-05 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.809013+0000", 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4790000000000004e-06 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.809063+0000", 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9481999999999998e-05 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.809071+0000", 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5650000000000002e-06 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.007286+0000", 2024-09-13T15:09:24.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.19821455399999999 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.007295+0000", 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2809999999999997e-06 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.410653+0000", 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4033579139999999 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.410788+0000", 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000134845 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3,6", 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.410926+0000", 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013798099999999999 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.415221+0000", 2024-09-13T15:09:24.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042954660000000004 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.415641+0000", 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041957000000000002 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419017+0000", 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033764960000000001 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419261+0000", 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024408500000000001 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419306+0000", 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4180000000000001e-05 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419394+0000", 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8726000000000004e-05 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.022 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.35787.0:213626 8.3 8:c1015b60:::benchmark_data_smithi154_261841_object97587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:04:04.819874+0000", 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 319.135635347, 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.604538, 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.35787", 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 213626 2024-09-13T15:09:24.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819874+0000", 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819874+0000", 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819875+0000", 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0139999999999999e-06 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819880+0000", 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9690000000000002e-06 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819891+0000", 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1005999999999999e-05 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819896+0000", 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3550000000000004e-06 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819951+0000", 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4732999999999999e-05 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:04.819959+0000", 2024-09-13T15:09:24.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9859999999999994e-06 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.002643+0000", 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.18268411600000001 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:05.002651+0000", 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0669999999999992e-06 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.415558+0000", 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.412906719 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.415686+0000", 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012860500000000001 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5,6", 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.415815+0000", 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000128813 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419159+0000", 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033432760000000001 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.419714+0000", 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055511600000000003 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.420398+0000", 2024-09-13T15:09:24.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00068404000000000002 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.424184+0000", 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037867059999999999 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.424297+0000", 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011244 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:04:06.424412+0000", 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011503899999999999 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18352 9.d 9:bdc393da:::benchmark_data_smithi154_291684_object18351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.155820+0000", 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 247.799689449, 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0755706260000002, 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.031 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18352 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.155820+0000", 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.155820+0000", 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.155821+0000", 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.714e-06 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.155849+0000", 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7654999999999999e-05 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.156962+0000", 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011125250000000001 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.156972+0000", 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0237999999999999e-05 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157029+0000", 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7172000000000001e-05 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157036+0000", 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4219999999999997e-06 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.158805+0000", 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001769098 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.158814+0000", 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8859999999999999e-06 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149600+0000", 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99078543699999999 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149615+0000", 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5221999999999999e-05 2024-09-13T15:09:24.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.230842+0000", 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0812272570000001 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.231390+0000", 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054799999999999998 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18368 9.d 9:b2794085:::benchmark_data_smithi154_291684_object18367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.156994+0000", 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 247.79851570299999, 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0831438489999998, 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18368 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.156994+0000", 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.156994+0000", 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.156995+0000", 2024-09-13T15:09:24.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1289999999999999e-06 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157023+0000", 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8822e-05 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157033+0000", 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4159999999999993e-06 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157040+0000", 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6440000000000003e-06 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157076+0000", 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5985000000000002e-05 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157081+0000", 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1490000000000002e-06 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.158838+0000", 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017570050000000001 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.158845+0000", 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8639999999999998e-06 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149645+0000", 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99080054699999998 2024-09-13T15:09:24.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149653+0000", 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3350000000000007e-06 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.043 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.231417+0000", 2024-09-13T15:09:24.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0817637529999999 2024-09-13T15:09:24.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.231621+0000", 2024-09-13T15:09:24.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000204081 2024-09-13T15:09:24.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:09:24.045 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.231801+0000", 2024-09-13T15:09:24.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018009999999999999 2024-09-13T15:09:24.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.237514+0000", 2024-09-13T15:09:24.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057130619999999997 2024-09-13T15:09:24.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.238999+0000", 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014846 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.239249+0000", 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00025022800000000002 2024-09-13T15:09:24.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.239337+0000", 2024-09-13T15:09:24.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7646999999999996e-05 2024-09-13T15:09:24.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.240137+0000", 2024-09-13T15:09:24.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080048200000000004 2024-09-13T15:09:24.049 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.049 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.049 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.050 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18373 9.d 9:bd85eaaa:::benchmark_data_smithi154_291684_object18372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:09:24.050 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.168984+0000", 2024-09-13T15:09:24.050 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 247.78652509299999, 2024-09-13T15:09:24.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.071990129, 2024-09-13T15:09:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.051 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18373 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.168984+0000", 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.168984+0000", 2024-09-13T15:09:24.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.168986+0000", 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9400000000000001e-06 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169016+0000", 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9912000000000001e-05 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169038+0000", 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2154000000000002e-05 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169050+0000", 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1432e-05 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169119+0000", 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9757000000000003e-05 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169132+0000", 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2376e-05 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149678+0000", 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98054579399999997 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149686+0000", 2024-09-13T15:09:24.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.85e-06 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.233090+0000", 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0834039900000001 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.233263+0000", 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000172275 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.233439+0000", 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017610600000000001 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.237523+0000", 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004084346 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.239037+0000", 2024-09-13T15:09:24.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015137709999999999 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.240321+0000", 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012844969999999999 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.240382+0000", 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0764000000000003e-05 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.240974+0000", 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00059216499999999997 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18381 9.d 9:b51423d5:::benchmark_data_smithi154_291684_object18380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.174094+0000", 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 247.78141532999999, 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.068140085, 2024-09-13T15:09:24.059 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18381 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174094+0000", 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.061 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174094+0000", 2024-09-13T15:09:24.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174095+0000", 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210000000000001e-06 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174113+0000", 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8406e-05 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174135+0000", 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2050999999999999e-05 2024-09-13T15:09:24.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174145+0000", 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6309999999999998e-06 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174224+0000", 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9103999999999993e-05 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.174235+0000", 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1012000000000001e-05 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149710+0000", 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97547453699999997 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149718+0000", 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1019999999999997e-06 2024-09-13T15:09:24.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.235346+0000", 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085627954 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.235830+0000", 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048466699999999999 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.236200+0000", 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00036973099999999999 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.240165+0000", 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039648080000000002 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.241195+0000", 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001029645 2024-09-13T15:09:24.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.241552+0000", 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000357778 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.241622+0000", 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9890999999999999e-05 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.242234+0000", 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00061174700000000005 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18384 9.d 9:b20d0c82:::benchmark_data_smithi154_291684_object18383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.176828+0000", 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 247.778681576, 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0659570220000001, 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.068 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18384 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176828+0000", 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176828+0000", 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176829+0000", 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.37e-06 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176856+0000", 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6534999999999999e-05 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176875+0000", 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9840999999999999e-05 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176881+0000", 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4450000000000004e-06 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176935+0000", 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4487999999999997e-05 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.176948+0000", 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3078000000000001e-05 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149743+0000", 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97279444000000004 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.149750+0000", 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1409999999999999e-06 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.237746+0000", 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879959299999999 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.237925+0000", 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000179574 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,5", 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.238097+0000", 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017143599999999999 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.240172+0000", 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020750090000000001 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.241243+0000", 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001070842 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.242286+0000", 2024-09-13T15:09:24.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010431349999999999 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.242345+0000", 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8953000000000003e-05 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.242785+0000", 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043980499999999998 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.076 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:140911 9.10 9:08447db5:::benchmark_data_smithi154_291684_object40450:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291497+0000", 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.66401216, 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.202716688, 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.077 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 140911 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291497+0000", 2024-09-13T15:09:24.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291497+0000", 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291498+0000", 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8800000000000002e-07 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291501+0000", 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6550000000000002e-06 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291511+0000", 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.251e-06 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291517+0000", 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4500000000000001e-06 2024-09-13T15:09:24.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291555+0000", 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.824e-05 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291560+0000", 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6319999999999999e-06 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293397+0000", 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018375819999999999 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293405+0000", 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7950000000000008e-06 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.485665+0000", 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1922600329999999 2024-09-13T15:09:24.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.485787+0000", 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121691 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.486001+0000", 2024-09-13T15:09:24.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000213845 2024-09-13T15:09:24.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.487944+0000", 2024-09-13T15:09:24.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001943132 2024-09-13T15:09:24.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.487991+0000", 2024-09-13T15:09:24.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7101999999999997e-05 2024-09-13T15:09:24.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493402+0000", 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005410804 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493548+0000", 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014655399999999999 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494037+0000", 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048849099999999997 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494095+0000", 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8155999999999997e-05 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494214+0000", 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118687 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141022 9.10 9:0f0fdebc:::benchmark_data_smithi154_291684_object40561:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291528+0000", 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.66398108, 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.202903995, 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.090 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141022 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291528+0000", 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291528+0000", 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291529+0000", 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2500000000000001e-07 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291531+0000", 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2330000000000001e-06 2024-09-13T15:09:24.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291539+0000", 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7489999999999998e-06 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291544+0000", 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7309999999999998e-06 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291590+0000", 2024-09-13T15:09:24.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5244000000000003e-05 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291595+0000", 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5419999999999996e-06 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293438+0000", 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001843157 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293445+0000", 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9879999999999998e-06 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.488047+0000", 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.194601429 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.489766+0000", 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001719555 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.490107+0000", 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034030600000000002 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493408+0000", 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003301568 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493450+0000", 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1712000000000003e-05 2024-09-13T15:09:24.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493502+0000", 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2663e-05 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493975+0000", 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047236799999999997 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494264+0000", 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028906299999999999 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494319+0000", 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4864000000000002e-05 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494432+0000", 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113398 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.098 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141039 9.10 9:09553730:::benchmark_data_smithi154_291684_object40578:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291556+0000", 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.66395274600001, 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.203133778, 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.099 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141039 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291556+0000", 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291556+0000", 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291557+0000", 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.82e-07 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291559+0000", 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.176e-06 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291566+0000", 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6069999999999996e-06 2024-09-13T15:09:24.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291572+0000", 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8170000000000004e-06 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291617+0000", 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5865999999999998e-05 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291622+0000", 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0629999999999996e-06 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293492+0000", 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018691300000000001 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293500+0000", 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3250000000000008e-06 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.491618+0000", 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1981177329999999 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.491764+0000", 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014618199999999999 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.492021+0000", 2024-09-13T15:09:24.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00025746200000000002 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493855+0000", 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001834139 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493878+0000", 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2707000000000001e-05 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.493925+0000", 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7037999999999999e-05 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494476+0000", 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055046800000000003 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494516+0000", 2024-09-13T15:09:24.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0778000000000002e-05 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494588+0000", 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1994000000000002e-05 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494690+0000", 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101811 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.107 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141045 9.10 9:0810491a:::benchmark_data_smithi154_291684_object40584:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291583+0000", 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.663925885, 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.213193221, 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.108 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141045 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291583+0000", 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291583+0000", 2024-09-13T15:09:24.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291584+0000", 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4299999999999998e-07 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291586+0000", 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1150000000000001e-06 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291592+0000", 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1709999999999999e-06 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291597+0000", 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9540000000000003e-06 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291648+0000", 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1038000000000002e-05 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291653+0000", 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6280000000000004e-06 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293542+0000", 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018894929999999999 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293551+0000", 2024-09-13T15:09:24.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6400000000000003e-06 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494724+0000", 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2011729710000001 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.494840+0000", 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116123 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.495126+0000", 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028620700000000002 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504414+0000", 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092876190000000004 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504460+0000", 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6279000000000002e-05 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504474+0000", 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4097000000000001e-05 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504521+0000", 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7184000000000001e-05 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504563+0000", 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1297000000000001e-05 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504629+0000", 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6315999999999993e-05 2024-09-13T15:09:24.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504777+0000", 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000147646 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141050 9.10 9:08ec8708:::benchmark_data_smithi154_291684_object40589:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291629+0000", 2024-09-13T15:09:24.117 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.663880029, 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.21573978, 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141050 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291629+0000", 2024-09-13T15:09:24.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291629+0000", 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291630+0000", 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0500000000000002e-07 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291634+0000", 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0040000000000001e-06 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291647+0000", 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3162999999999999e-05 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291655+0000", 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1669999999999999e-06 2024-09-13T15:09:24.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291699+0000", 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3624000000000001e-05 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291704+0000", 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2009999999999998e-06 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293582+0000", 2024-09-13T15:09:24.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001877919 2024-09-13T15:09:24.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293589+0000", 2024-09-13T15:09:24.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2579999999999998e-06 2024-09-13T15:09:24.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:24.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.496408+0000", 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2028185709999999 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.496515+0000", 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107155 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.496733+0000", 2024-09-13T15:09:24.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021781900000000001 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504480+0000", 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077474019999999996 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504817+0000", 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033662500000000001 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504856+0000", 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8654e-05 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507134+0000", 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002278764 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507189+0000", 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4494999999999999e-05 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507246+0000", 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7370000000000001e-05 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507369+0000", 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012268399999999999 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141073 9.10 9:0e4d1f18:::benchmark_data_smithi154_291684_object40612:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291671+0000", 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.663838216, 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2160390089999999, 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141073 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.128 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291671+0000", 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291671+0000", 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291672+0000", 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2799999999999996e-07 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291674+0000", 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1399999999999998e-06 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291681+0000", 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8279999999999997e-06 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291686+0000", 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1959999999999999e-06 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291731+0000", 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5164999999999999e-05 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291736+0000", 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7709999999999997e-06 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293958+0000", 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022219520000000001 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293966+0000", 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9549999999999992e-06 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.499117+0000", 2024-09-13T15:09:24.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.205151426 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.499256+0000", 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013941 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.499567+0000", 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031089399999999998 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.504486+0000", 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049191929999999997 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507411+0000", 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029240289999999999 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507447+0000", 2024-09-13T15:09:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6638999999999999e-05 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507486+0000", 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9190999999999998e-05 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507528+0000", 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2077999999999999e-05 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507583+0000", 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4886999999999998e-05 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507710+0000", 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012672700000000001 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141133 9.10 9:0d1f1530:::benchmark_data_smithi154_291684_object40672:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.136 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291693+0000", 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.663816411, 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.216393098, 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141133 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291693+0000", 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291693+0000", 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291693+0000", 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6300000000000001e-07 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291700+0000", 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6309999999999997e-06 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291706+0000", 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4690000000000002e-06 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291713+0000", 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2099999999999996e-06 2024-09-13T15:09:24.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291795+0000", 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1538000000000004e-05 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291800+0000", 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8269999999999997e-06 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293992+0000", 2024-09-13T15:09:24.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00219243 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.293999+0000", 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.742e-06 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.502531+0000", 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.208531491 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.502660+0000", 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012913700000000001 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.502972+0000", 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031255399999999999 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507147+0000", 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041750870000000001 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507754+0000", 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060647700000000001 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507796+0000", 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2438999999999998e-05 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507837+0000", 2024-09-13T15:09:24.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0448999999999998e-05 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507889+0000", 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1882999999999998e-05 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.507939+0000", 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0736999999999999e-05 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.508086+0000", 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014663400000000001 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141167 9.10 9:09ce90c9:::benchmark_data_smithi154_291684_object40706:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291769+0000", 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.66373999699999, 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2425973020000001, 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.145 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141167 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291769+0000", 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291769+0000", 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291770+0000", 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5000000000000003e-07 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291772+0000", 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.604e-06 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291781+0000", 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3359999999999995e-06 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291787+0000", 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9880000000000001e-06 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291824+0000", 2024-09-13T15:09:24.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7200999999999998e-05 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291828+0000", 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0099999999999997e-06 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294023+0000", 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021946819999999999 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294029+0000", 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2310000000000001e-06 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.508159+0000", 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2141304340000001 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.508291+0000", 2024-09-13T15:09:24.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000132192 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.511724+0000", 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034322990000000002 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525550+0000", 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013826409 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525601+0000", 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1277999999999999e-05 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525647+0000", 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5899000000000002e-05 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525691+0000", 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3557000000000003e-05 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534180+0000", 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084895419999999992 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534240+0000", 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9963999999999997e-05 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534367+0000", 2024-09-13T15:09:24.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012612600000000001 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141243 9.10 9:0e7b59e9:::benchmark_data_smithi154_291684_object40782:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291796+0000", 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.663713468, 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.24274581, 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141243 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291796+0000", 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291796+0000", 2024-09-13T15:09:24.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291796+0000", 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9799999999999999e-07 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291798+0000", 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8819999999999999e-06 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291804+0000", 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.178e-06 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291811+0000", 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4470000000000003e-06 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291853+0000", 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1953999999999998e-05 2024-09-13T15:09:24.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291871+0000", 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8036000000000001e-05 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294053+0000", 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021824510000000002 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294059+0000", 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.314e-06 2024-09-13T15:09:24.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.513139+0000", 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.219079542 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.513263+0000", 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012423600000000001 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.513529+0000", 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026612100000000002 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525735+0000", 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012205522 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525774+0000", 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9075000000000001e-05 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525814+0000", 2024-09-13T15:09:24.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0473999999999999e-05 2024-09-13T15:09:24.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525856+0000", 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1498999999999997e-05 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534374+0000", 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085182020000000008 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534426+0000", 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1848999999999998e-05 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534542+0000", 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011563 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141250 9.10 9:0bae4a86:::benchmark_data_smithi154_291684_object40789:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291825+0000", 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.663684162, 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2475243599999999, 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.163 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141250 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291825+0000", 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291825+0000", 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291825+0000", 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1899999999999998e-07 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291828+0000", 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0379999999999998e-06 2024-09-13T15:09:24.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291834+0000", 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9639999999999997e-06 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291841+0000", 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2280000000000003e-06 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294085+0000", 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022440239999999998 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294091+0000", 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4300000000000003e-06 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.525892+0000", 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2318012620000001 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.526020+0000", 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012743499999999999 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.526392+0000", 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037246300000000001 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534167+0000", 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007774825 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534549+0000", 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038172400000000001 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534599+0000", 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0077000000000002e-05 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534646+0000", 2024-09-13T15:09:24.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6910000000000003e-05 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539164+0000", 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045181420000000002 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539223+0000", 2024-09-13T15:09:24.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9012999999999999e-05 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539349+0000", 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000126406 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141263 9.10 9:0810bf7a:::benchmark_data_smithi154_291684_object40802:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e412)", 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:28.291849+0000", 2024-09-13T15:09:24.171 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.663660079, 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2478335220000001, 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141263 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.172 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291849+0000", 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291849+0000", 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291849+0000", 2024-09-13T15:09:24.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1100000000000002e-07 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291851+0000", 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.95e-06 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291870+0000", 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8403e-05 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.291876+0000", 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9519999999999999e-06 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294174+0000", 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022985409999999999 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:28.294182+0000", 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5320000000000001e-06 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.527914+0000", 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2337323739999999 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.528033+0000", 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011928400000000001 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2,4,6", 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.528275+0000", 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241642 2024-09-13T15:09:24.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.534554+0000", 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006279242 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539394+0000", 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048400589999999999 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539434+0000", 2024-09-13T15:09:24.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9598999999999998e-05 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539475+0000", 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0691000000000001e-05 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539513+0000", 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8686999999999997e-05 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:09:24.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539569+0000", 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5926e-05 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.539683+0000", 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113329 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:09:24.179 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:09:24.202 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:24.207 INFO:teuthology.orchestra.run.smithi154.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-13T15:09:24.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:24.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:24.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:24.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:24 smithi154 ceph-mon[102314]: pgmap v1423: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:24.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1624393188' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:09:24.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:24 smithi154 ceph-mon[107038]: pgmap v1423: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:24.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1624393188' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:09:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:24 smithi172 ceph-mon[100696]: pgmap v1423: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1624393188' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-13T15:09:24.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:24.572 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:24.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:24.695 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:24.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.819 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:24.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:24.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:24.942 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:24.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:25.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:25.066 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:25.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:09:25.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.195 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:25.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:25.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:25.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:25.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.441 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:25.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:25.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1624393188' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:09:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[102314]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T15:09:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:25.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:25.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1624393188' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:09:25.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[107038]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T15:09:25.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:25.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:25 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1624393188' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:09:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:25 smithi172 ceph-mon[100696]: osdmap e452: 8 total, 8 up, 8 in 2024-09-13T15:09:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:25.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:25.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:25.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:25.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:25.686 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:25.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:25.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:25.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:25.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:25.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:26.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:26.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:26.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:26.183 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:26.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:26.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:26.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:26.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:26.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:26 smithi154 ceph-mon[102314]: pgmap v1425: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:26.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:26 smithi154 ceph-mon[102314]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:09:26.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:26 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:09:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:26 smithi154 ceph-mon[107038]: pgmap v1425: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:26 smithi154 ceph-mon[107038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:09:26.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:26 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:09:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:26 smithi172 ceph-mon[100696]: pgmap v1425: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:26 smithi172 ceph-mon[100696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:09:26.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:26 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:09:26.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:26.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:26.562 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:26.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:26.679 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:26.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:26.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:26.803 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:26.884 DEBUG:teuthology.orchestra.run.smithi154:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-13T15:09:26.885 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T15:09:26.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:09:26.888 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-13T15:09:26.888 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-13T15:09:26.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:26.928 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:26.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:26.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:hints = 1 2024-09-13T15:09:26.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-13T15:09:26.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object prefix: benchmark_data_smithi154_317818 2024-09-13T15:09:26.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:09:26.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 0 0 0 0 0 0 - 0 2024-09-13T15:09:27.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:27.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:27.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:27.213 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:27.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:27.353 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:27.455 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:27.489 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:27.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:27 smithi172 ceph-mon[100696]: pgmap v1426: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:27.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:27.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:27.690 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:27 smithi154 ceph-mon[102314]: pgmap v1426: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:27.690 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:27 smithi154 ceph-mon[107038]: pgmap v1426: 89 pgs: 89 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:09:27.726 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:27.744 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:27.756 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:27.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:27.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:27.892 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:27.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:27.938 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 1 15 3674 3659 228.675 228.688 0.00316326 0.00435538 2024-09-13T15:09:27.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:28.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:28.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:28.160 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:28.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:28.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:28.396 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:28.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:28.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:28.560 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:28.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:28.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:28.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:28.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:28.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:28.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 2 16 7307 7291 227.651 227 0.0030035 0.00438269 2024-09-13T15:09:28.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:29.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:29.126 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:29.159 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:29.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:29.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:29.308 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:29 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:29.308 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:29 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:29.308 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:29 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:29.308 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:29 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:29.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:29.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:29 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:29.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:29 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:29.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:29.579 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:29.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:29.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:29.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:29.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:29.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 3 16 10391 10375 216.02 192.75 0.00240255 0.00462056 2024-09-13T15:09:29.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:29.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:29.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:30.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:30.126 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:30.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:30.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:30.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:30.399 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:30.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:30.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:30 smithi154 ceph-mon[102314]: pgmap v1427: 89 pgs: 89 active+clean; 216 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s wr, 338 op/s 2024-09-13T15:09:30.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:30 smithi154 ceph-mon[107038]: pgmap v1427: 89 pgs: 89 active+clean; 216 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s wr, 338 op/s 2024-09-13T15:09:30.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:30 smithi172 ceph-mon[100696]: pgmap v1427: 89 pgs: 89 active+clean; 216 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 21 MiB/s wr, 338 op/s 2024-09-13T15:09:30.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:30.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:30.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:30.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:30.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:30.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:30.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 4 16 13628 13612 212.585 202.312 0.00628815 0.00469611 2024-09-13T15:09:30.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:30.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:31.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:31.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:31.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:31.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:31.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.352 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:31.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:31.453 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:31.487 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:31.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:31.621 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:31.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:31 smithi154 ceph-mon[102314]: pgmap v1428: 89 pgs: 89 active+clean; 321 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 32 MiB/s wr, 506 op/s 2024-09-13T15:09:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:31 smithi154 ceph-mon[107038]: pgmap v1428: 89 pgs: 89 active+clean; 321 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 32 MiB/s wr, 506 op/s 2024-09-13T15:09:31.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:31 smithi172 ceph-mon[100696]: pgmap v1428: 89 pgs: 89 active+clean; 321 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 32 MiB/s wr, 506 op/s 2024-09-13T15:09:31.750 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:31.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:31.890 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:31.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:31.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 5 16 16137 16121 201.431 156.812 0.00410263 0.00495753 2024-09-13T15:09:31.992 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:32.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:32.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:32.136 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:09:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:09:31] "GET /metrics HTTP/1.1" 200 36589 "" "Prometheus/2.51.0" 2024-09-13T15:09:32.166 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:32.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:32.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:32.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:32.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:32.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:32.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:32.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:32.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:32.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:32.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:32.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 6 16 18274 18258 190.121 133.562 0.00149203 0.00522602 2024-09-13T15:09:32.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:32.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:32.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:33.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:33.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:33.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:33.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:33.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:33.375 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:33.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:33.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:33.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:33.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:33.683 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:33 smithi154 ceph-mon[102314]: pgmap v1429: 89 pgs: 89 active+clean; 692 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T15:09:33.684 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:33 smithi154 ceph-mon[107038]: pgmap v1429: 89 pgs: 89 active+clean; 692 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T15:09:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:33 smithi172 ceph-mon[100696]: pgmap v1429: 89 pgs: 89 active+clean; 692 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-13T15:09:33.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:33.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:33.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:33.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:33.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:33.940 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 7 16 20188 20172 180.051 119.625 0.00151463 0.0055416 2024-09-13T15:09:34.035 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:34.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:34.170 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:34.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:34.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.328 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:34.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:34.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:34.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:34.568 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:34.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:34.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:34.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:34 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2617268956' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:34 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2617268956' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:34 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2617268956' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:09:34.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:34.844 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:34.880 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:34.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:34.908 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:09:34.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 8 16 22223 22207 173.443 127.188 0.00571091 0.00575989 2024-09-13T15:09:34.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:35.013 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:35.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:35.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:35.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:35.283 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:35.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:35.410 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:35.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:35.511 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:35.546 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:35.648 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:35.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2617268956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:09:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[102314]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T15:09:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[102314]: pgmap v1431: 89 pgs: 89 active+clean; 1.1 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T15:09:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2617268956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:09:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[107038]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T15:09:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:35 smithi154 ceph-mon[107038]: pgmap v1431: 89 pgs: 89 active+clean; 1.1 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T15:09:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:35 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2617268956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:09:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:35 smithi172 ceph-mon[100696]: osdmap e453: 8 total, 8 up, 8 in 2024-09-13T15:09:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:35 smithi172 ceph-mon[100696]: pgmap v1431: 89 pgs: 89 active+clean; 1.1 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-13T15:09:35.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:35.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:35.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:35.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 9 16 24434 24418 169.525 138.188 0.00580633 0.00589268 2024-09-13T15:09:35.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:35.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:36.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:36.093 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:36.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:36.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:36.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:36.364 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:36.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:36.499 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:36.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:36.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:36.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:36.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:36.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:36.912 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:36.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:36.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 10 16 26718 26702 166.847 142.75 0.00769207 0.00598811 2024-09-13T15:09:37.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:37.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:37.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:37.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:37.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:37.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:37.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:37.420 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:37.443 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:37.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:37.566 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:37.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:37.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:37 smithi154 ceph-mon[102314]: pgmap v1432: 89 pgs: 89 active+clean; 1.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-13T15:09:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:37 smithi154 ceph-mon[107038]: pgmap v1432: 89 pgs: 89 active+clean; 1.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-13T15:09:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:37 smithi172 ceph-mon[100696]: pgmap v1432: 89 pgs: 89 active+clean; 1.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-13T15:09:37.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:37.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:37.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:37.941 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 11 16 28385 28369 161.14 104.188 0.0110783 0.00619947 2024-09-13T15:09:37.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:37.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:38.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:38.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:38.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:38.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:38.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:38.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:38.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:38.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:38.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:38.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:38.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:38.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:38.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:38.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:38.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:38.942 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 12 16 30600 30584 159.244 138.438 0.00562406 0.00627256 2024-09-13T15:09:39.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:39.066 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:39.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:39.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:39.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.325 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:39.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:39.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:39.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:39.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:39.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:39.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:39.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:39 smithi154 ceph-mon[102314]: pgmap v1433: 89 pgs: 89 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s 2024-09-13T15:09:39.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:39 smithi154 ceph-mon[107038]: pgmap v1433: 89 pgs: 89 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s 2024-09-13T15:09:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:39 smithi172 ceph-mon[100696]: pgmap v1433: 89 pgs: 89 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 154 MiB/s wr, 2.47k op/s 2024-09-13T15:09:39.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:39.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:39.942 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 13 16 32508 32492 156.167 119.25 0.0053994 0.00639826 2024-09-13T15:09:39.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:39.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:39.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:40.096 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:40.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:40.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:40.256 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:40.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:40.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:40.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:40.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:40.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:40.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:40.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:40.797 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:40.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:40.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:40.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:40.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 14 16 34626 34610 154.468 132.375 0.00776768 0.00646887 2024-09-13T15:09:41.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:41.046 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:40 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1291073551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:41.046 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:40 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1291073551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:41.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:41.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:41.217 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:40 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1291073551' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:41.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:41.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:41.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:41.508 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:41.553 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:09:41.553 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:09:41.553 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:41.554 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:41.555 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:41.555 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:41.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:41.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:41.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:41.758 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:41.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:41.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:41.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:41.931 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:41.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:41.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 15 16 36689 36673 152.759 128.938 0.00829783 0.00654082 2024-09-13T15:09:42.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:42.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:42.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:42.176 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:42 smithi154 ceph-mon[107038]: pgmap v1434: 89 pgs: 89 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-09-13T15:09:42.177 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:09:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:09:41] "GET /metrics HTTP/1.1" 200 36589 "" "Prometheus/2.51.0" 2024-09-13T15:09:42.177 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:42 smithi154 ceph-mon[102314]: pgmap v1434: 89 pgs: 89 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-09-13T15:09:42.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:42.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:42.339 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:42.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:42.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:41 smithi172 ceph-mon[100696]: pgmap v1434: 89 pgs: 89 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-09-13T15:09:42.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:42.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:42.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:42.744 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:42.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:42.881 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:42.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:42.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 16 16 38836 38820 151.598 134.188 0.00395786 0.00659093 2024-09-13T15:09:42.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:43.014 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:43.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.139 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:43.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:43.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:43.271 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:43.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:43.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:09:43.412 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:43.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:43.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:43.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:43.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:43.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:43.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:43.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:43.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 17 16 40534 40518 148.923 106.125 0.00484897 0.00670961 2024-09-13T15:09:43.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:43.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:44.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:44.112 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:44.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:44.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:09:44.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:44.293 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:44.293 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:44.293 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:44 smithi172 ceph-mon[100696]: pgmap v1435: 89 pgs: 89 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-13T15:09:44.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:44.374 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:44.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:44.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:44.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:44 smithi154 ceph-mon[102314]: pgmap v1435: 89 pgs: 89 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-13T15:09:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:44.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:44 smithi154 ceph-mon[107038]: pgmap v1435: 89 pgs: 89 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-13T15:09:44.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:44.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:44.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.647 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:44.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:44.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:44.786 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:44.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:44.920 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:44.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:44.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 18 16 42445 42429 147.285 119.438 0.00620257 0.00678482 2024-09-13T15:09:45.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:45.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:45.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:45.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:45.280 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:45.312 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:45.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.451 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:45.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:45.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:45.559 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:45 smithi154 ceph-mon[102314]: pgmap v1436: 89 pgs: 89 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T15:09:45.559 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:45 smithi154 ceph-mon[107038]: pgmap v1436: 89 pgs: 89 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T15:09:45.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:45.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:45.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:45.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:45 smithi172 ceph-mon[100696]: pgmap v1436: 89 pgs: 89 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-13T15:09:45.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:45.847 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:45.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 19 16 43912 43896 144.359 91.6875 0.00422964 0.00692149 2024-09-13T15:09:45.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:45.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:45.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:45.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:46.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:46.122 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:46.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:46.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:46.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:46.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:46.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:46.397 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:46.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:46.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:46.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:46.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:46.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:46.673 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:46.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:46.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:46.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:46.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:46.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:46.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:09:46.942961+0000 min lat: 0.00103447 max lat: 0.220427 avg lat: 0.00699329 2024-09-13T15:09:46.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:09:46.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 20 16 45741 45725 142.856 114.312 0.0210788 0.00699329 2024-09-13T15:09:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:46.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:46.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:47.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:47.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:47.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:47.090 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:47.092 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:09:47.092 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:09:47.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:47.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:47.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:47.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:47.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:47.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:47.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:47 smithi154 ceph-mon[102314]: pgmap v1437: 89 pgs: 89 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T15:09:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:47 smithi154 ceph-mon[107038]: pgmap v1437: 89 pgs: 89 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T15:09:47.498 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:47.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:47.600 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:47.631 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:47.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:47.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:47.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:47 smithi172 ceph-mon[100696]: pgmap v1437: 89 pgs: 89 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-09-13T15:09:47.765 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:47.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:47.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:47.892 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:47.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:47.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 21 16 47455 47439 141.154 107.125 0.00987404 0.00707835 2024-09-13T15:09:47.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:48.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:48.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:48.171 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:48.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:48.301 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:48.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:48.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:48.536 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:09:48.537 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:09:48.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:48.569 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:48.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:48.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:48.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:48.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:48.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:09:48.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 22 16 49147 49131 139.543 105.75 0.00884679 0.00716033 2024-09-13T15:09:48.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:48.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:48.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:49.091 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:49.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:49.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:49.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:49.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:49.256 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:49.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:49.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:49.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:49.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:49.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:49.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:49.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:49.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:49.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:49.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:49.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:49 smithi154 ceph-mon[102314]: from='client.46205 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:49.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:49 smithi154 ceph-mon[102314]: pgmap v1438: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T15:09:49.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:49 smithi154 ceph-mon[107038]: from='client.46205 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:49.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:49 smithi154 ceph-mon[107038]: pgmap v1438: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T15:09:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:49 smithi172 ceph-mon[100696]: from='client.46205 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:09:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:49 smithi172 ceph-mon[100696]: pgmap v1438: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-13T15:09:49.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:49.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:49.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:49.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:49.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:49.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:49.943 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 23 16 50824 50808 138.033 104.812 0.0060787 0.00723986 2024-09-13T15:09:50.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:50.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:50.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:50.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:50.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:50.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:50.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:50.346 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:50.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:50.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:50.476 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:50.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:50.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:50.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:50.606 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:50.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:50.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:50.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:50.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:50.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:50.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:50.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:50.944 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 24 16 52455 52439 136.529 101.938 0.00301915 0.00731591 2024-09-13T15:09:50.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:51.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:51.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:51.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:51.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:51.262 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:51.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:51.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:51.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:51.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:51.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:51.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:51.771 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:51.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:51.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:51.944 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 25 16 53865 53849 134.59 88.125 0.00621554 0.00742519 2024-09-13T15:09:51.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:51.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:51.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:52.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:52.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:52.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:09:51] "GET /metrics HTTP/1.1" 200 36656 "" "Prometheus/2.51.0" 2024-09-13T15:09:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[102314]: pgmap v1439: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T15:09:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:09:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:09:52.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[107038]: pgmap v1439: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T15:09:52.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:09:52.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:09:52.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:51 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:51 smithi172 ceph-mon[100696]: pgmap v1439: 89 pgs: 89 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-13T15:09:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:09:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:09:52.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:51 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:09:52.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:52.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:52.315 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1437,"stamp":"2024-09-13T15:09:47.177373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2798655461,"num_objects":42659,"num_object_clones":0,"num_object_copies":127977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42659,"num_whiteouts":0,"num_read":6319,"num_read_kb":13776,"num_write":43754,"num_write_kb":2743425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":4169211,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43813,"ondisk_log_size":43813,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":268,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12860968,"kb_used_data":7926824,"kb_used_omap":40,"kb_used_meta":4934039,"kb_avail":737100248,"statfs":{"total":767960285184,"available":754790653952,"internally_reserved":0,"allocated":8117067776,"data_stored":8097113559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":5052456052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1651441664,"num_objects":25199,"num_object_clones":0,"num_object_copies":75597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25199,"num_write_kb":1612736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":129,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.364259+0000","last_change":"2024-09-13T15:08:31.190995+0000","last_active":"2024-09-13T15:09:34.364259+0000","last_peered":"2024-09-13T15:09:34.364259+0000","last_clean":"2024-09-13T15:09:34.364259+0000","last_became_active":"2024-09-13T15:08:31.190610+0000","last_became_peered":"2024-09-13T15:08:31.190610+0000","last_unstale":"2024-09-13T15:09:34.364259+0000","last_undegraded":"2024-09-13T15:09:34.364259+0000","last_fullsized":"2024-09-13T15:09:34.364259+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:42.124757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":33,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.725822+0000","last_change":"2024-09-13T15:08:31.191585+0000","last_active":"2024-09-13T15:09:24.725822+0000","last_peered":"2024-09-13T15:09:24.725822+0000","last_clean":"2024-09-13T15:09:24.725822+0000","last_became_active":"2024-09-13T15:08:31.184700+0000","last_became_peered":"2024-09-13T15:08:31.184700+0000","last_unstale":"2024-09-13T15:09:24.725822+0000","last_undegraded":"2024-09-13T15:09:24.725822+0000","last_fullsized":"2024-09-13T15:09:24.725822+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:14.376834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":309,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.210392+0000","last_change":"2024-09-13T15:08:46.080613+0000","last_active":"2024-09-13T15:09:24.210392+0000","last_peered":"2024-09-13T15:09:24.210392+0000","last_clean":"2024-09-13T15:09:24.210392+0000","last_became_active":"2024-09-13T15:08:46.080320+0000","last_became_peered":"2024-09-13T15:08:46.080320+0000","last_unstale":"2024-09-13T15:09:24.210392+0000","last_undegraded":"2024-09-13T15:09:24.210392+0000","last_fullsized":"2024-09-13T15:09:24.210392+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:09.205609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":745,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.338063+0000","last_change":"2024-09-13T15:08:31.196449+0000","last_active":"2024-09-13T15:09:34.338063+0000","last_peered":"2024-09-13T15:09:34.338063+0000","last_clean":"2024-09-13T15:09:34.338063+0000","last_became_active":"2024-09-13T15:08:31.196143+0000","last_became_peered":"2024-09-13T15:08:31.196143+0000","last_unstale":"2024-09-13T15:09:34.338063+0000","last_undegraded":"2024-09-13T15:09:34.338063+0000","last_fullsized":"2024-09-13T15:09:34.338063+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:53.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":749,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337797+0000","last_change":"2024-09-13T15:08:31.195499+0000","last_active":"2024-09-13T15:09:34.337797+0000","last_peered":"2024-09-13T15:09:34.337797+0000","last_clean":"2024-09-13T15:09:34.337797+0000","last_became_active":"2024-09-13T15:08:31.194928+0000","last_became_peered":"2024-09-13T15:08:31.194928+0000","last_unstale":"2024-09-13T15:09:34.337797+0000","last_undegraded":"2024-09-13T15:09:34.337797+0000","last_fullsized":"2024-09-13T15:09:34.337797+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.112396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.725483+0000","last_change":"2024-09-13T15:08:31.185658+0000","last_active":"2024-09-13T15:09:24.725483+0000","last_peered":"2024-09-13T15:09:24.725483+0000","last_clean":"2024-09-13T15:09:24.725483+0000","last_became_active":"2024-09-13T15:08:31.185202+0000","last_became_peered":"2024-09-13T15:08:31.185202+0000","last_unstale":"2024-09-13T15:09:24.725483+0000","last_undegraded":"2024-09-13T15:09:24.725483+0000","last_fullsized":"2024-09-13T15:09:24.725483+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:04.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":249,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337325+0000","last_change":"2024-09-13T15:08:31.199682+0000","last_active":"2024-09-13T15:09:34.337325+0000","last_peered":"2024-09-13T15:09:34.337325+0000","last_clean":"2024-09-13T15:09:34.337325+0000","last_became_active":"2024-09-13T15:08:31.199110+0000","last_became_peered":"2024-09-13T15:08:31.199110+0000","last_unstale":"2024-09-13T15:09:34.337325+0000","last_undegraded":"2024-09-13T15:09:34.337325+0000","last_fullsized":"2024-09-13T15:09:34.337325+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:30.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":355,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.369795+0000","last_change":"2024-09-13T15:04:05.021292+0000","last_active":"2024-09-13T15:09:34.369795+0000","last_peered":"2024-09-13T15:09:34.369795+0000","last_clean":"2024-09-13T15:09:34.369795+0000","last_became_active":"2024-09-13T15:04:05.011146+0000","last_became_peered":"2024-09-13T15:04:05.011146+0000","last_unstale":"2024-09-13T15:09:34.369795+0000","last_undegraded":"2024-09-13T15:09:34.369795+0000","last_fullsized":"2024-09-13T15:09:34.369795+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:00.137694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"453'2525","reported_seq":2539,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.962175+0000","last_change":"2024-09-13T15:09:16.042122+0000","last_active":"2024-09-13T15:09:43.962175+0000","last_peered":"2024-09-13T15:09:43.962175+0000","last_clean":"2024-09-13T15:09:43.962175+0000","last_became_active":"2024-09-13T15:09:16.041789+0000","last_became_peered":"2024-09-13T15:09:16.041789+0000","last_unstale":"2024-09-13T15:09:43.962175+0000","last_undegraded":"2024-09-13T15:09:43.962175+0000","last_fullsized":"2024-09-13T15:09:43.962175+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2525,"log_dups_size":0,"ondisk_log_size":2525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:50.955816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165478400,"num_objects":2525,"num_object_clones":0,"num_object_copies":7575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2525,"num_write_kb":161600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":31,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.726142+0000","last_change":"2024-09-13T15:08:31.351504+0000","last_active":"2024-09-13T15:09:24.726142+0000","last_peered":"2024-09-13T15:09:24.726142+0000","last_clean":"2024-09-13T15:09:24.726142+0000","last_became_active":"2024-09-13T15:08:31.320950+0000","last_became_peered":"2024-09-13T15:08:31.320950+0000","last_unstale":"2024-09-13T15:09:24.726142+0000","last_undegraded":"2024-09-13T15:09:24.726142+0000","last_fullsized":"2024-09-13T15:09:24.726142+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:38.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":355,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209878+0000","last_change":"2024-09-13T15:08:46.078296+0000","last_active":"2024-09-13T15:09:24.209878+0000","last_peered":"2024-09-13T15:09:24.209878+0000","last_clean":"2024-09-13T15:09:24.209878+0000","last_became_active":"2024-09-13T15:08:46.078037+0000","last_became_peered":"2024-09-13T15:08:46.078037+0000","last_unstale":"2024-09-13T15:09:24.209878+0000","last_undegraded":"2024-09-13T15:09:24.209878+0000","last_fullsized":"2024-09-13T15:09:24.209878+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:49.959361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"453'2795","reported_seq":2816,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.158750+0000","last_change":"2024-09-13T15:09:16.070197+0000","last_active":"2024-09-13T15:09:46.158750+0000","last_peered":"2024-09-13T15:09:46.158750+0000","last_clean":"2024-09-13T15:09:46.158750+0000","last_became_active":"2024-09-13T15:09:16.069805+0000","last_became_peered":"2024-09-13T15:09:16.069805+0000","last_unstale":"2024-09-13T15:09:46.158750+0000","last_undegraded":"2024-09-13T15:09:46.158750+0000","last_fullsized":"2024-09-13T15:09:46.158750+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2795,"log_dups_size":0,"ondisk_log_size":2795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:17.465317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183173120,"num_objects":2795,"num_object_clones":0,"num_object_copies":8385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2795,"num_write_kb":178880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":31,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.726036+0000","last_change":"2024-09-13T15:08:31.201939+0000","last_active":"2024-09-13T15:09:24.726036+0000","last_peered":"2024-09-13T15:09:24.726036+0000","last_clean":"2024-09-13T15:09:24.726036+0000","last_became_active":"2024-09-13T15:08:31.191089+0000","last_became_peered":"2024-09-13T15:08:31.191089+0000","last_unstale":"2024-09-13T15:09:24.726036+0000","last_undegraded":"2024-09-13T15:09:24.726036+0000","last_fullsized":"2024-09-13T15:09:24.726036+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:06.249945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":519,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:28.749063+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:09:28.749063+0000","last_peered":"2024-09-13T15:09:28.749063+0000","last_clean":"2024-09-13T15:09:28.749063+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:09:28.749063+0000","last_undegraded":"2024-09-13T15:09:28.749063+0000","last_fullsized":"2024-09-13T15:09:28.749063+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"453'2934","reported_seq":2954,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.759132+0000","last_change":"2024-09-13T15:09:16.042793+0000","last_active":"2024-09-13T15:09:46.759132+0000","last_peered":"2024-09-13T15:09:46.759132+0000","last_clean":"2024-09-13T15:09:46.759132+0000","last_became_active":"2024-09-13T15:09:16.042016+0000","last_became_peered":"2024-09-13T15:09:16.042016+0000","last_unstale":"2024-09-13T15:09:46.759132+0000","last_undegraded":"2024-09-13T15:09:46.759132+0000","last_fullsized":"2024-09-13T15:09:46.759132+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2934,"log_dups_size":0,"ondisk_log_size":2934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:40.268202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192282624,"num_objects":2934,"num_object_clones":0,"num_object_copies":8802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2934,"num_write_kb":187776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":31,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.726020+0000","last_change":"2024-09-13T15:08:31.199736+0000","last_active":"2024-09-13T15:09:24.726020+0000","last_peered":"2024-09-13T15:09:24.726020+0000","last_clean":"2024-09-13T15:09:24.726020+0000","last_became_active":"2024-09-13T15:08:31.186018+0000","last_became_peered":"2024-09-13T15:08:31.186018+0000","last_unstale":"2024-09-13T15:09:24.726020+0000","last_undegraded":"2024-09-13T15:09:24.726020+0000","last_fullsized":"2024-09-13T15:09:24.726020+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:26.855187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":485,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336977+0000","last_change":"2024-09-13T15:08:31.193478+0000","last_active":"2024-09-13T15:09:34.336977+0000","last_peered":"2024-09-13T15:09:34.336977+0000","last_clean":"2024-09-13T15:09:34.336977+0000","last_became_active":"2024-09-13T15:08:31.193038+0000","last_became_peered":"2024-09-13T15:08:31.193038+0000","last_unstale":"2024-09-13T15:09:34.336977+0000","last_undegraded":"2024-09-13T15:09:34.336977+0000","last_fullsized":"2024-09-13T15:09:34.336977+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:05.600387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"453'2468","reported_seq":2489,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.696259+0000","last_change":"2024-09-13T15:09:16.043416+0000","last_active":"2024-09-13T15:09:43.696259+0000","last_peered":"2024-09-13T15:09:43.696259+0000","last_clean":"2024-09-13T15:09:43.696259+0000","last_became_active":"2024-09-13T15:09:16.043131+0000","last_became_peered":"2024-09-13T15:09:16.043131+0000","last_unstale":"2024-09-13T15:09:43.696259+0000","last_undegraded":"2024-09-13T15:09:43.696259+0000","last_fullsized":"2024-09-13T15:09:43.696259+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:21.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":157952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":326,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.210072+0000","last_change":"2024-09-13T15:08:46.073352+0000","last_active":"2024-09-13T15:09:24.210072+0000","last_peered":"2024-09-13T15:09:24.210072+0000","last_clean":"2024-09-13T15:09:24.210072+0000","last_became_active":"2024-09-13T15:08:46.073048+0000","last_became_peered":"2024-09-13T15:08:46.073048+0000","last_unstale":"2024-09-13T15:09:24.210072+0000","last_undegraded":"2024-09-13T15:09:24.210072+0000","last_fullsized":"2024-09-13T15:09:24.210072+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:57.277685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":657,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337377+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:09:34.337377+0000","last_peered":"2024-09-13T15:09:34.337377+0000","last_clean":"2024-09-13T15:09:34.337377+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:09:34.337377+0000","last_undegraded":"2024-09-13T15:09:34.337377+0000","last_fullsized":"2024-09-13T15:09:34.337377+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"453'2729","reported_seq":2750,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.533891+0000","last_change":"2024-09-13T15:09:16.042156+0000","last_active":"2024-09-13T15:09:46.533891+0000","last_peered":"2024-09-13T15:09:46.533891+0000","last_clean":"2024-09-13T15:09:46.533891+0000","last_became_active":"2024-09-13T15:09:16.041712+0000","last_became_peered":"2024-09-13T15:09:16.041712+0000","last_unstale":"2024-09-13T15:09:46.533891+0000","last_undegraded":"2024-09-13T15:09:46.533891+0000","last_fullsized":"2024-09-13T15:09:46.533891+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:57.243483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178847744,"num_objects":2729,"num_object_clones":0,"num_object_copies":8187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2729,"num_write_kb":174656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":783,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336920+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:09:34.336920+0000","last_peered":"2024-09-13T15:09:34.336920+0000","last_clean":"2024-09-13T15:09:34.336920+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:09:34.336920+0000","last_undegraded":"2024-09-13T15:09:34.336920+0000","last_fullsized":"2024-09-13T15:09:34.336920+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":461,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209200+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:09:24.209200+0000","last_peered":"2024-09-13T15:09:24.209200+0000","last_clean":"2024-09-13T15:09:24.209200+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:09:24.209200+0000","last_undegraded":"2024-09-13T15:09:24.209200+0000","last_fullsized":"2024-09-13T15:09:24.209200+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"453'2485","reported_seq":2506,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.684137+0000","last_change":"2024-09-13T15:09:16.042855+0000","last_active":"2024-09-13T15:09:43.684137+0000","last_peered":"2024-09-13T15:09:43.684137+0000","last_clean":"2024-09-13T15:09:43.684137+0000","last_became_active":"2024-09-13T15:09:16.042443+0000","last_became_peered":"2024-09-13T15:09:16.042443+0000","last_unstale":"2024-09-13T15:09:43.684137+0000","last_undegraded":"2024-09-13T15:09:43.684137+0000","last_fullsized":"2024-09-13T15:09:43.684137+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2485,"log_dups_size":0,"ondisk_log_size":2485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:28.644770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162856960,"num_objects":2485,"num_object_clones":0,"num_object_copies":7455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2485,"num_write_kb":159040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":259,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:28.749088+0000","last_change":"2024-09-13T15:04:05.013503+0000","last_active":"2024-09-13T15:09:28.749088+0000","last_peered":"2024-09-13T15:09:28.749088+0000","last_clean":"2024-09-13T15:09:28.749088+0000","last_became_active":"2024-09-13T15:04:05.013204+0000","last_became_peered":"2024-09-13T15:04:05.013204+0000","last_unstale":"2024-09-13T15:09:28.749088+0000","last_undegraded":"2024-09-13T15:09:28.749088+0000","last_fullsized":"2024-09-13T15:09:28.749088+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:17.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":455,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209466+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:09:24.209466+0000","last_peered":"2024-09-13T15:09:24.209466+0000","last_clean":"2024-09-13T15:09:24.209466+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:09:24.209466+0000","last_undegraded":"2024-09-13T15:09:24.209466+0000","last_fullsized":"2024-09-13T15:09:24.209466+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"453'2823","reported_seq":2844,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.527970+0000","last_change":"2024-09-13T15:09:16.070224+0000","last_active":"2024-09-13T15:09:46.527970+0000","last_peered":"2024-09-13T15:09:46.527970+0000","last_clean":"2024-09-13T15:09:46.527970+0000","last_became_active":"2024-09-13T15:09:16.069845+0000","last_became_peered":"2024-09-13T15:09:16.069845+0000","last_unstale":"2024-09-13T15:09:46.527970+0000","last_undegraded":"2024-09-13T15:09:46.527970+0000","last_fullsized":"2024-09-13T15:09:46.527970+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2823,"log_dups_size":0,"ondisk_log_size":2823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:44.584670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185008128,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":180672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":480,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:28.749073+0000","last_change":"2024-09-13T15:08:31.184973+0000","last_active":"2024-09-13T15:09:28.749073+0000","last_peered":"2024-09-13T15:09:28.749073+0000","last_clean":"2024-09-13T15:09:28.749073+0000","last_became_active":"2024-09-13T15:08:31.184600+0000","last_became_peered":"2024-09-13T15:08:31.184600+0000","last_unstale":"2024-09-13T15:09:28.749073+0000","last_undegraded":"2024-09-13T15:09:28.749073+0000","last_fullsized":"2024-09-13T15:09:28.749073+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:09.452895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":488,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209556+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:09:24.209556+0000","last_peered":"2024-09-13T15:09:24.209556+0000","last_clean":"2024-09-13T15:09:24.209556+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:09:24.209556+0000","last_undegraded":"2024-09-13T15:09:24.209556+0000","last_fullsized":"2024-09-13T15:09:24.209556+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"453'2454","reported_seq":2468,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.742433+0000","last_change":"2024-09-13T15:09:16.040707+0000","last_active":"2024-09-13T15:09:43.742433+0000","last_peered":"2024-09-13T15:09:43.742433+0000","last_clean":"2024-09-13T15:09:43.742433+0000","last_became_active":"2024-09-13T15:09:16.040411+0000","last_became_peered":"2024-09-13T15:09:16.040411+0000","last_unstale":"2024-09-13T15:09:43.742433+0000","last_undegraded":"2024-09-13T15:09:43.742433+0000","last_fullsized":"2024-09-13T15:09:43.742433+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:13.485945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160825344,"num_objects":2454,"num_object_clones":0,"num_object_copies":7362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":157056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":249,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.280286+0000","last_change":"2024-09-13T15:08:31.198602+0000","last_active":"2024-09-13T15:09:34.280286+0000","last_peered":"2024-09-13T15:09:34.280286+0000","last_clean":"2024-09-13T15:09:34.280286+0000","last_became_active":"2024-09-13T15:08:31.198289+0000","last_became_peered":"2024-09-13T15:08:31.198289+0000","last_unstale":"2024-09-13T15:09:34.280286+0000","last_undegraded":"2024-09-13T15:09:34.280286+0000","last_fullsized":"2024-09-13T15:09:34.280286+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:36.803833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":529,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.364966+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:09:34.364966+0000","last_peered":"2024-09-13T15:09:34.364966+0000","last_clean":"2024-09-13T15:09:34.364966+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:09:34.364966+0000","last_undegraded":"2024-09-13T15:09:34.364966+0000","last_fullsized":"2024-09-13T15:09:34.364966+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"453'2910","reported_seq":2930,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.559824+0000","last_change":"2024-09-13T15:09:16.069640+0000","last_active":"2024-09-13T15:09:46.559824+0000","last_peered":"2024-09-13T15:09:46.559824+0000","last_clean":"2024-09-13T15:09:46.559824+0000","last_became_active":"2024-09-13T15:09:16.069319+0000","last_became_peered":"2024-09-13T15:09:16.069319+0000","last_unstale":"2024-09-13T15:09:46.559824+0000","last_undegraded":"2024-09-13T15:09:46.559824+0000","last_fullsized":"2024-09-13T15:09:46.559824+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2910,"log_dups_size":0,"ondisk_log_size":2910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:52.471344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190709760,"num_objects":2910,"num_object_clones":0,"num_object_copies":8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":186240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":794,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.725312+0000","last_change":"2024-09-13T15:08:59.386963+0000","last_active":"2024-09-13T15:09:24.725312+0000","last_peered":"2024-09-13T15:09:24.725312+0000","last_clean":"2024-09-13T15:09:24.725312+0000","last_became_active":"2024-09-13T15:08:59.386027+0000","last_became_peered":"2024-09-13T15:08:59.386027+0000","last_unstale":"2024-09-13T15:09:24.725312+0000","last_undegraded":"2024-09-13T15:09:24.725312+0000","last_fullsized":"2024-09-13T15:09:24.725312+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T14:24:31.653483+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:26:59.002642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1122,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.340609+0000","last_change":"2024-09-13T15:08:57.728783+0000","last_active":"2024-09-13T15:09:34.340609+0000","last_peered":"2024-09-13T15:09:34.340609+0000","last_clean":"2024-09-13T15:09:34.340609+0000","last_became_active":"2024-09-13T15:08:47.077688+0000","last_became_peered":"2024-09-13T15:08:47.077688+0000","last_unstale":"2024-09-13T15:09:34.340609+0000","last_undegraded":"2024-09-13T15:09:34.340609+0000","last_fullsized":"2024-09-13T15:09:34.340609+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:52.710051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":745,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336782+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:09:34.336782+0000","last_peered":"2024-09-13T15:09:34.336782+0000","last_clean":"2024-09-13T15:09:34.336782+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:09:34.336782+0000","last_undegraded":"2024-09-13T15:09:34.336782+0000","last_fullsized":"2024-09-13T15:09:34.336782+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":307,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209333+0000","last_change":"2024-09-13T15:08:31.187510+0000","last_active":"2024-09-13T15:09:24.209333+0000","last_peered":"2024-09-13T15:09:24.209333+0000","last_clean":"2024-09-13T15:09:24.209333+0000","last_became_active":"2024-09-13T15:08:31.187108+0000","last_became_peered":"2024-09-13T15:08:31.187108+0000","last_unstale":"2024-09-13T15:09:24.209333+0000","last_undegraded":"2024-09-13T15:09:24.209333+0000","last_fullsized":"2024-09-13T15:09:24.209333+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:25.096543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.8","version":"453'2849","reported_seq":2870,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.748043+0000","last_change":"2024-09-13T15:09:16.042809+0000","last_active":"2024-09-13T15:09:46.748043+0000","last_peered":"2024-09-13T15:09:46.748043+0000","last_clean":"2024-09-13T15:09:46.748043+0000","last_became_active":"2024-09-13T15:09:16.041824+0000","last_became_peered":"2024-09-13T15:09:16.041824+0000","last_unstale":"2024-09-13T15:09:46.748043+0000","last_undegraded":"2024-09-13T15:09:46.748043+0000","last_fullsized":"2024-09-13T15:09:46.748043+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2849,"log_dups_size":0,"ondisk_log_size":2849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:05.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712064,"num_objects":2849,"num_object_clones":0,"num_object_copies":8547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2849,"num_write_kb":182336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":45,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.725210+0000","last_change":"2024-09-13T15:08:31.218115+0000","last_active":"2024-09-13T15:09:24.725210+0000","last_peered":"2024-09-13T15:09:24.725210+0000","last_clean":"2024-09-13T15:09:24.725210+0000","last_became_active":"2024-09-13T15:08:31.196118+0000","last_became_peered":"2024-09-13T15:08:31.196118+0000","last_unstale":"2024-09-13T15:09:24.725210+0000","last_undegraded":"2024-09-13T15:09:24.725210+0000","last_fullsized":"2024-09-13T15:09:24.725210+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:08.721976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":731,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337244+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:09:34.337244+0000","last_peered":"2024-09-13T15:09:34.337244+0000","last_clean":"2024-09-13T15:09:34.337244+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:09:34.337244+0000","last_undegraded":"2024-09-13T15:09:34.337244+0000","last_fullsized":"2024-09-13T15:09:34.337244+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":726,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337459+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:09:34.337459+0000","last_peered":"2024-09-13T15:09:34.337459+0000","last_clean":"2024-09-13T15:09:34.337459+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:09:34.337459+0000","last_undegraded":"2024-09-13T15:09:34.337459+0000","last_fullsized":"2024-09-13T15:09:34.337459+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"453'2529","reported_seq":2543,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.961801+0000","last_change":"2024-09-13T15:09:16.040901+0000","last_active":"2024-09-13T15:09:43.961801+0000","last_peered":"2024-09-13T15:09:43.961801+0000","last_clean":"2024-09-13T15:09:43.961801+0000","last_became_active":"2024-09-13T15:09:16.040576+0000","last_became_peered":"2024-09-13T15:09:16.040576+0000","last_unstale":"2024-09-13T15:09:43.961801+0000","last_undegraded":"2024-09-13T15:09:43.961801+0000","last_fullsized":"2024-09-13T15:09:43.961801+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2529,"log_dups_size":0,"ondisk_log_size":2529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:43.937156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":7587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":161856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1166,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.364493+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:09:34.364493+0000","last_peered":"2024-09-13T15:09:34.364493+0000","last_clean":"2024-09-13T15:09:34.364493+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:09:34.364493+0000","last_undegraded":"2024-09-13T15:09:34.364493+0000","last_fullsized":"2024-09-13T15:09:34.364493+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":795,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336550+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:09:34.336550+0000","last_peered":"2024-09-13T15:09:34.336550+0000","last_clean":"2024-09-13T15:09:34.336550+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:09:34.336550+0000","last_undegraded":"2024-09-13T15:09:34.336550+0000","last_fullsized":"2024-09-13T15:09:34.336550+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":533,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336977+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:09:34.336977+0000","last_peered":"2024-09-13T15:09:34.336977+0000","last_clean":"2024-09-13T15:09:34.336977+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:09:34.336977+0000","last_undegraded":"2024-09-13T15:09:34.336977+0000","last_fullsized":"2024-09-13T15:09:34.336977+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"453'2883","reported_seq":2901,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.767153+0000","last_change":"2024-09-13T15:09:16.069575+0000","last_active":"2024-09-13T15:09:46.767153+0000","last_peered":"2024-09-13T15:09:46.767153+0000","last_clean":"2024-09-13T15:09:46.767153+0000","last_became_active":"2024-09-13T15:09:16.069234+0000","last_became_peered":"2024-09-13T15:09:16.069234+0000","last_unstale":"2024-09-13T15:09:46.767153+0000","last_undegraded":"2024-09-13T15:09:46.767153+0000","last_fullsized":"2024-09-13T15:09:46.767153+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2883,"log_dups_size":0,"ondisk_log_size":2883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:43.686408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188940288,"num_objects":2883,"num_object_clones":0,"num_object_copies":8649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":184512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":376,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.210228+0000","last_change":"2024-09-13T15:08:46.074249+0000","last_active":"2024-09-13T15:09:24.210228+0000","last_peered":"2024-09-13T15:09:24.210228+0000","last_clean":"2024-09-13T15:09:24.210228+0000","last_became_active":"2024-09-13T15:08:46.073842+0000","last_became_peered":"2024-09-13T15:08:46.073842+0000","last_unstale":"2024-09-13T15:09:24.210228+0000","last_undegraded":"2024-09-13T15:09:24.210228+0000","last_fullsized":"2024-09-13T15:09:24.210228+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:44.854067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":480,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209425+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:09:24.209425+0000","last_peered":"2024-09-13T15:09:24.209425+0000","last_clean":"2024-09-13T15:09:24.209425+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:09:24.209425+0000","last_undegraded":"2024-09-13T15:09:24.209425+0000","last_fullsized":"2024-09-13T15:09:24.209425+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":293,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209806+0000","last_change":"2024-09-13T15:08:46.075360+0000","last_active":"2024-09-13T15:09:24.209806+0000","last_peered":"2024-09-13T15:09:24.209806+0000","last_clean":"2024-09-13T15:09:24.209806+0000","last_became_active":"2024-09-13T15:08:46.075046+0000","last_became_peered":"2024-09-13T15:08:46.075046+0000","last_unstale":"2024-09-13T15:09:24.209806+0000","last_undegraded":"2024-09-13T15:09:24.209806+0000","last_fullsized":"2024-09-13T15:09:24.209806+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.044150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"453'2486","reported_seq":2499,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.674156+0000","last_change":"2024-09-13T15:09:16.042846+0000","last_active":"2024-09-13T15:09:43.674156+0000","last_peered":"2024-09-13T15:09:43.674156+0000","last_clean":"2024-09-13T15:09:43.674156+0000","last_became_active":"2024-09-13T15:09:16.042571+0000","last_became_peered":"2024-09-13T15:09:16.042571+0000","last_unstale":"2024-09-13T15:09:43.674156+0000","last_undegraded":"2024-09-13T15:09:43.674156+0000","last_fullsized":"2024-09-13T15:09:43.674156+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2486,"log_dups_size":0,"ondisk_log_size":2486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:16.701976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162922496,"num_objects":2486,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":326,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.210336+0000","last_change":"2024-09-13T15:08:46.079918+0000","last_active":"2024-09-13T15:09:24.210336+0000","last_peered":"2024-09-13T15:09:24.210336+0000","last_clean":"2024-09-13T15:09:24.210336+0000","last_became_active":"2024-09-13T15:08:46.075051+0000","last_became_peered":"2024-09-13T15:08:46.075051+0000","last_unstale":"2024-09-13T15:09:24.210336+0000","last_undegraded":"2024-09-13T15:09:24.210336+0000","last_fullsized":"2024-09-13T15:09:24.210336+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:54.485416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1447,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336298+0000","last_change":"2024-09-13T15:08:59.341688+0000","last_active":"2024-09-13T15:09:34.336298+0000","last_peered":"2024-09-13T15:09:34.336298+0000","last_clean":"2024-09-13T15:09:34.336298+0000","last_became_active":"2024-09-13T15:08:59.340925+0000","last_became_peered":"2024-09-13T15:08:59.340925+0000","last_unstale":"2024-09-13T15:09:34.336298+0000","last_undegraded":"2024-09-13T15:09:34.336298+0000","last_fullsized":"2024-09-13T15:09:34.336298+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:22.966119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":376,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337949+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:09:34.337949+0000","last_peered":"2024-09-13T15:09:34.337949+0000","last_clean":"2024-09-13T15:09:34.337949+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:09:34.337949+0000","last_undegraded":"2024-09-13T15:09:34.337949+0000","last_fullsized":"2024-09-13T15:09:34.337949+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"453'2496","reported_seq":2510,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.751237+0000","last_change":"2024-09-13T15:09:16.069726+0000","last_active":"2024-09-13T15:09:43.751237+0000","last_peered":"2024-09-13T15:09:43.751237+0000","last_clean":"2024-09-13T15:09:43.751237+0000","last_became_active":"2024-09-13T15:09:16.069375+0000","last_became_peered":"2024-09-13T15:09:16.069375+0000","last_unstale":"2024-09-13T15:09:43.751237+0000","last_undegraded":"2024-09-13T15:09:43.751237+0000","last_fullsized":"2024-09-13T15:09:43.751237+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2496,"log_dups_size":0,"ondisk_log_size":2496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:29.701410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":2496,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":159744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":787,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336205+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:09:34.336205+0000","last_peered":"2024-09-13T15:09:34.336205+0000","last_clean":"2024-09-13T15:09:34.336205+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:09:34.336205+0000","last_undegraded":"2024-09-13T15:09:34.336205+0000","last_fullsized":"2024-09-13T15:09:34.336205+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":45,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.725882+0000","last_change":"2024-09-13T15:08:31.209459+0000","last_active":"2024-09-13T15:09:24.725882+0000","last_peered":"2024-09-13T15:09:24.725882+0000","last_clean":"2024-09-13T15:09:24.725882+0000","last_became_active":"2024-09-13T15:08:31.188811+0000","last_became_peered":"2024-09-13T15:08:31.188811+0000","last_unstale":"2024-09-13T15:09:24.725882+0000","last_undegraded":"2024-09-13T15:09:24.725882+0000","last_fullsized":"2024-09-13T15:09:24.725882+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.667223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":735,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337517+0000","last_change":"2024-09-13T15:08:31.197894+0000","last_active":"2024-09-13T15:09:34.337517+0000","last_peered":"2024-09-13T15:09:34.337517+0000","last_clean":"2024-09-13T15:09:34.337517+0000","last_became_active":"2024-09-13T15:08:31.197516+0000","last_became_peered":"2024-09-13T15:08:31.197516+0000","last_unstale":"2024-09-13T15:09:34.337517+0000","last_undegraded":"2024-09-13T15:09:34.337517+0000","last_fullsized":"2024-09-13T15:09:34.337517+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:42.650595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"453'2524","reported_seq":2538,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:43.686379+0000","last_change":"2024-09-13T15:09:16.069586+0000","last_active":"2024-09-13T15:09:43.686379+0000","last_peered":"2024-09-13T15:09:43.686379+0000","last_clean":"2024-09-13T15:09:43.686379+0000","last_became_active":"2024-09-13T15:09:16.069287+0000","last_became_peered":"2024-09-13T15:09:16.069287+0000","last_unstale":"2024-09-13T15:09:43.686379+0000","last_undegraded":"2024-09-13T15:09:43.686379+0000","last_fullsized":"2024-09-13T15:09:43.686379+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2524,"log_dups_size":0,"ondisk_log_size":2524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:30.843370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165412864,"num_objects":2524,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":161536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":838,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337535+0000","last_change":"2024-09-13T15:08:31.202795+0000","last_active":"2024-09-13T15:09:34.337535+0000","last_peered":"2024-09-13T15:09:34.337535+0000","last_clean":"2024-09-13T15:09:34.337535+0000","last_became_active":"2024-09-13T15:08:31.199963+0000","last_became_peered":"2024-09-13T15:08:31.199963+0000","last_unstale":"2024-09-13T15:09:34.337535+0000","last_undegraded":"2024-09-13T15:09:34.337535+0000","last_fullsized":"2024-09-13T15:09:34.337535+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:08.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1273,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337562+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:09:34.337562+0000","last_peered":"2024-09-13T15:09:34.337562+0000","last_clean":"2024-09-13T15:09:34.337562+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:09:34.337562+0000","last_undegraded":"2024-09-13T15:09:34.337562+0000","last_fullsized":"2024-09-13T15:09:34.337562+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":543,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.280019+0000","last_change":"2024-09-13T15:08:31.198729+0000","last_active":"2024-09-13T15:09:34.280019+0000","last_peered":"2024-09-13T15:09:34.280019+0000","last_clean":"2024-09-13T15:09:34.280019+0000","last_became_active":"2024-09-13T15:08:31.198423+0000","last_became_peered":"2024-09-13T15:08:31.198423+0000","last_unstale":"2024-09-13T15:09:34.280019+0000","last_undegraded":"2024-09-13T15:09:34.280019+0000","last_fullsized":"2024-09-13T15:09:34.280019+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:58.362919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"453'2741","reported_seq":2762,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:46.154190+0000","last_change":"2024-09-13T15:09:16.070043+0000","last_active":"2024-09-13T15:09:46.154190+0000","last_peered":"2024-09-13T15:09:46.154190+0000","last_clean":"2024-09-13T15:09:46.154190+0000","last_became_active":"2024-09-13T15:09:16.069704+0000","last_became_peered":"2024-09-13T15:09:16.069704+0000","last_unstale":"2024-09-13T15:09:46.154190+0000","last_undegraded":"2024-09-13T15:09:46.154190+0000","last_fullsized":"2024-09-13T15:09:46.154190+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":2741,"log_dups_size":0,"ondisk_log_size":2741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:30.439736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179634176,"num_objects":2741,"num_object_clones":0,"num_object_copies":8223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2741,"num_write_kb":175424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":662,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337895+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:09:34.337895+0000","last_peered":"2024-09-13T15:09:34.337895+0000","last_clean":"2024-09-13T15:09:34.337895+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:09:34.337895+0000","last_undegraded":"2024-09-13T15:09:34.337895+0000","last_fullsized":"2024-09-13T15:09:34.337895+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":884,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.726129+0000","last_change":"2024-09-13T15:08:59.333844+0000","last_active":"2024-09-13T15:09:24.726129+0000","last_peered":"2024-09-13T15:09:24.726129+0000","last_clean":"2024-09-13T15:09:24.726129+0000","last_became_active":"2024-09-13T15:08:59.332369+0000","last_became_peered":"2024-09-13T15:08:59.332369+0000","last_unstale":"2024-09-13T15:09:24.726129+0000","last_undegraded":"2024-09-13T15:09:24.726129+0000","last_fullsized":"2024-09-13T15:09:24.726129+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:36.490753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":476,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.208920+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:09:24.208920+0000","last_peered":"2024-09-13T15:09:24.208920+0000","last_clean":"2024-09-13T15:09:24.208920+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:09:24.208920+0000","last_undegraded":"2024-09-13T15:09:24.208920+0000","last_fullsized":"2024-09-13T15:09:24.208920+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":460,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209353+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:09:24.209353+0000","last_peered":"2024-09-13T15:09:24.209353+0000","last_clean":"2024-09-13T15:09:24.209353+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:09:24.209353+0000","last_undegraded":"2024-09-13T15:09:24.209353+0000","last_fullsized":"2024-09-13T15:09:24.209353+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":592,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209318+0000","last_change":"2024-09-13T15:08:31.193477+0000","last_active":"2024-09-13T15:09:24.209318+0000","last_peered":"2024-09-13T15:09:24.209318+0000","last_clean":"2024-09-13T15:09:24.209318+0000","last_became_active":"2024-09-13T15:08:31.181861+0000","last_became_peered":"2024-09-13T15:08:31.181861+0000","last_unstale":"2024-09-13T15:09:24.209318+0000","last_undegraded":"2024-09-13T15:09:24.209318+0000","last_fullsized":"2024-09-13T15:09:24.209318+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:02.424164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":33292,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":798,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337146+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:09:34.337146+0000","last_peered":"2024-09-13T15:09:34.337146+0000","last_clean":"2024-09-13T15:09:34.337146+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:09:34.337146+0000","last_undegraded":"2024-09-13T15:09:34.337146+0000","last_fullsized":"2024-09-13T15:09:34.337146+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":33,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.725887+0000","last_change":"2024-09-13T15:08:31.204593+0000","last_active":"2024-09-13T15:09:24.725887+0000","last_peered":"2024-09-13T15:09:24.725887+0000","last_clean":"2024-09-13T15:09:24.725887+0000","last_became_active":"2024-09-13T15:08:31.198883+0000","last_became_peered":"2024-09-13T15:08:31.198883+0000","last_unstale":"2024-09-13T15:09:24.725887+0000","last_undegraded":"2024-09-13T15:09:24.725887+0000","last_fullsized":"2024-09-13T15:09:24.725887+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:08.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":349,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.210269+0000","last_change":"2024-09-13T15:08:31.188790+0000","last_active":"2024-09-13T15:09:24.210269+0000","last_peered":"2024-09-13T15:09:24.210269+0000","last_clean":"2024-09-13T15:09:24.210269+0000","last_became_active":"2024-09-13T15:08:31.188441+0000","last_became_peered":"2024-09-13T15:08:31.188441+0000","last_unstale":"2024-09-13T15:09:24.210269+0000","last_undegraded":"2024-09-13T15:09:24.210269+0000","last_fullsized":"2024-09-13T15:09:24.210269+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:52.934163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":786,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.280121+0000","last_change":"2024-09-13T15:08:31.197112+0000","last_active":"2024-09-13T15:09:34.280121+0000","last_peered":"2024-09-13T15:09:34.280121+0000","last_clean":"2024-09-13T15:09:34.280121+0000","last_became_active":"2024-09-13T15:08:31.196687+0000","last_became_peered":"2024-09-13T15:08:31.196687+0000","last_unstale":"2024-09-13T15:09:34.280121+0000","last_undegraded":"2024-09-13T15:09:34.280121+0000","last_fullsized":"2024-09-13T15:09:34.280121+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:47.532912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":754,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337765+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:09:34.337765+0000","last_peered":"2024-09-13T15:09:34.337765+0000","last_clean":"2024-09-13T15:09:34.337765+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:09:34.337765+0000","last_undegraded":"2024-09-13T15:09:34.337765+0000","last_fullsized":"2024-09-13T15:09:34.337765+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":816,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.280140+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:09:34.280140+0000","last_peered":"2024-09-13T15:09:34.280140+0000","last_clean":"2024-09-13T15:09:34.280140+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:09:34.280140+0000","last_undegraded":"2024-09-13T15:09:34.280140+0000","last_fullsized":"2024-09-13T15:09:34.280140+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":255,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.210454+0000","last_change":"2024-09-13T15:08:46.075112+0000","last_active":"2024-09-13T15:09:24.210454+0000","last_peered":"2024-09-13T15:09:24.210454+0000","last_clean":"2024-09-13T15:09:24.210454+0000","last_became_active":"2024-09-13T15:08:46.074818+0000","last_became_peered":"2024-09-13T15:08:46.074818+0000","last_unstale":"2024-09-13T15:09:24.210454+0000","last_undegraded":"2024-09-13T15:09:24.210454+0000","last_fullsized":"2024-09-13T15:09:24.210454+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:56.288263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":779,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.280181+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:09:34.280181+0000","last_peered":"2024-09-13T15:09:34.280181+0000","last_clean":"2024-09-13T15:09:34.280181+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:09:34.280181+0000","last_undegraded":"2024-09-13T15:09:34.280181+0000","last_fullsized":"2024-09-13T15:09:34.280181+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":751,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337721+0000","last_change":"2024-09-13T15:08:31.195215+0000","last_active":"2024-09-13T15:09:34.337721+0000","last_peered":"2024-09-13T15:09:34.337721+0000","last_clean":"2024-09-13T15:09:34.337721+0000","last_became_active":"2024-09-13T15:08:31.194526+0000","last_became_peered":"2024-09-13T15:08:31.194526+0000","last_unstale":"2024-09-13T15:09:34.337721+0000","last_undegraded":"2024-09-13T15:09:34.337721+0000","last_fullsized":"2024-09-13T15:09:34.337721+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:56.358764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":753,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.335942+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:09:34.335942+0000","last_peered":"2024-09-13T15:09:34.335942+0000","last_clean":"2024-09-13T15:09:34.335942+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:09:34.335942+0000","last_undegraded":"2024-09-13T15:09:34.335942+0000","last_fullsized":"2024-09-13T15:09:34.335942+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":617,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336676+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:09:34.336676+0000","last_peered":"2024-09-13T15:09:34.336676+0000","last_clean":"2024-09-13T15:09:34.336676+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:09:34.336676+0000","last_undegraded":"2024-09-13T15:09:34.336676+0000","last_fullsized":"2024-09-13T15:09:34.336676+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":455,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.208974+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:09:24.208974+0000","last_peered":"2024-09-13T15:09:24.208974+0000","last_clean":"2024-09-13T15:09:24.208974+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:09:24.208974+0000","last_undegraded":"2024-09-13T15:09:24.208974+0000","last_fullsized":"2024-09-13T15:09:24.208974+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":368,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.364382+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:09:34.364382+0000","last_peered":"2024-09-13T15:09:34.364382+0000","last_clean":"2024-09-13T15:09:34.364382+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:09:34.364382+0000","last_undegraded":"2024-09-13T15:09:34.364382+0000","last_fullsized":"2024-09-13T15:09:34.364382+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":739,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337657+0000","last_change":"2024-09-13T15:08:31.194912+0000","last_active":"2024-09-13T15:09:34.337657+0000","last_peered":"2024-09-13T15:09:34.337657+0000","last_clean":"2024-09-13T15:09:34.337657+0000","last_became_active":"2024-09-13T15:08:31.194553+0000","last_became_peered":"2024-09-13T15:08:31.194553+0000","last_unstale":"2024-09-13T15:09:34.337657+0000","last_undegraded":"2024-09-13T15:09:34.337657+0000","last_fullsized":"2024-09-13T15:09:34.337657+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:17.457690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":465,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209283+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:09:24.209283+0000","last_peered":"2024-09-13T15:09:24.209283+0000","last_clean":"2024-09-13T15:09:24.209283+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:09:24.209283+0000","last_undegraded":"2024-09-13T15:09:24.209283+0000","last_fullsized":"2024-09-13T15:09:24.209283+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":293,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.209992+0000","last_change":"2024-09-13T15:08:46.073232+0000","last_active":"2024-09-13T15:09:24.209992+0000","last_peered":"2024-09-13T15:09:24.209992+0000","last_clean":"2024-09-13T15:09:24.209992+0000","last_became_active":"2024-09-13T15:08:46.072807+0000","last_became_peered":"2024-09-13T15:08:46.072807+0000","last_unstale":"2024-09-13T15:09:24.209992+0000","last_undegraded":"2024-09-13T15:09:24.209992+0000","last_fullsized":"2024-09-13T15:09:24.209992+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:11.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.208891+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:09:24.208891+0000","last_peered":"2024-09-13T15:09:24.208891+0000","last_clean":"2024-09-13T15:09:24.208891+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:09:24.208891+0000","last_undegraded":"2024-09-13T15:09:24.208891+0000","last_fullsized":"2024-09-13T15:09:24.208891+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":321,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.210396+0000","last_change":"2024-09-13T15:08:46.079600+0000","last_active":"2024-09-13T15:09:24.210396+0000","last_peered":"2024-09-13T15:09:24.210396+0000","last_clean":"2024-09-13T15:09:24.210396+0000","last_became_active":"2024-09-13T15:08:46.075458+0000","last_became_peered":"2024-09-13T15:08:46.075458+0000","last_unstale":"2024-09-13T15:09:24.210396+0000","last_undegraded":"2024-09-13T15:09:24.210396+0000","last_fullsized":"2024-09-13T15:09:24.210396+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:41.707452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":789,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.280329+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:09:34.280329+0000","last_peered":"2024-09-13T15:09:34.280329+0000","last_clean":"2024-09-13T15:09:34.280329+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:09:34.280329+0000","last_undegraded":"2024-09-13T15:09:34.280329+0000","last_fullsized":"2024-09-13T15:09:34.280329+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":812,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.336775+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:09:34.336775+0000","last_peered":"2024-09-13T15:09:34.336775+0000","last_clean":"2024-09-13T15:09:34.336775+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:09:34.336775+0000","last_undegraded":"2024-09-13T15:09:34.336775+0000","last_fullsized":"2024-09-13T15:09:34.336775+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":487,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-09-13T15:09:24.208780+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:09:24.208780+0000","last_peered":"2024-09-13T15:09:24.208780+0000","last_clean":"2024-09-13T15:09:24.208780+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:09:24.208780+0000","last_undegraded":"2024-09-13T15:09:24.208780+0000","last_fullsized":"2024-09-13T15:09:24.208780+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":778,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-13T15:09:34.337546+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:09:34.337546+0000","last_peered":"2024-09-13T15:09:34.337546+0000","last_clean":"2024-09-13T15:09:34.337546+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:09:34.337546+0000","last_undegraded":"2024-09-13T15:09:34.337546+0000","last_fullsized":"2024-09-13T15:09:34.337546+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2793865216,"num_objects":42631,"num_object_clones":0,"num_object_copies":127893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42631,"num_write_kb":2728384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8265465856,"data_stored":8265465856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42631,"ondisk_log_size":42631,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":36578,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234508,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1082556,"kb_used_data":628148,"kb_used_omap":1,"kb_used_meta":454398,"kb_avail":92662596,"statfs":{"total":95995035648,"available":94886498304,"internally_reserved":0,"allocated":643223552,"data_stored":640716951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":465304011},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496144,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1675632,"kb_used_data":1090792,"kb_used_omap":1,"kb_used_meta":584830,"kb_avail":92069520,"statfs":{"total":95995035648,"available":94279188480,"internally_reserved":0,"allocated":1116971008,"data_stored":1114476295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":598866375},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757779,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1539624,"kb_used_data":786592,"kb_used_omap":1,"kb_used_meta":753022,"kb_avail":92205528,"statfs":{"total":95995035648,"available":94418460672,"internally_reserved":0,"allocated":805470208,"data_stored":802979300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":771094983},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019419,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1960948,"kb_used_data":1280108,"kb_used_omap":1,"kb_used_meta":680830,"kb_avail":91784204,"statfs":{"total":95995035648,"available":93987024896,"internally_reserved":0,"allocated":1310830592,"data_stored":1308331878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697170374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468304,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1773476,"kb_used_data":897756,"kb_used_omap":4,"kb_used_meta":875707,"kb_avail":91971676,"statfs":{"total":95995035648,"available":94178996224,"internally_reserved":0,"allocated":919302144,"data_stored":916815036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":896724132},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273203,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1294572,"kb_used_data":949796,"kb_used_omap":12,"kb_used_meta":344755,"kb_avail":92450580,"statfs":{"total":95995035648,"available":94669393920,"internally_reserved":0,"allocated":972591104,"data_stored":970096352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":353030016},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795356,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367268,"kb_used_data":528988,"kb_used_omap":7,"kb_used_meta":838264,"kb_avail":92377884,"statfs":{"total":95995035648,"available":94594953216,"internally_reserved":0,"allocated":541683712,"data_stored":539196533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":858382962},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502138,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2166892,"kb_used_data":1764644,"kb_used_omap":10,"kb_used_meta":402229,"kb_avail":91578260,"statfs":{"total":95995035648,"available":93776138240,"internally_reserved":0,"allocated":1806995456,"data_stored":1804501214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":411883199},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1865547776,"data_stored":1865547776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":554696704,"data_stored":554696704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":980877312,"data_stored":980877312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":911015936,"data_stored":911015936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1313406976,"data_stored":1313406976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":831258624,"data_stored":831258624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1155530752,"data_stored":1155530752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":653131776,"data_stored":653131776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:09:52.321 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T15:09:52.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-13T15:09:52.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:52.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:52.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:52.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:52.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:52.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:52.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:52.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:52.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:52.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:52.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:52.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:52.944 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 26 16 55465 55449 133.26 100 0.00219902 0.00749422 2024-09-13T15:09:52.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:52.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:52.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:53.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:53.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:53.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:53.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:53.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:53.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:53.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:53.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:53.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:53.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:53.525 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:53.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:53.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:53.658 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:53 smithi154 ceph-mon[102314]: pgmap v1440: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T15:09:53.659 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:53 smithi154 ceph-mon[107038]: pgmap v1440: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T15:09:53.663 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:53.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:53 smithi172 ceph-mon[100696]: pgmap v1440: 89 pgs: 89 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-13T15:09:53.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:53.796 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:53.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:53.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:53.935 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:53.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:53.944 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 27 16 57040 57024 131.97 98.4375 0.00807011 0.00757277 2024-09-13T15:09:54.020 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:54.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:54.091 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:54.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:54.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:54.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:54.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:54.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:54.368 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:54.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:54.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:54.535 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:54.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:54.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:54.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:54.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:54.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:54.813 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:54.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:54.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:54.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 28 16 58610 58594 130.759 98.125 0.00719716 0.00764319 2024-09-13T15:09:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:54.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:54.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:55.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:55.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:55.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:55.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:55.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:55.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:55.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:55.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:55.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:55.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:55.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:55.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:55.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:55.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:55.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:55.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:55.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:09:55.764 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:55.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:09:55.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:55.902 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:55.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:55.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:55.945 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 29 16 60177 60161 129.627 97.9375 0.0162269 0.00770932 2024-09-13T15:09:56.004 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:56.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:56.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:56.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:56.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:56.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:56.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:56.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:56.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:56.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:56.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:56.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:56.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:56 smithi154 ceph-mon[102314]: pgmap v1441: 89 pgs: 89 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-13T15:09:56.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:56 smithi154 ceph-mon[107038]: pgmap v1441: 89 pgs: 89 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-13T15:09:56.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:56 smithi172 ceph-mon[100696]: pgmap v1441: 89 pgs: 89 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-13T15:09:56.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:56.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:56.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:56.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:56.736 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:56.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:56.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:56.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:56.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:56.945 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 30 16 61682 61666 128.441 94.0625 0.00685345 0.00778061 2024-09-13T15:09:56.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:57.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:57.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:57.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:57.124 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:09:57.149 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:57.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:57.286 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:57.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:57.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:57.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3755927874' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:57.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:57 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:57.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3755927874' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:57.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:57 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3755927874' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:57 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:09:57.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:57.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:57.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:57.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:57.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:57.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:57.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:57.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 31 16 63212 63196 127.381 95.625 0.0138857 0.00784527 2024-09-13T15:09:57.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:57.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:58.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:58.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:58.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[102314]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[102314]: pgmap v1443: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:58.219 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[107038]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T15:09:58.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[107038]: pgmap v1443: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-13T15:09:58.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:58.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:58.251 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:58.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:58.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:58.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:58.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:58.397 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:09:58.397 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:09:58.431 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:09:58.432 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:09:58.465 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:09:58.465 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:09:58.472 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:09:58.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:58 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:09:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:09:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:58 smithi172 ceph-mon[100696]: osdmap e454: 8 total, 8 up, 8 in 2024-09-13T15:09:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:58 smithi172 ceph-mon[100696]: pgmap v1443: 89 pgs: 89 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-13T15:09:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:09:58.506 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:09:58.507 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:09:58.537 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:09:58.537 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:09:58.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:58.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:58.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:58.571 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:09:58.571 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:09:58.601 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:09:58.602 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:09:58.637 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:09:58.637 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:09:58.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:58.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:58.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:58.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:09:58.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:58.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:09:58.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:09:58.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 32 16 64635 64619 126.179 88.9375 0.0149093 0.00791966 2024-09-13T15:09:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:58.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:58.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:09:59.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:09:59.108 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:59.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:09:59.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:59.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:59.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:59.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:59.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:59.382 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:59.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:59.392 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:09:59 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2138964900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:59.392 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:09:59 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2138964900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:59.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:09:59.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:09:59 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2138964900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:09:59.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:59.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:09:59.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:09:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:59.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:59.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:09:59.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:09:59.794 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:09:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:59.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:09:59.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:09:59.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:09:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:09:59.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:09:59.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 33 16 66170 66154 125.263 95.9375 0.00613162 0.00797779 2024-09-13T15:09:59.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:10:00.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:00.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:00.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:00.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:00.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:00.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:00.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:00.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:00.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:00.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:00 smithi154 ceph-mon[102314]: pgmap v1444: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-13T15:10:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:00 smithi154 ceph-mon[102314]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-13T15:10:00.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:00 smithi154 ceph-mon[102314]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T15:10:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:00 smithi154 ceph-mon[107038]: pgmap v1444: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-13T15:10:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:00 smithi154 ceph-mon[107038]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-13T15:10:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:00 smithi154 ceph-mon[107038]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T15:10:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:00 smithi172 ceph-mon[100696]: pgmap v1444: 89 pgs: 89 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-13T15:10:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:00 smithi172 ceph-mon[100696]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-13T15:10:00.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:00 smithi172 ceph-mon[100696]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T15:10:00.495 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:00.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:00.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:00.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:00.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:00.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:00.756 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:00.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:00.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:00.897 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:00.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:00.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 34 16 67573 67557 124.158 87.6875 0.00377879 0.00804491 2024-09-13T15:10:00.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:01.030 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:01.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:01.166 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:01.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:01.298 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:01.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:01.436 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:01 smithi154 ceph-mon[102314]: pgmap v1445: 89 pgs: 89 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T15:10:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:01 smithi154 ceph-mon[107038]: pgmap v1445: 89 pgs: 89 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T15:10:01.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:01.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:01.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:01.699 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:01 smithi172 ceph-mon[100696]: pgmap v1445: 89 pgs: 89 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-13T15:10:01.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:01.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:01.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:01.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 35 16 68840 68824 122.873 79.1875 0.00266967 0.00812875 2024-09-13T15:10:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:01.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:01.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:02.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:02.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:02.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:02.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:02.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:02.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:10:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:10:01] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T15:10:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:02.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:02.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:02.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:02.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:02.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:02.486 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:02.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:02.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:02.611 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:02.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:02.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:02.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:02.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:02.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:02.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:02.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:02.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:02.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 36 16 69653 69637 120.871 50.8125 0.0179736 0.00826822 2024-09-13T15:10:02.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:03.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:03.129 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:03.158 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:03.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:03.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:03.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:03.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:03.433 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:03.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:03.575 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:03.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:03.716 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:03.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:03 smithi154 ceph-mon[102314]: pgmap v1446: 89 pgs: 89 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-13T15:10:03.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:03 smithi154 ceph-mon[107038]: pgmap v1446: 89 pgs: 89 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-13T15:10:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:03 smithi172 ceph-mon[100696]: pgmap v1446: 89 pgs: 89 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-13T15:10:03.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:03.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:03.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 37 16 71038 71022 119.944 86.5625 0.00897807 0.00833211 2024-09-13T15:10:03.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:03.990 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:03.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:04.092 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:04.123 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:04.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:04.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:04.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:04.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:04.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:04.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:04.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:04.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:04.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:04.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:04.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:04.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:04.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:04.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:04.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:04.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:04.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 38 16 72439 72423 119.091 87.5625 0.00878751 0.00839221 2024-09-13T15:10:04.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:04.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:04.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:05.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:05.137 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:05.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:05.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:05.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:05.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:05.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:05.380 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:05.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:05.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:05.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:05.538 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:05 smithi154 ceph-mon[102314]: pgmap v1447: 89 pgs: 89 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T15:10:05.539 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:05 smithi154 ceph-mon[107038]: pgmap v1447: 89 pgs: 89 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T15:10:05.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:05.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:05.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:05.690 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:05.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:05 smithi172 ceph-mon[100696]: pgmap v1447: 89 pgs: 89 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-13T15:10:05.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:05.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:05.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:05.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:05.946 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 39 16 73762 73746 118.158 82.6875 0.00481878 0.00845823 2024-09-13T15:10:05.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:05.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:06.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:06.077 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:06.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:06.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:06.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:06.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:06.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:06.451 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:06.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:06.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:06.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:06.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:06.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:06.858 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:06.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:10:06.946400+0000 min lat: 0.00103447 max lat: 0.561012 avg lat: 0.00860996 2024-09-13T15:10:06.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:10:06.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 40 16 74315 74299 116.068 34.5625 0.00661119 0.00860996 2024-09-13T15:10:06.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:06.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:06.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:07.047 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:10:07.047 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:10:07.047 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:10:07.047 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:10:07.047 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:10:07.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:07.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:07.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:07.236 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:07.264 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:07.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:07.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:07.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:10:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:07.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:07.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:07.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:07.533 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:07.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:07.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:07.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:07.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:07.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:07 smithi154 ceph-mon[107038]: pgmap v1448: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T15:10:07.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:07 smithi154 ceph-mon[102314]: pgmap v1448: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T15:10:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:07 smithi172 ceph-mon[100696]: pgmap v1448: 89 pgs: 89 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-13T15:10:07.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:07.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:07.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:07.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:07.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:07.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:07.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 41 16 75446 75430 114.961 70.6875 0.0034068 0.00869105 2024-09-13T15:10:08.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:08.072 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:08.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:08.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:08.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:08.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:08.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:08.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.465 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:08.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:08.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.604 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:08.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:08.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2624580025' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T15:10:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:08 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T15:10:08.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2624580025' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T15:10:08.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:08 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T15:10:08.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:08.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2624580025' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T15:10:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:08 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-13T15:10:08.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:08.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:08.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 42 16 76414 76398 113.665 60.5 0.00892748 0.00879311 2024-09-13T15:10:08.959 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:08.981 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:08.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:09.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:09.104 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:09.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:09.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:09.246 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:09.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:09.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:09.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:09.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:09.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:09.527 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:09.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:09.595 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T15:10:09.595 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[102314]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T15:10:09.595 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[102314]: pgmap v1450: 89 pgs: 89 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T15:10:09.595 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T15:10:09.596 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T15:10:09.596 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[107038]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T15:10:09.596 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[107038]: pgmap v1450: 89 pgs: 89 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T15:10:09.596 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T15:10:09.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:09.659 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:09.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:09 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-13T15:10:09.760 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:09 smithi172 ceph-mon[100696]: osdmap e455: 8 total, 8 up, 8 in 2024-09-13T15:10:09.760 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:09 smithi172 ceph-mon[100696]: pgmap v1450: 89 pgs: 89 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-13T15:10:09.760 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-13T15:10:09.760 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 1 pg_num to 11 2024-09-13T15:10:09.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:09.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:09.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:09.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:09.947 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 43 16 77512 77496 112.617 68.625 0.00766673 0.00887488 2024-09-13T15:10:09.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:09.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:10.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:10.090 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:10.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:10.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:10.230 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:10.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:10.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:10.368 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:10.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:10.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:10.505 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:10.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:10.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:10 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T15:10:10.585 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:10 smithi154 ceph-mon[102314]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T15:10:10.585 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:10 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T15:10:10.585 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:10 smithi154 ceph-mon[107038]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T15:10:10.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:10.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:10.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:10 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-13T15:10:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:10 smithi172 ceph-mon[100696]: osdmap e456: 8 total, 8 up, 8 in 2024-09-13T15:10:10.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:10.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:10.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:10.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:10.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:10.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:10.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:10:10.948 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 44 16 78842 78826 111.945 83.125 0.00572245 0.00892764 2024-09-13T15:10:11.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:11.083 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:11.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:11.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:11.213 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:11.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:11.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:11.339 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:11.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:11.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:11.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:11.494 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:11.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:11.536 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:11 smithi154 ceph-mon[102314]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T15:10:11.536 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:11 smithi154 ceph-mon[102314]: pgmap v1453: 99 pgs: 10 unknown, 89 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T15:10:11.536 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:11 smithi154 ceph-mon[107038]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T15:10:11.537 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:11 smithi154 ceph-mon[107038]: pgmap v1453: 99 pgs: 10 unknown, 89 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T15:10:11.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:11.624 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:11.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:11.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:11 smithi172 ceph-mon[100696]: osdmap e457: 8 total, 8 up, 8 in 2024-09-13T15:10:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:11 smithi172 ceph-mon[100696]: pgmap v1453: 99 pgs: 10 unknown, 89 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-13T15:10:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:11.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:11.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:11.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:11.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:11.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:11.948 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 45 16 79955 79939 111.003 69.5625 0.00325476 0.00900002 2024-09-13T15:10:12.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:12.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:12.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:12.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:12.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:12.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:12.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:10:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:10:11] "GET /metrics HTTP/1.1" 200 36659 "" "Prometheus/2.51.0" 2024-09-13T15:10:12.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:12.329 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:12.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:12.385 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:10:12.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:12.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:12.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:12.493 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/613339103' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:10:12.493 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:12 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:10:12.493 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/613339103' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:10:12.494 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:12 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:10:12.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:12.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:12.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:12.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:12.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:12.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:12.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/613339103' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:10:12.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:12 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:10:12.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:12.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:12.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:12.949 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 46 16 81228 81212 110.319 79.5625 0.00824938 0.00905948 2024-09-13T15:10:12.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:13.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:13.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.155 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:13.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:13.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:13.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:13.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:13.424 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:13.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:13.566 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:13.566 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:10:13.566 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[102314]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T15:10:13.566 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:13.566 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[102314]: pgmap v1455: 99 pgs: 10 unknown, 89 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T15:10:13.566 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:13.566 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:10:13.567 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[107038]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T15:10:13.567 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:13.567 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:13 smithi154 ceph-mon[107038]: pgmap v1455: 99 pgs: 10 unknown, 89 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T15:10:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:13.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:13.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.700 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:13.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:13 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:13 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:10:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:13 smithi172 ceph-mon[100696]: osdmap e458: 8 total, 8 up, 8 in 2024-09-13T15:10:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:13 smithi172 ceph-mon[100696]: pgmap v1455: 99 pgs: 10 unknown, 89 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-13T15:10:13.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:13.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:13.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:13.948 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 47 16 82521 82505 109.691 80.8125 0.00753898 0.00911131 2024-09-13T15:10:13.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:13.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:14.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:14.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:14.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:14.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:14.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:14.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:14.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:14.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:14.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:14.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:14.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:14.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:14.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:14.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:14.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:14.702 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:10:14.703 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:10:14.737 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:10:14.737 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:10:14.752 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:14.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:14.774 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:10:14.774 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:10:14.774 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2521936503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:14.774 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2521936503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:14.809 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:10:14.809 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:10:14.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:14.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:14.844 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:10:14.844 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:10:14.879 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:10:14.880 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:10:14.914 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:10:14.914 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:10:14.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:14.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 48 16 83749 83733 109.001 76.75 0.0102738 0.00916878 2024-09-13T15:10:14.952 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:10:14.952 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:10:14.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:14.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2521936503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:15.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:15.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:15.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:15.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:15.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:15.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:15.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:15.390 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:15.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:15.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:15.492 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:10:15.492 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:10:15.492 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T15:10:15.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-13T15:10:15.525 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:15.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:15.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:15.671 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:15.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:15.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:15.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:15.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:15.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:15.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 49 16 84943 84927 108.3 74.625 0.0144999 0.00922811 2024-09-13T15:10:15.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:15.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:15.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:15 smithi172 ceph-mon[100696]: pgmap v1456: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T15:10:15.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:15 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T15:10:16.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:15 smithi154 ceph-mon[102314]: pgmap v1456: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T15:10:16.000 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:15 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T15:10:16.000 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:15 smithi154 ceph-mon[107038]: pgmap v1456: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T15:10:16.000 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:15 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-13T15:10:16.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:16.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:10:16.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:16.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:16.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:16.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:16.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:16.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:16.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:16.396 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:16.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:16.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:16.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:16.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:16.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:16.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:16.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:16.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:16.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:16.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:16.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:16.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:16.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 50 16 86074 86058 107.547 70.6875 0.0115274 0.00929262 2024-09-13T15:10:16.951 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:16.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:16.965 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:16 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T15:10:16.966 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:16 smithi154 ceph-mon[102314]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T15:10:16.966 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:16 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T15:10:16.966 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:16 smithi154 ceph-mon[107038]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T15:10:17.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:17.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:17.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:17.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:17.233 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:17.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:17.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:16 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-13T15:10:17.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:16 smithi172 ceph-mon[100696]: osdmap e459: 8 total, 8 up, 8 in 2024-09-13T15:10:17.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:17.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:17.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:17.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:17.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:17.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:17.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:17.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:17.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:17.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:17.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:17.791 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:17.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:17.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:17.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:10:17.921 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:17.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:17.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 51 16 87138 87122 106.743 66.5 0.00696822 0.00936296 2024-09-13T15:10:18.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:18.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:18.160 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:18.189 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:18.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[102314]: 1.0 scrub starts 2024-09-13T15:10:18.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[102314]: 1.0 scrub ok 2024-09-13T15:10:18.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[102314]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T15:10:18.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[102314]: pgmap v1459: 99 pgs: 8 remapped+peering, 91 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T15:10:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[107038]: 1.0 scrub starts 2024-09-13T15:10:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[107038]: 1.0 scrub ok 2024-09-13T15:10:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[107038]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T15:10:18.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:17 smithi154 ceph-mon[107038]: pgmap v1459: 99 pgs: 8 remapped+peering, 91 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T15:10:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:17 smithi172 ceph-mon[100696]: 1.0 scrub starts 2024-09-13T15:10:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:17 smithi172 ceph-mon[100696]: 1.0 scrub ok 2024-09-13T15:10:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:17 smithi172 ceph-mon[100696]: osdmap e460: 8 total, 8 up, 8 in 2024-09-13T15:10:18.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:17 smithi172 ceph-mon[100696]: pgmap v1459: 99 pgs: 8 remapped+peering, 91 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-13T15:10:18.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:18.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:18.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:18.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:18.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:18.596 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:18.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:18.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:18.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:18.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:18.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 52 16 88267 88251 106.047 70.5625 0.027318 0.00942495 2024-09-13T15:10:18.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:18.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:18 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-09-13T15:10:18.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:18 smithi154 ceph-mon[102314]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T15:10:18.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:18 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-09-13T15:10:18.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:18 smithi154 ceph-mon[107038]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T15:10:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:18.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:18.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:19.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:19.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:19.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:19.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:18 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-09-13T15:10:19.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:18 smithi172 ceph-mon[100696]: osdmap e461: 8 total, 8 up, 8 in 2024-09-13T15:10:19.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:19.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:19.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:19.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:19.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:19.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:19.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:19.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:19.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:19.668 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:19.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:19.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:19.795 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:19.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:19.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:19.913 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:10:19.913 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:10:19.934 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:19.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:19.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:19.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 53 15 89184 89169 105.128 57.375 0.0118408 0.00950787 2024-09-13T15:10:20.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:20.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:20.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:20.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:20.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:20.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:20.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:19 smithi154 ceph-mon[102314]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T15:10:20.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:19 smithi154 ceph-mon[102314]: pgmap v1462: 99 pgs: 4 peering, 2 active+undersized+remapped, 4 remapped+peering, 89 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-13T15:10:20.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:19 smithi154 ceph-mon[107038]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T15:10:20.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:19 smithi154 ceph-mon[107038]: pgmap v1462: 99 pgs: 4 peering, 2 active+undersized+remapped, 4 remapped+peering, 89 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-13T15:10:20.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:19 smithi172 ceph-mon[100696]: osdmap e462: 8 total, 8 up, 8 in 2024-09-13T15:10:20.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:19 smithi172 ceph-mon[100696]: pgmap v1462: 99 pgs: 4 peering, 2 active+undersized+remapped, 4 remapped+peering, 89 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 93 MiB/s wr, 1.48k op/s 2024-09-13T15:10:20.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:20.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:20.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:20.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:20.467 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:20.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:20.569 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:20.596 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:20.600 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:20.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:20.605 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:20.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:20.731 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:20.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:20.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:20.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:20.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:20.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 54 16 90268 90252 104.435 67.6875 0.0158305 0.00956998 2024-09-13T15:10:20.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:21.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:21.044 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:20 smithi154 ceph-mon[102314]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T15:10:21.044 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:20 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:10:21.044 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:20 smithi154 ceph-mon[102314]: 1.6 scrub ok 2024-09-13T15:10:21.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:20 smithi154 ceph-mon[107038]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T15:10:21.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:20 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:10:21.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:20 smithi154 ceph-mon[107038]: 1.6 scrub ok 2024-09-13T15:10:21.121 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:21.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:20 smithi172 ceph-mon[100696]: osdmap e463: 8 total, 8 up, 8 in 2024-09-13T15:10:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:20 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:10:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:20 smithi172 ceph-mon[100696]: 1.6 scrub ok 2024-09-13T15:10:21.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:21.296 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:21.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:21.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:21.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:21.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:21.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:21.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:21.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:21.861 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:21.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[102314]: 1.8 scrub starts 2024-09-13T15:10:21.876 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[102314]: 1.8 scrub ok 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[102314]: 1.4 scrub starts 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[102314]: 1.4 scrub ok 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[102314]: 1.3 scrub starts 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[102314]: 1.3 scrub ok 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[102314]: pgmap v1464: 99 pgs: 4 peering, 2 active+undersized+remapped, 4 remapped+peering, 89 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 34 KiB/s rd, 57 MiB/s wr, 935 op/s; 223 KiB/s, 0 objects/s recovering 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[107038]: 1.8 scrub starts 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[107038]: 1.8 scrub ok 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[107038]: 1.4 scrub starts 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[107038]: 1.4 scrub ok 2024-09-13T15:10:21.877 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[107038]: 1.3 scrub starts 2024-09-13T15:10:21.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[107038]: 1.3 scrub ok 2024-09-13T15:10:21.878 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-mon[107038]: pgmap v1464: 99 pgs: 4 peering, 2 active+undersized+remapped, 4 remapped+peering, 89 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 34 KiB/s rd, 57 MiB/s wr, 935 op/s; 223 KiB/s, 0 objects/s recovering 2024-09-13T15:10:21.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 55 16 91442 91426 103.87 73.375 0.0152597 0.0096228 2024-09-13T15:10:21.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:21.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:21.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:22.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:22.133 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:22.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:22.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:10:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:10:21] "GET /metrics HTTP/1.1" 200 36653 "" "Prometheus/2.51.0" 2024-09-13T15:10:22.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:21 smithi172 ceph-mon[100696]: 1.8 scrub starts 2024-09-13T15:10:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:21 smithi172 ceph-mon[100696]: 1.8 scrub ok 2024-09-13T15:10:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:21 smithi172 ceph-mon[100696]: 1.4 scrub starts 2024-09-13T15:10:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:21 smithi172 ceph-mon[100696]: 1.4 scrub ok 2024-09-13T15:10:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:21 smithi172 ceph-mon[100696]: 1.3 scrub starts 2024-09-13T15:10:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:21 smithi172 ceph-mon[100696]: 1.3 scrub ok 2024-09-13T15:10:22.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:21 smithi172 ceph-mon[100696]: pgmap v1464: 99 pgs: 4 peering, 2 active+undersized+remapped, 4 remapped+peering, 89 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 34 KiB/s rd, 57 MiB/s wr, 935 op/s; 223 KiB/s, 0 objects/s recovering 2024-09-13T15:10:22.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:22.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:22.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:22.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:22.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:22.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:22.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:22.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:22.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:22.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:22.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:22.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:22.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:22.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:22.950 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 56 16 92645 92629 103.358 75.1875 0.00928214 0.00966943 2024-09-13T15:10:22.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:22.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:22.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:23.049 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:22 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:10:23.049 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:22 smithi154 ceph-mon[102314]: 1.7 scrub ok 2024-09-13T15:10:23.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:22 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:10:23.049 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:22 smithi154 ceph-mon[107038]: 1.7 scrub ok 2024-09-13T15:10:23.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:23.123 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:23.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:23.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:22 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:10:23.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:22 smithi172 ceph-mon[100696]: 1.7 scrub ok 2024-09-13T15:10:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:23.260 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:23.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:23.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:10:23.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:23.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:23.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:23.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:23.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:23.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:23.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:23.635 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:23.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:23.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:23.670 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:23.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:23.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:23.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:23.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:23.934 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:23 smithi154 ceph-mon[102314]: 1.a scrub starts 2024-09-13T15:10:23.934 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:23 smithi154 ceph-mon[102314]: 1.a scrub ok 2024-09-13T15:10:23.934 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:23 smithi154 ceph-mon[102314]: pgmap v1465: 99 pgs: 4 peering, 2 active+undersized+remapped, 93 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 58 MiB/s wr, 919 op/s 2024-09-13T15:10:23.935 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:23 smithi154 ceph-mon[107038]: 1.a scrub starts 2024-09-13T15:10:23.935 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:23 smithi154 ceph-mon[107038]: 1.a scrub ok 2024-09-13T15:10:23.935 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:23 smithi154 ceph-mon[107038]: pgmap v1465: 99 pgs: 4 peering, 2 active+undersized+remapped, 93 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 58 MiB/s wr, 919 op/s 2024-09-13T15:10:23.941 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:23.945 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:23.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:23.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 57 16 93770 93754 102.778 70.3125 0.0137289 0.00972436 2024-09-13T15:10:24.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:24.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:24.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:24.116 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:23 smithi172 ceph-mon[100696]: 1.a scrub starts 2024-09-13T15:10:24.116 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:23 smithi172 ceph-mon[100696]: 1.a scrub ok 2024-09-13T15:10:24.116 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:23 smithi172 ceph-mon[100696]: pgmap v1465: 99 pgs: 4 peering, 2 active+undersized+remapped, 93 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 58 MiB/s wr, 919 op/s 2024-09-13T15:10:24.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:24.218 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:24.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:24.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:24.351 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:24.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:24.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:24.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:24.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:24.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:24.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:24.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:24.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:24.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:24.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:24.843 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:10:24.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:24.907 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:24.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:24.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 58 16 94812 94796 102.128 65.125 0.0350226 0.009787 2024-09-13T15:10:25.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:25.047 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:25.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:25.144 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:24 smithi154 ceph-mon[102314]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:25.144 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:24 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2568036660' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:25.144 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:24 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:25.144 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:24 smithi154 ceph-mon[107038]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:25.145 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:24 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2568036660' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:25.145 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:24 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:25.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:25.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:25.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:25.200 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:24 smithi172 ceph-mon[100696]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:10:25.200 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:24 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2568036660' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:25.200 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:24 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:25.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:25.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:25.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:25.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:25.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:25.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:25.550 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:25.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:25.599 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:25.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:25.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:25.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:25.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:25.786 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:10:25.786 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:10:25.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:25.879 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:25.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:25.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 59 16 95945 95929 101.598 70.8125 0.0070323 0.00983715 2024-09-13T15:10:25.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:26.014 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:26.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:26.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:25 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:10:26.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:25 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:26.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:25 smithi172 ceph-mon[100696]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T15:10:26.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:25 smithi172 ceph-mon[100696]: 1.5 scrub starts 2024-09-13T15:10:26.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:25 smithi172 ceph-mon[100696]: 1.5 scrub ok 2024-09-13T15:10:26.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:25 smithi172 ceph-mon[100696]: pgmap v1467: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T15:10:26.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:26.149 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:26.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:26.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:10:26.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:26.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[102314]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T15:10:26.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[102314]: 1.5 scrub starts 2024-09-13T15:10:26.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[102314]: 1.5 scrub ok 2024-09-13T15:10:26.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[102314]: pgmap v1467: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T15:10:26.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:10:26.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:26.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[107038]: osdmap e464: 8 total, 8 up, 8 in 2024-09-13T15:10:26.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[107038]: 1.5 scrub starts 2024-09-13T15:10:26.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[107038]: 1.5 scrub ok 2024-09-13T15:10:26.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:25 smithi154 ceph-mon[107038]: pgmap v1467: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-13T15:10:26.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:26.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:26.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:26.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:26.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:26.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:26.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:10:26.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:26.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:26.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:26.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:26.737 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:26.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:26.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:26.886 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:26.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:26.907 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:26.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:10:26.951050+0000 min lat: 0.00103447 max lat: 0.561012 avg lat: 0.0098891 2024-09-13T15:10:26.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:10:26.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 60 16 97065 97049 101.071 70 0.0205309 0.0098891 2024-09-13T15:10:26.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:27.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:27.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:27.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:27.164 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:27.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:27.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[102314]: 1.2 deep-scrub starts 2024-09-13T15:10:27.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[102314]: 1.2 deep-scrub ok 2024-09-13T15:10:27.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T15:10:27.185 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:10:27.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[107038]: 1.2 deep-scrub starts 2024-09-13T15:10:27.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[107038]: 1.2 deep-scrub ok 2024-09-13T15:10:27.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T15:10:27.186 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:26 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:10:27.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-13T15:10:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:26 smithi172 ceph-mon[100696]: 1.2 deep-scrub starts 2024-09-13T15:10:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:26 smithi172 ceph-mon[100696]: 1.2 deep-scrub ok 2024-09-13T15:10:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:26 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-13T15:10:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:26 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:10:27.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:27.295 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:27.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:27.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:27.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:27.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:27.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:27.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:27.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:27.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:27.745 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:27.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:27.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:27.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:10:27.875 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:27.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:27.910 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:10:27.911 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:10:27.950 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:27.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 61 16 98042 98026 100.415 61.0625 0.0160784 0.00995295 2024-09-13T15:10:27.954 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:10:27.955 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:10:27.976 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:28.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:28.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:27 smithi154 ceph-mon[102314]: 1.9 deep-scrub starts 2024-09-13T15:10:28.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:27 smithi154 ceph-mon[102314]: 1.9 deep-scrub ok 2024-09-13T15:10:28.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:27 smithi154 ceph-mon[102314]: pgmap v1468: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.02k op/s 2024-09-13T15:10:28.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:27 smithi154 ceph-mon[107038]: 1.9 deep-scrub starts 2024-09-13T15:10:28.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:27 smithi154 ceph-mon[107038]: 1.9 deep-scrub ok 2024-09-13T15:10:28.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:27 smithi154 ceph-mon[107038]: pgmap v1468: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.02k op/s 2024-09-13T15:10:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.003 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:10:28.003 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:10:28.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:28.036 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:10:28.037 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:10:28.079 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:10:28.079 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:10:28.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:28.116 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:10:28.116 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:10:28.132 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:28.148 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:10:28.149 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:10:28.179 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:10:28.180 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:10:28.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:27 smithi172 ceph-mon[100696]: 1.9 deep-scrub starts 2024-09-13T15:10:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:27 smithi172 ceph-mon[100696]: 1.9 deep-scrub ok 2024-09-13T15:10:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:27 smithi172 ceph-mon[100696]: pgmap v1468: 99 pgs: 99 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 65 MiB/s wr, 1.02k op/s 2024-09-13T15:10:28.270 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:28.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:28.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:28.393 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:28.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:28.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:28.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:28.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:28.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:28.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:28.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:28.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 62 16 98991 98975 99.752 59.3125 0.0128519 0.0100198 2024-09-13T15:10:28.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:28.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:29.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:29.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:29.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:29.098 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:28 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2189633781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:29.098 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:10:29.098 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:29.098 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:28 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2189633781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:29.098 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:10:29.098 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:29.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:29.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:29.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:28 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2189633781' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:10:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:29.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:29.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:29.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:29.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:29.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:29.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:29.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:29.656 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:29.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:29.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:29.789 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:29.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:29.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:29.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:29.938 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:29 smithi154 ceph-mon[107038]: 1.1 scrub starts 2024-09-13T15:10:29.938 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:29 smithi154 ceph-mon[107038]: 1.1 scrub ok 2024-09-13T15:10:29.939 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:29 smithi154 ceph-mon[107038]: pgmap v1469: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-09-13T15:10:29.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:29 smithi154 ceph-mon[102314]: 1.1 scrub starts 2024-09-13T15:10:29.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:29 smithi154 ceph-mon[102314]: 1.1 scrub ok 2024-09-13T15:10:29.939 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:29 smithi154 ceph-mon[102314]: pgmap v1469: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-09-13T15:10:29.951 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 63 16 100053 100037 99.2222 66.375 0.0092474 0.0100735 2024-09-13T15:10:30.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:30.068 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:30.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:30.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:30.198 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:30.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:29 smithi172 ceph-mon[100696]: 1.1 scrub starts 2024-09-13T15:10:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:29 smithi172 ceph-mon[100696]: 1.1 scrub ok 2024-09-13T15:10:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:29 smithi172 ceph-mon[100696]: pgmap v1469: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 72 MiB/s wr, 1.14k op/s 2024-09-13T15:10:30.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:30.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:30.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:30.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:30.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:30.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:30.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:30.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:30.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:30.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:30.767 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:30.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:30.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:30.900 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:30.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:30.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 64 16 101131 101115 98.7246 67.375 0.0235339 0.0101218 2024-09-13T15:10:31.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:31.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:31.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:31.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:31.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:31.282 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:31 smithi154 ceph-mon[102314]: pgmap v1470: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.07k op/s 2024-09-13T15:10:31.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:31.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:31.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:31.461 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:10:31.461 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:10:31.461 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:10:31.461 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:10:31.461 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:10:31.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:31.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:31.529 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:31.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:31.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:31.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:31.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:31 smithi154 ceph-mon[107038]: pgmap v1470: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.07k op/s 2024-09-13T15:10:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:31 smithi172 ceph-mon[100696]: pgmap v1470: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 68 MiB/s wr, 1.07k op/s 2024-09-13T15:10:31.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:31.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:31.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:31.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:31.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:31.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 65 16 102073 102057 98.1115 58.875 0.0152723 0.0101873 2024-09-13T15:10:31.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:10:31.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:32.015 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:32.023 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:10:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:10:31] "GET /metrics HTTP/1.1" 200 36658 "" "Prometheus/2.51.0" 2024-09-13T15:10:32.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-13T15:10:32.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:32.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:32.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:10:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:32.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:32.285 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:32.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:32.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:32.432 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:32.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:32.485 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:32.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:32.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:32.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:32.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:32.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:32.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:32.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:32.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 66 16 102926 102910 97.4327 53.3125 0.00629351 0.0102589 2024-09-13T15:10:32.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:32.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:33.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:33.119 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:33.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:33.221 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:33.256 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:33.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:33.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:33.395 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:33.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:33.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:33 smithi154 ceph-mon[102314]: pgmap v1471: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-09-13T15:10:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:33 smithi154 ceph-mon[107038]: pgmap v1471: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-09-13T15:10:33.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:33.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:33.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:33.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:33.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:33.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:33.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:33.717 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:33 smithi172 ceph-mon[100696]: pgmap v1471: 99 pgs: 99 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-09-13T15:10:33.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:33.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:33.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:33.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:33.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 67 16 103990 103974 96.971 66.5 0.0175704 0.0103074 2024-09-13T15:10:33.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:33.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:34.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:34.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:34.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:34.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:34.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:34.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:34.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:34.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:34.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:34.381 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:10:34.381 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213596 8.5 8:a9c8065b:::benchmark_data_smithi154_261841_object97557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.764454+0000", 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 389.61272035399998, 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.646727278, 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.382 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213596 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764454+0000", 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764454+0000", 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764452+0000", 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764457+0000", 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8219999999999997e-06 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764465+0000", 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3389999999999993e-06 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764471+0000", 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4570000000000002e-06 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.767499+0000", 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003027596 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.767511+0000", 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1590000000000001e-05 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002167+0000", 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.23465654899999999 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002178+0000", 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1062e-05 2024-09-13T15:10:34.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.399719+0000", 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.397541014 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.399855+0000", 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558299999999999 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.399959+0000", 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103823 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404097+0000", 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00413862 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404107+0000", 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8490000000000001e-06 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.410946+0000", 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068393359999999997 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411005+0000", 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8853e-05 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411055+0000", 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0142000000000002e-05 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411181+0000", 2024-09-13T15:10:34.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012549999999999999 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213599 8.5 8:a4956294:::benchmark_data_smithi154_261841_object97560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.780614+0000", 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 389.596559985, 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.638991925, 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.392 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213599 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780614+0000", 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780614+0000", 2024-09-13T15:10:34.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780613+0000", 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780618+0000", 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6189999999999999e-06 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780627+0000", 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549999999999995e-06 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780632+0000", 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3449999999999996e-06 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780668+0000", 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5544999999999999e-05 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780685+0000", 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7547999999999999e-05 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002203+0000", 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.22151791100000001 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002211+0000", 2024-09-13T15:10:34.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-13T15:10:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404144+0000", 2024-09-13T15:10:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.401932687 2024-09-13T15:10:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404291+0000", 2024-09-13T15:10:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014680699999999999 2024-09-13T15:10:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:10:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404431+0000", 2024-09-13T15:10:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140438 2024-09-13T15:10:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.410961+0000", 2024-09-13T15:10:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065295869999999999 2024-09-13T15:10:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411228+0000", 2024-09-13T15:10:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000267304 2024-09-13T15:10:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419169+0000", 2024-09-13T15:10:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079415240000000002 2024-09-13T15:10:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419445+0000", 2024-09-13T15:10:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027508099999999999 2024-09-13T15:10:34.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419496+0000", 2024-09-13T15:10:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1307999999999997e-05 2024-09-13T15:10:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419606+0000", 2024-09-13T15:10:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011013 2024-09-13T15:10:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213601 8.5 8:a4f419b3:::benchmark_data_smithi154_261841_object97562:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:10:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.781971+0000", 2024-09-13T15:10:34.406 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 389.59520348799998, 2024-09-13T15:10:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.637816929, 2024-09-13T15:10:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:10:34.407 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:10:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213601 2024-09-13T15:10:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781971+0000", 2024-09-13T15:10:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781971+0000", 2024-09-13T15:10:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781970+0000", 2024-09-13T15:10:34.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781975+0000", 2024-09-13T15:10:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3909999999999997e-06 2024-09-13T15:10:34.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781988+0000", 2024-09-13T15:10:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2799000000000001e-05 2024-09-13T15:10:34.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781994+0000", 2024-09-13T15:10:34.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4119999999999998e-06 2024-09-13T15:10:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.782096+0000", 2024-09-13T15:10:34.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010180500000000001 2024-09-13T15:10:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.782109+0000", 2024-09-13T15:10:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2653000000000001e-05 2024-09-13T15:10:34.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002234+0000", 2024-09-13T15:10:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.220124976 2024-09-13T15:10:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002241+0000", 2024-09-13T15:10:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7190000000000001e-06 2024-09-13T15:10:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.407884+0000", 2024-09-13T15:10:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4056430579999999 2024-09-13T15:10:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.407977+0000", 2024-09-13T15:10:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2819999999999996e-05 2024-09-13T15:10:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:10:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.408068+0000", 2024-09-13T15:10:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0521999999999998e-05 2024-09-13T15:10:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411452+0000", 2024-09-13T15:10:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003384569 2024-09-13T15:10:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411465+0000", 2024-09-13T15:10:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2619000000000001e-05 2024-09-13T15:10:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419368+0000", 2024-09-13T15:10:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079028039999999994 2024-09-13T15:10:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419637+0000", 2024-09-13T15:10:34.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000269435 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419681+0000", 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3640000000000002e-05 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419787+0000", 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106662 2024-09-13T15:10:34.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213607 8.5 8:aaa2bad7:::benchmark_data_smithi154_261841_object97568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.791517+0000", 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 389.585656884, 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.636411989, 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:10:34.425 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213607 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791517+0000", 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791517+0000", 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791516+0000", 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791522+0000", 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791537+0000", 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.517e-05 2024-09-13T15:10:34.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791546+0000", 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9649999999999997e-06 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791609+0000", 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791620+0000", 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0609e-05 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002264+0000", 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.21064449800000001 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002272+0000", 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1950000000000003e-06 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411518+0000", 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4092451719999999 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411648+0000", 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130488 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411786+0000", 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013809200000000001 2024-09-13T15:10:34.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419180+0000", 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0073940580000000002 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419403+0000", 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222943 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419854+0000", 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000450965 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427743+0000", 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078889719999999993 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427797+0000", 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4205e-05 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427929+0000", 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013182299999999999 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213617 8.5 8:ad66873f:::benchmark_data_smithi154_261841_object97578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.802801+0000", 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 389.57437318400002, 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.634649596, 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.433 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213617 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802801+0000", 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802801+0000", 2024-09-13T15:10:34.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802800+0000", 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802805+0000", 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4989999999999996e-06 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802815+0000", 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0200000000000001e-05 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802820+0000", 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5330000000000002e-06 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802870+0000", 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9747999999999999e-05 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802880+0000", 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0244000000000001e-05 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002295+0000", 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.19941504400000001 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002303+0000", 2024-09-13T15:10:34.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.415020+0000", 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4127169369999999 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.415150+0000", 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129488 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:10:34.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.415294+0000", 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014416200000000001 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419617+0000", 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043235770000000003 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419898+0000", 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028118300000000001 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419938+0000", 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9638000000000002e-05 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437248+0000", 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017309486999999998 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437330+0000", 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2735e-05 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437450+0000", 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120112 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213622 8.5 8:a3a31e6c:::benchmark_data_smithi154_261841_object97583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.812504+0000", 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 389.56467034500002, 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.625395114, 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.441 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213622 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812504+0000", 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812504+0000", 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812503+0000", 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812508+0000", 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2959999999999998e-06 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812519+0000", 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1374e-05 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812525+0000", 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4580000000000001e-06 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812578+0000", 2024-09-13T15:10:34.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3149000000000002e-05 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812590+0000", 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1857e-05 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002327+0000", 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.189736714 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002353+0000", 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6916000000000001e-05 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419970+0000", 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.417616773 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.420096+0000", 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012563900000000001 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.420199+0000", 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010286 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427696+0000", 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976080000000002 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427705+0000", 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3820000000000004e-06 2024-09-13T15:10:34.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437185+0000", 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094806119999999994 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437504+0000", 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000319062 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437575+0000", 2024-09-13T15:10:34.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0785999999999996e-05 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437899+0000", 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032361199999999998 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18364 9.e 9:7d038a3a:::benchmark_data_smithi154_291684_object18363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.155489+0000", 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 318.22168467699998, 2024-09-13T15:10:34.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.113528574, 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18364 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155489+0000", 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155489+0000", 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155488+0000", 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155655+0000", 2024-09-13T15:10:34.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167205 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155666+0000", 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599e-05 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155680+0000", 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.38e-05 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155740+0000", 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9725000000000002e-05 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155752+0000", 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1722999999999999e-05 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157766+0000", 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.002014593 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157780+0000", 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3730999999999999e-05 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.258720+0000", 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.100940357 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.259080+0000", 2024-09-13T15:10:34.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035937099999999999 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.259194+0000", 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114016 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263014+0000", 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038206659999999999 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263071+0000", 2024-09-13T15:10:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7049999999999998e-05 2024-09-13T15:10:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.268384+0000", 2024-09-13T15:10:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005312092 2024-09-13T15:10:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.268422+0000", 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8442999999999998e-05 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269018+0000", 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00059586600000000002 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.458 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18366 9.e 9:7f15721d:::benchmark_data_smithi154_291684_object18365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 318.22147485199997, 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1138882319999999, 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18366 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:10:34.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155781+0000", 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2699000000000002e-05 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155791+0000", 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7049999999999994e-06 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155797+0000", 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3740000000000003e-06 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155900+0000", 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010247300000000001 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155909+0000", 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8650000000000007e-06 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157806+0000", 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0018974 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157814+0000", 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.260479+0000", 2024-09-13T15:10:34.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1026644379999999 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.260631+0000", 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015289599999999999 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.260821+0000", 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018936100000000001 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263902+0000", 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003081056 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.265074+0000", 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001172183 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269229+0000", 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041554729999999998 2024-09-13T15:10:34.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269263+0000", 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3890999999999998e-05 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269587+0000", 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032401100000000003 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18371 9.e 9:7b44ede2:::benchmark_data_smithi154_291684_object18370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.158169+0000", 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 318.21900519500002, 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1119104590000002, 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18371 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158169+0000", 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158169+0000", 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158168+0000", 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158243+0000", 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5414999999999995e-05 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158259+0000", 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5985e-05 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158267+0000", 2024-09-13T15:10:34.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3189999999999995e-06 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158318+0000", 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.079e-05 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158328+0000", 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5289999999999997e-06 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.158078+0000", 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99975051999999998 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.158090+0000", 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1610999999999999e-05 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.261496+0000", 2024-09-13T15:10:34.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.103406554 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.261596+0000", 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9580000000000005e-05 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.261722+0000", 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012574400000000001 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263906+0000", 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002184044 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.268326+0000", 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044207480000000004 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269633+0000", 2024-09-13T15:10:34.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013068559999999999 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269681+0000", 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7639999999999998e-05 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.270079+0000", 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039841099999999999 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:66495 9.e 9:764958b5:::benchmark_data_smithi154_291684_object66494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:02.044075+0000", 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 272.33309949099998, 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.005924974, 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 66495 2024-09-13T15:10:34.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044075+0000", 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044075+0000", 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044073+0000", 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044136+0000", 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3113000000000004e-05 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044147+0000", 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1216999999999999e-05 2024-09-13T15:10:34.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044153+0000", 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6910000000000002e-06 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044373+0000", 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022015399999999999 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044386+0000", 2024-09-13T15:10:34.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2850999999999999e-05 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.043047+0000", 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99866041400000005 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.043168+0000", 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012136799999999999 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.043278+0000", 2024-09-13T15:10:34.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110513 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.048577+0000", 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052982929999999999 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049856+0000", 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012793310000000001 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049921+0000", 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5167999999999997e-05 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049961+0000", 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9283999999999999e-05 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049999+0000", 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8924000000000002e-05 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:66505 9.e 9:7e252385:::benchmark_data_smithi154_291684_object66504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:02.053055+0000", 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 272.32411898399999, 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0004483980000001, 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 66505 2024-09-13T15:10:34.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053055+0000", 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053055+0000", 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053054+0000", 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053101+0000", 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7327000000000003e-05 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053113+0000", 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.189e-05 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053118+0000", 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6319999999999996e-06 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053160+0000", 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1854999999999998e-05 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053168+0000", 2024-09-13T15:10:34.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5920000000000003e-06 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.045594+0000", 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99242650899999996 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.045699+0000", 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104552 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:10:34.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.045791+0000", 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1812999999999997e-05 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.050075+0000", 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042843610000000004 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053358+0000", 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003282807 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053412+0000", 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4205e-05 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053464+0000", 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1737000000000003e-05 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053503+0000", 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9548000000000002e-05 2024-09-13T15:10:34.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:66538 9.e 9:78a5717b:::benchmark_data_smithi154_291684_object66537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:02.086696+0000", 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 272.290478162, 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97339183900000004, 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 66538 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086696+0000", 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086696+0000", 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086695+0000", 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086785+0000", 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0914000000000004e-05 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086801+0000", 2024-09-13T15:10:34.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5778999999999999e-05 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086809+0000", 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0849999999999997e-06 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086870+0000", 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0584000000000001e-05 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086883+0000", 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3322000000000001e-05 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.046925+0000", 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96004212200000005 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.047053+0000", 2024-09-13T15:10:34.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127439 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.047143+0000", 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0160000000000004e-05 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053368+0000", 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062250099999999996 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053568+0000", 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000199736 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.060002+0000", 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064343539999999998 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.060053+0000", 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1057999999999998e-05 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.060088+0000", 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4586999999999999e-05 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:140809 9.10 9.ae4d0b30 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:07:27.322545+0000", 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 187.05462885599999, 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96802707600000004, 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.496 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140809 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322545+0000", 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322545+0000", 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322544+0000", 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322549+0000", 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0790000000000002e-06 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322557+0000", 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3979999999999999e-06 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322562+0000", 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322598+0000", 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6134000000000003e-05 2024-09-13T15:10:34.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322604+0000", 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7520000000000002e-06 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:28.290560+0000", 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96795643499999995 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:28.290572+0000", 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1846e-05 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:181320 9.16 9:6caf4932:::benchmark_data_smithi154_291684_object80859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:27.178644+0000", 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.19852988700001, 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0000961740000001, 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.501 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 181320 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178644+0000", 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178644+0000", 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178643+0000", 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178651+0000", 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.106e-06 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178665+0000", 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4652e-05 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178681+0000", 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.525e-05 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178756+0000", 2024-09-13T15:10:34.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5454e-05 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178766+0000", 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0172e-05 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.167116+0000", 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98834951999999998 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.167194+0000", 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7991000000000006e-05 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.167473+0000", 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279017 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.171334+0000", 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038613100000000002 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.174131+0000", 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027966689999999999 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178679+0000", 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045485079999999997 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178715+0000", 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6291000000000002e-05 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178740+0000", 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4800999999999999e-05 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:181321 9.16 9:6bf1129e:::benchmark_data_smithi154_291684_object80860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.198473698, 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0060002299999999, 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.508 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 181321 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:10:34.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178703+0000", 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5439999999999999e-06 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178710+0000", 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4150000000000004e-06 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178715+0000", 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5369999999999997e-06 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178795+0000", 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9561000000000006e-05 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178805+0000", 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0088999999999999e-05 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.168868+0000", 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99006304199999995 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.168941+0000", 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3268000000000004e-05 2024-09-13T15:10:34.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.169198+0000", 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025707900000000001 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.174072+0000", 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048734690000000001 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178772+0000", 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046999190000000003 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.184612+0000", 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058403880000000002 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.184656+0000", 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4199999999999997e-05 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.184701+0000", 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4305999999999999e-05 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73776 10.1 10:8adf5dcd:::benchmark_data_smithi154_317818_object73775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.958770+0000", 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.418403582, 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.55018106300000003, 2024-09-13T15:10:34.515 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73776 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958770+0000", 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958770+0000", 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958769+0000", 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958857+0000", 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7634999999999997e-05 2024-09-13T15:10:34.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958867+0000", 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9860000000000006e-06 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958872+0000", 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0390000000000004e-06 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958915+0000", 2024-09-13T15:10:34.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2933999999999999e-05 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959073+0000", 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015779800000000001 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959212+0000", 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013958800000000001 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.962946+0000", 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037334719999999998 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.966378+0000", 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034321540000000002 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508836+0000", 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54245829000000001 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508909+0000", 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2791000000000002e-05 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508951+0000", 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2485999999999999e-05 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.521 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73779 10.1 10:806e7cc1:::benchmark_data_smithi154_317818_object73778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.417391777999999, 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54928832000000005, 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73779 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959857+0000", 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4925999999999994e-05 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959865+0000", 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1280000000000008e-06 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959869+0000", 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5310000000000003e-06 2024-09-13T15:10:34.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.961136+0000", 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012669070000000001 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.961262+0000", 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012542599999999999 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:10:34.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.961404+0000", 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142538 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.963208+0000", 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018036040000000001 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.966437+0000", 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003228982 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508999+0000", 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54256280499999998 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509044+0000", 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4308000000000003e-05 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509071+0000", 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6755e-05 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.527 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73803 10.1 10:87b245ab:::benchmark_data_smithi154_317818_object73802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.980377+0000", 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.396797071999998, 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.52882779800000002, 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:10:34.528 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73803 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980377+0000", 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980377+0000", 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980375+0000", 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980421+0000", 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469000000000001e-05 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980432+0000", 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0791000000000001e-05 2024-09-13T15:10:34.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980437+0000", 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0180000000000001e-06 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980475+0000", 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8210000000000002e-05 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980586+0000", 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110992 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980709+0000", 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012335499999999999 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.982686+0000", 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019766470000000002 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.983185+0000", 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049875699999999995 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509123+0000", 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.52593843100000004 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509177+0000", 2024-09-13T15:10:34.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4243999999999998e-05 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509205+0000", 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7412000000000001e-05 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:76399 10.1 10:86e2ac82:::benchmark_data_smithi154_317818_object76398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:10:34.534 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:08.937537+0000", 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.439637379000001, 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.201735461, 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 76399 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937537+0000", 2024-09-13T15:10:34.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937537+0000", 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937535+0000", 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937599+0000", 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3637000000000001e-05 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937613+0000", 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4076999999999999e-05 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937620+0000", 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5480000000000004e-06 2024-09-13T15:10:34.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937669+0000", 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9054999999999997e-05 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937828+0000", 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000158634 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.937967+0000", 2024-09-13T15:10:34.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013946400000000001 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944819+0000", 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068518119999999997 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944834+0000", 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4484000000000001e-05 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:09.139187+0000", 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.19435345100000001 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:09.139235+0000", 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7695999999999999e-05 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:09.139272+0000", 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7289000000000002e-05 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.541 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:76410 10.1 10:80f57d7a:::benchmark_data_smithi154_317818_object76409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-13T15:10:34.541 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:08.944744+0000", 2024-09-13T15:10:34.541 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.432429521, 2024-09-13T15:10:34.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.19974735099999999, 2024-09-13T15:10:34.541 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:34.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:34.542 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:34.542 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:10:34.542 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 76410 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944744+0000", 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944744+0000", 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:34.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944743+0000", 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944815+0000", 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1647e-05 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:34.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944825+0000", 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0317e-05 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.944832+0000", 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8940000000000004e-06 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.945534+0000", 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070234900000000001 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.945702+0000", 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167489 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.945854+0000", 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151867 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.948124+0000", 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022703720000000001 2024-09-13T15:10:34.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:08.948765+0000", 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00064048499999999999 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:09.144407+0000", 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.19564208599999999 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:09.144453+0000", 2024-09-13T15:10:34.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6727000000000002e-05 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:09.144492+0000", 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8463000000000001e-05 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:34.548 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:10:34.578 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:34.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:34.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:34.705 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:34.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:34.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:34.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:34.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:34.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:34.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 68 16 104952 104936 96.4292 60.125 0.0151746 0.0103653 2024-09-13T15:10:34.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:34.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:35.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:35.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:35.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:35.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:35.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:35.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:35.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:35.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:35.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:35.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:10:35.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:35.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:35.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:35.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:35.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:35.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:35 smithi154 ceph-mon[102314]: pgmap v1472: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-09-13T15:10:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:35 smithi154 ceph-mon[107038]: pgmap v1472: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-09-13T15:10:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:35 smithi172 ceph-mon[100696]: pgmap v1472: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-09-13T15:10:35.785 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:35.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:35.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:35.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:35.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:35.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:35.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:35.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 69 16 105913 105897 95.9021 60.0625 0.0160343 0.0104225 2024-09-13T15:10:36.050 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:36.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:36.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:36.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:36.250 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:36.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:36.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:36.389 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:36.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:36.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:36.512 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:36.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:36.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:36.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:36.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:36.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:36.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:36.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:36.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:36.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:36.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:36.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 70 16 106835 106819 95.3553 57.625 0.0134951 0.0104828 2024-09-13T15:10:37.019 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:37.052 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:37.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:37.123 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:37.125 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:37.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:37.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:37.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:37.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:37.363 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:37.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:37.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:37 smithi172 ceph-mon[100696]: pgmap v1473: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:10:37.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:37.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:37.565 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:37 smithi154 ceph-mon[102314]: pgmap v1473: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:10:37.565 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:37 smithi154 ceph-mon[107038]: pgmap v1473: 99 pgs: 99 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-13T15:10:37.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:37.654 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:37.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:37.689 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:10:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:10:37.689 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T15:10:37.689 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:10:37.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:37.790 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:37.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:37.892 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:37.925 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:37.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:37.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:37.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 71 16 107825 107809 94.8837 61.875 0.00493552 0.0105346 2024-09-13T15:10:38.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:38.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:38.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:38.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:38.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:38.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:38.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:38.304 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:38.342 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:38.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:38.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:38.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:10:38.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:38.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:38.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:38.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:38.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:38.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:38.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:38.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:38.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:38.810 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:38.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:38.890 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:38.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:38.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 72 16 108824 108808 94.433 62.4375 0.0215706 0.0105842 2024-09-13T15:10:38.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:39.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:39.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:39.077 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:10:39.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:39.164 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:39.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:39 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3769509313' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:10:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:39 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:10:39.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:39.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:39.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:39.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:39.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:39.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:39.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:39 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3769509313' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:10:39.465 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:39 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:10:39.465 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:39 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3769509313' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:10:39.465 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:39 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:10:39.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:39.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:39.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:39.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:39.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:39.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:39.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:39.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:39.866 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:39.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:39.952 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 73 16 109820 109804 93.9921 62.25 0.0142708 0.0106342 2024-09-13T15:10:39.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:40.000 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:40.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.145 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:40.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:40.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:40.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:40 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:40.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:40 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:10:40.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:40 smithi172 ceph-mon[100696]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T15:10:40.269 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:40 smithi172 ceph-mon[100696]: pgmap v1475: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-13T15:10:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:40.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:40.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:40.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:40.447 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:40.448 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:10:40.448 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[102314]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T15:10:40.448 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[102314]: pgmap v1475: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-13T15:10:40.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:40.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:10:40.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[107038]: osdmap e465: 8 total, 8 up, 8 in 2024-09-13T15:10:40.448 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:40 smithi154 ceph-mon[107038]: pgmap v1475: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-13T15:10:40.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.562 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:40.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:40.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:40.698 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:40.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:40.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:40.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:40.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:40.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 74 16 110839 110823 93.582 63.6875 0.0299773 0.0106801 2024-09-13T15:10:40.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:40.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:41.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:41.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:41.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:41.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:41.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:41.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:41.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:41.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:41.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:41.427 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:10:41.427 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:10:41.427 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:10:41.428 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:10:41.428 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:10:41.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:41.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:41.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:41.534 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:41 smithi154 ceph-mon[102314]: pgmap v1476: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-13T15:10:41.534 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:41 smithi154 ceph-mon[107038]: pgmap v1476: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-13T15:10:41.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:41.658 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:41.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:41 smithi172 ceph-mon[100696]: pgmap v1476: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-13T15:10:41.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:41.796 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:41.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:41.834 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:10:41.834 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:10:41.875 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:10:41.876 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:10:41.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:41.909 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:10:41.910 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:10:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:41.927 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:41.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:41.944 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:10:41.944 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:10:41.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 75 16 111756 111740 93.0984 57.3125 0.0168709 0.0107362 2024-09-13T15:10:41.977 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:10:41.978 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:10:42.016 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:10:42.016 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:10:42.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:42.048 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:10:42.048 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:10:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:42.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:42.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:42.089 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:10:42.089 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:10:42.090 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:10:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:10:41] "GET /metrics HTTP/1.1" 200 36658 "" "Prometheus/2.51.0" 2024-09-13T15:10:42.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:10:42.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:42.225 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:42.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:42.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:42.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:42.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:42.394 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:42 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3417830897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:42.394 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:42 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3417830897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:42.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:42.482 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:42 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3417830897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:42.525 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:42.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:42.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:42.659 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:42.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:42.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:42.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:42.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:42.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:42.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:42.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:10:42.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:42.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:42.953 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 76 16 112760 112744 92.6991 62.75 0.0208387 0.0107831 2024-09-13T15:10:43.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:43.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:43.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:43.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:43.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:43.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:43.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:43.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:43.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:43.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:43.491 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:43.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:43.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:43.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:43.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:43.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-13T15:10:43.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:43.778 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:43.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:43.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:43.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:43.954 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 77 16 113695 113679 92.253 58.4375 0.0133384 0.0108344 2024-09-13T15:10:43.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:43.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:44.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:44.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:44.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:44.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:10:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:44 smithi154 ceph-mon[102314]: pgmap v1477: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-13T15:10:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:10:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:44 smithi154 ceph-mon[107038]: pgmap v1477: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-13T15:10:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:44.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:44.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:44.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:44.380 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:10:44.381 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:44.381 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:44 smithi172 ceph-mon[100696]: pgmap v1477: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-13T15:10:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:44.390 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:44.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:44.484 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:10:44.484 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:10:44.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:44.529 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:44.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:44.625 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:44.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:44.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:44.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:44.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:44.807 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:44.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:44.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:44.942 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:44.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:44.954 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 78 16 114643 114627 91.8299 59.25 0.0109706 0.0108848 2024-09-13T15:10:45.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:10:45.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:45.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:45.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:45.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:45.239 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:45.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:45.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:45.366 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:45.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:45.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:45.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:45 smithi154 ceph-mon[102314]: pgmap v1478: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-13T15:10:45.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:45 smithi154 ceph-mon[107038]: pgmap v1478: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-13T15:10:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:45 smithi172 ceph-mon[100696]: pgmap v1478: 99 pgs: 99 active+clean; 6.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-13T15:10:45.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:45.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:45.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:45.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:45.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:45.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:45.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:45.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:45.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:45.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:45.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:45.954 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 79 16 115593 115577 91.4191 59.375 0.0119909 0.010933 2024-09-13T15:10:46.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:46.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:46.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:46.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:46.200 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:46.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:46.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:46.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:46.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:46.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:46.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:10:46.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:46.514 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:46.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:46.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:46.656 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:46.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:46.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:46.801 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:46.809 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:46.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:46.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:46.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:46.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:46.955 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:2024-09-13T15:10:46.954798+0000 min lat: 0.00103447 max lat: 0.561012 avg lat: 0.0109854 2024-09-13T15:10:46.955 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-13T15:10:46.955 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 80 16 116491 116475 90.9771 56.125 0.00949229 0.0109854 2024-09-13T15:10:47.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:47.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:47.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:47.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:47.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:47.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:47.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:47.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:47.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:47.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:47.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:47.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:47.602 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:47 smithi154 ceph-mon[102314]: pgmap v1479: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T15:10:47.602 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:47 smithi154 ceph-mon[107038]: pgmap v1479: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T15:10:47.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:47.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:47.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:47.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:47 smithi172 ceph-mon[100696]: pgmap v1479: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-13T15:10:47.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:47.778 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:47.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:47.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:47.919 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:47.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:47.955 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 81 16 117445 117429 90.5901 59.625 0.00944073 0.0110328 2024-09-13T15:10:48.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:48.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:48.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:48.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:48.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:48.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:48.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:48.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:48.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:48.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:48.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:48.496 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:48.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:48.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:48.602 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:48.639 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:48.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:48.695 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:10:48.695 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:10:48.718 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:10:48.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:48.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:48.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:48.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:48.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:48.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:48.955 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 82 16 118285 118269 90.1256 52.5 0.0164472 0.0110908 2024-09-13T15:10:48.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4098132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:48.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:48 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:48.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4098132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:48.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:48 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:48.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4098132' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:48.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:48 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:10:49.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:49.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:49.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:49.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:49.187 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:49.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:49.289 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:49.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:49.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:49.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:49.452 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:10:49.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:49.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:49.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:49.606 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:49.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:49.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-13T15:10:49.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:49.730 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:49 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:49.730 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:49 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:49.730 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:49 smithi172 ceph-mon[100696]: osdmap e466: 8 total, 8 up, 8 in 2024-09-13T15:10:49.730 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:49 smithi172 ceph-mon[100696]: pgmap v1481: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-13T15:10:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:49.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:49.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:49.791 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:49.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:49.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:49.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:49.956 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 83 16 119254 119238 89.7679 60.5625 0.0164983 0.0111349 2024-09-13T15:10:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:49.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[102314]: osdmap e466: 8 total, 8 up, 8 in 2024-09-13T15:10:49.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[102314]: pgmap v1481: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-13T15:10:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:10:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:10:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[107038]: osdmap e466: 8 total, 8 up, 8 in 2024-09-13T15:10:49.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:49 smithi154 ceph-mon[107038]: pgmap v1481: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-13T15:10:50.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:50.095 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:50.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:50.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:50.230 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:50.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:50.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:50.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:50.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:50.462 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:50.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:50.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:50.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:50.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:50.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:50.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:50.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:50.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:50.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:50.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:50.957 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 84 16 120218 120202 89.4165 60.25 0.0226209 0.0111786 2024-09-13T15:10:51.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:51.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:51.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:51.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:51.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:51.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:51.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:51.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:51.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:51.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:51.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:51.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:51.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:51.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:51.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:51.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:51 smithi154 ceph-mon[102314]: pgmap v1482: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 59 MiB/s wr, 947 op/s 2024-09-13T15:10:51.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:51 smithi154 ceph-mon[107038]: pgmap v1482: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 59 MiB/s wr, 947 op/s 2024-09-13T15:10:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:51 smithi172 ceph-mon[100696]: pgmap v1482: 99 pgs: 99 active+clean; 7.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 59 MiB/s wr, 947 op/s 2024-09-13T15:10:51.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:51.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:51.849 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:51.878 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:51.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:51.957 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 85 16 121169 121153 89.0638 59.4375 0.00822492 0.0112219 2024-09-13T15:10:51.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:52.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:52.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:52.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:10:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:52.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:10:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:10:51] "GET /metrics HTTP/1.1" 200 36646 "" "Prometheus/2.51.0" 2024-09-13T15:10:52.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:52.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:52.400 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:52.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:52.447 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:52.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:52.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:52.672 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:52 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:10:52.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:52.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:52 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:10:52.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:52 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:10:52.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:52.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:52.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:52.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:52.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 86 16 122054 122038 88.6711 55.3125 0.0094678 0.0112723 2024-09-13T15:10:52.981 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:52.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:53.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:53.098 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:10:53.098 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:10:53.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:53.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:53.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:53.255 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:53.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:53.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:53.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:53.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:53.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:53 smithi154 ceph-mon[102314]: pgmap v1483: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s 2024-09-13T15:10:53.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:53 smithi154 ceph-mon[107038]: pgmap v1483: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s 2024-09-13T15:10:53.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:53.513 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:53.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:53.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:53.653 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:53.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:53.681 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:53 smithi172 ceph-mon[100696]: pgmap v1483: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 59 MiB/s wr, 940 op/s 2024-09-13T15:10:53.756 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:53.797 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:53.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:53.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:10:53.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:53.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:53.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:53.957 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 87 16 122943 122927 88.2905 55.5625 0.00862541 0.0113215 2024-09-13T15:10:54.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:54.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:54.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:54.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:54.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:54.239 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:54.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:54.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:54.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:54.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:54.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:54.516 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:54.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:54.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:54.672 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:54.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:54.702 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:54.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:54.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:54.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:54.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:54.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:54.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:54.957 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 88 16 123847 123831 87.9292 56.5 0.0124474 0.0113681 2024-09-13T15:10:55.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:55.110 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:55.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:55.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:55.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:55.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:55.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:55.364 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:10:55.381 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:10:55.381 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:10:55.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:55.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:55.427 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:55 smithi154 ceph-mon[102314]: pgmap v1484: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T15:10:55.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:55 smithi154 ceph-mon[107038]: pgmap v1484: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T15:10:55.431 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:10:55.432 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:10:55.462 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:10:55.462 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:10:55.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:55 smithi172 ceph-mon[100696]: pgmap v1484: 99 pgs: 99 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-13T15:10:55.498 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:10:55.498 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:10:55.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:55.534 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:10:55.534 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:10:55.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:55.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:55.568 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:10:55.569 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:10:55.601 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:10:55.602 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:10:55.635 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:10:55.636 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:10:55.647 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:55.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:55.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:55.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:55.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:55.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:55.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:55.957 INFO:tasks.radosbench.radosbench.0.smithi154.stdout: 89 16 124730 124714 87.5613 55.1875 0.0128528 0.0114151 2024-09-13T15:10:55.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:55.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:56.079 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:56.112 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:56.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:56.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:56.248 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:56.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:56.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:56.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:56.397 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:56.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:56.411 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:56 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1626862473' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:56.412 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:56 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1626862473' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:56.501 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:56.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:56.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:56.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:56.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:56.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:56 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1626862473' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:10:56.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:56.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:56.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:56.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:56.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:56.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total time run: 90.0135 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Total writes made: 125673 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Write size: 65536 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Object size: 65536 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Bandwidth (MB/sec): 87.2598 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Bandwidth: 37.8259 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max bandwidth (MB/sec): 228.688 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min bandwidth (MB/sec): 34.5625 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average IOPS: 1396 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev IOPS: 605.215 2024-09-13T15:10:56.958 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max IOPS: 3659 2024-09-13T15:10:56.959 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min IOPS: 553 2024-09-13T15:10:56.959 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Average Latency(s): 0.0114556 2024-09-13T15:10:56.959 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Stddev Latency(s): 0.0119288 2024-09-13T15:10:56.959 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Max latency(s): 0.561012 2024-09-13T15:10:56.959 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Min latency(s): 0.00103447 2024-09-13T15:10:56.965 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Cleaning up (deleting benchmark objects) 2024-09-13T15:10:57.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:57.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:57.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:57.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:57.239 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:57.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:57.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:57.381 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:57.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:57.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:57 smithi172 ceph-mon[100696]: pgmap v1485: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 57 MiB/s wr, 919 op/s 2024-09-13T15:10:57.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:10:57.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:57.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:10:57.520 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:57 smithi154 ceph-mon[102314]: pgmap v1485: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 57 MiB/s wr, 919 op/s 2024-09-13T15:10:57.520 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:57 smithi154 ceph-mon[107038]: pgmap v1485: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 57 MiB/s wr, 919 op/s 2024-09-13T15:10:57.563 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:10:57.563 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:10:57.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:10:57.563 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:10:57.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.563 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:135001 9.0 e407/374)", 2024-09-13T15:10:57.563 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:07:21.253118+0000", 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 216.30436144500001, 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.092129774999999997, 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253118+0000", 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253118+0000", 2024-09-13T15:10:57.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253117+0000", 2024-09-13T15:10:57.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253126+0000", 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4409999999999999e-06 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253142+0000", 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6256000000000002e-05 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253149+0000", 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5579999999999997e-06 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.345228+0000", 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.092078907000000002 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.345247+0000", 2024-09-13T15:10:57.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806999999999999e-05 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:135004 9.1 e407/390)", 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:07:21.256337+0000", 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 216.30114244399999, 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.113890226, 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:10:57.568 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256337+0000", 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256337+0000", 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256336+0000", 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256345+0000", 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9169999999999995e-06 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256363+0000", 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7989e-05 2024-09-13T15:10:57.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256371+0000", 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.303e-06 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.370207+0000", 2024-09-13T15:10:57.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.113835671 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.370227+0000", 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0285999999999999e-05 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.572 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:101345 10.7 e464/448 10:ed23c9b0:::benchmark_data_smithi154_317818_object101344:head v 464'6266, mlcod=464'6263)", 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:31.190218+0000", 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.367261224, 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029591577000000001, 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.190218+0000", 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.190218+0000", 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.190217+0000", 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.208646+0000", 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018429109999999999 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.208669+0000", 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2611000000000001e-05 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.208688+0000", 2024-09-13T15:10:57.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9171e-05 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.208943+0000", 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000255212 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.208964+0000", 2024-09-13T15:10:57.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1274999999999999e-05 2024-09-13T15:10:57.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.219770+0000", 2024-09-13T15:10:57.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010805208 2024-09-13T15:10:57.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:31.219810+0000", 2024-09-13T15:10:57.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9923999999999999e-05 2024-09-13T15:10:57.579 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.579 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.579 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.580 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:106027 10.7 e464/448 10:e0c80ac8:::benchmark_data_smithi154_317818_object106026:head v 464'6562, mlcod=464'6559)", 2024-09-13T15:10:57.580 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:36.063869+0000", 2024-09-13T15:10:57.580 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.493610114999999, 2024-09-13T15:10:57.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027751372999999999, 2024-09-13T15:10:57.580 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.581 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.581 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.063869+0000", 2024-09-13T15:10:57.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.063869+0000", 2024-09-13T15:10:57.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.063868+0000", 2024-09-13T15:10:57.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.063902+0000", 2024-09-13T15:10:57.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3702999999999999e-05 2024-09-13T15:10:57.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.063919+0000", 2024-09-13T15:10:57.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7387000000000001e-05 2024-09-13T15:10:57.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.063927+0000", 2024-09-13T15:10:57.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6480000000000003e-06 2024-09-13T15:10:57.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.066445+0000", 2024-09-13T15:10:57.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002518143 2024-09-13T15:10:57.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.066465+0000", 2024-09-13T15:10:57.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0421999999999999e-05 2024-09-13T15:10:57.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.091276+0000", 2024-09-13T15:10:57.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024810684999999999 2024-09-13T15:10:57.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.091620+0000", 2024-09-13T15:10:57.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000344264 2024-09-13T15:10:57.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.591 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:106030 10.7 e464/448 10:e2e8dfca:::benchmark_data_smithi154_317818_object106029:head v 464'6563, mlcod=464'6559)", 2024-09-13T15:10:57.592 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:36.068982+0000", 2024-09-13T15:10:57.592 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.488497473999999, 2024-09-13T15:10:57.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029076972999999999, 2024-09-13T15:10:57.592 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.592 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.593 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.068982+0000", 2024-09-13T15:10:57.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.068982+0000", 2024-09-13T15:10:57.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.068980+0000", 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.069022+0000", 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2030000000000002e-05 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.069043+0000", 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1041999999999999e-05 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.069054+0000", 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0518e-05 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.070132+0000", 2024-09-13T15:10:57.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010777530000000001 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.070181+0000", 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9254e-05 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.097858+0000", 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027677118000000001 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:36.098059+0000", 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020075200000000001 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:106918 10.f e464/448 10:f21363e6:::benchmark_data_smithi154_317818_object106917:head v 464'6557, mlcod=464'6554)", 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:37.039950+0000", 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.517529152000002, 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029953252, 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.039950+0000", 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.039950+0000", 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.039949+0000", 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.039982+0000", 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2651000000000003e-05 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.039996+0000", 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3824e-05 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.040002+0000", 2024-09-13T15:10:57.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9280000000000004e-06 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.040934+0000", 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093115699999999995 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.040956+0000", 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603999999999999e-05 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.069726+0000", 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028769863999999999 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:37.069903+0000", 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177147 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.603 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:110129 10.e e465/448 10:76c5518a:::benchmark_data_smithi154_317818_object110128:head v 465'6877, mlcod=465'6873)", 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:40.263572+0000", 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.293907406999999, 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042002386000000003, 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263572+0000", 2024-09-13T15:10:57.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263572+0000", 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263571+0000", 2024-09-13T15:10:57.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263595+0000", 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4114e-05 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263615+0000", 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9522e-05 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263634+0000", 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9315999999999999e-05 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263813+0000", 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017923700000000001 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263857+0000", 2024-09-13T15:10:57.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3520999999999999e-05 2024-09-13T15:10:57.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305474+0000", 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041617020999999997 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305574+0000", 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100549 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:110130 10.e e465/448 10:7f0c4a40:::benchmark_data_smithi154_317818_object110129:head v 465'6878, mlcod=465'6874)", 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:40.269985+0000", 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.287493756, 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035673300999999998, 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.269985+0000", 2024-09-13T15:10:57.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.269985+0000", 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.269984+0000", 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270007+0000", 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270019+0000", 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1476e-05 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270026+0000", 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.939e-06 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270531+0000", 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050518399999999999 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270552+0000", 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1248000000000001e-05 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305580+0000", 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035027957999999998 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305659+0000", 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8716999999999999e-05 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:110135 10.e e465/448 10:750fac30:::benchmark_data_smithi154_317818_object110134:head v 465'6879, mlcod=465'6874)", 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:40.272516+0000", 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.284963045000001, 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033235885, 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.272516+0000", 2024-09-13T15:10:57.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.272516+0000", 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.272515+0000", 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.272539+0000", 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3487999999999999e-05 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.272551+0000", 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.185e-05 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.272557+0000", 2024-09-13T15:10:57.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5560000000000004e-06 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.284067+0000", 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011509398000000001 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.284084+0000", 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.785e-05 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305661+0000", 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021576442000000001 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305752+0000", 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1178e-05 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.619 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125395 10.d e466/448 10:bb86cf95:::benchmark_data_smithi154_317818_object125394:head v 466'7793, mlcod=466'7789)", 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:56.665422+0000", 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.89205756800000002, 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027395676000000001, 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.665422+0000", 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.665422+0000", 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.665421+0000", 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.665448+0000", 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7487999999999998e-05 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.665459+0000", 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0378e-05 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.665465+0000", 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1449999999999996e-06 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.671267+0000", 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058018590000000004 2024-09-13T15:10:57.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.671289+0000", 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2708e-05 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.692721+0000", 2024-09-13T15:10:57.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021431802 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:56.692817+0000", 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6074000000000002e-05 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125908 10.7 e466/448 10:e0fbea1e:::benchmark_data_smithi154_317818_object232:head v 466'7768, mlcod=466'7766)", 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:10:57.624 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.364849704, 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041463458000000002, 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192637+0000", 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9710000000000004e-06 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192654+0000", 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7268000000000001e-05 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192661+0000", 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3760000000000002e-06 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192720+0000", 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9172000000000002e-05 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192748+0000", 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7070999999999999e-05 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.233921+0000", 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041173123999999998 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234093+0000", 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000172299 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125909 10.f e466/448 10:f830a73c:::benchmark_data_smithi154_317818_object233:head v 466'7757, mlcod=466'7755)", 2024-09-13T15:10:57.629 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.193728+0000", 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.36375164599999998, 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041072277999999997, 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193728+0000", 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193728+0000", 2024-09-13T15:10:57.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193727+0000", 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193734+0000", 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5560000000000002e-06 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193749+0000", 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193757+0000", 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5850000000000002e-06 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.196627+0000", 2024-09-13T15:10:57.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028706080000000002 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.196682+0000", 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4557e-05 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234098+0000", 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037416505000000003 2024-09-13T15:10:57.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234800+0000", 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070148499999999996 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125912 10.f e466/448 10:f29520b2:::benchmark_data_smithi154_317818_object236:head v 466'7758, mlcod=466'7755)", 2024-09-13T15:10:57.634 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.197918+0000", 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.35956106300000001, 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037050997000000002, 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197918+0000", 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197918+0000", 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197917+0000", 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197926+0000", 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2459999999999992e-06 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197943+0000", 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6891000000000001e-05 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197951+0000", 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4290000000000004e-06 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.200598+0000", 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026478259999999998 2024-09-13T15:10:57.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.200614+0000", 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5959000000000001e-05 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234804+0000", 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034189123000000002 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234969+0000", 2024-09-13T15:10:57.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016555300000000001 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125914 10.f e466/448 10:fd8544f6:::benchmark_data_smithi154_317818_object238:head v 466'7759, mlcod=466'7755)", 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.201166+0000", 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.35631274499999999, 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033953497999999999, 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.201166+0000", 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.201166+0000", 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.201165+0000", 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.201176+0000", 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.094e-05 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.201189+0000", 2024-09-13T15:10:57.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2512e-05 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.201197+0000", 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5259999999999993e-06 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.204405+0000", 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032081660000000001 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.204463+0000", 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7904e-05 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234975+0000", 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030512096999999998 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.235120+0000", 2024-09-13T15:10:57.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000144704 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125923 10.7 e466/448 10:e74e1317:::benchmark_data_smithi154_317818_object247:head v 466'7769, mlcod=466'7766)", 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.205698+0000", 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.35178151400000002, 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032959362999999998, 2024-09-13T15:10:57.644 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.205698+0000", 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.205698+0000", 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.205696+0000", 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.205708+0000", 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1596e-05 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.205727+0000", 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9029999999999999e-05 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.205736+0000", 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0440000000000001e-06 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.206596+0000", 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085965599999999998 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.206652+0000", 2024-09-13T15:10:57.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6323000000000003e-05 2024-09-13T15:10:57.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.238468+0000", 2024-09-13T15:10:57.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031816252000000003 2024-09-13T15:10:57.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.238657+0000", 2024-09-13T15:10:57.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018882299999999999 2024-09-13T15:10:57.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.650 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.651 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125924 10.7 e466/448 10:e15db355:::benchmark_data_smithi154_317818_object248:head v 466'7770, mlcod=466'7766)", 2024-09-13T15:10:57.651 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.209077+0000", 2024-09-13T15:10:57.651 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.34840202300000001, 2024-09-13T15:10:57.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029749464, 2024-09-13T15:10:57.651 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.651 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.651 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.209077+0000", 2024-09-13T15:10:57.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.209077+0000", 2024-09-13T15:10:57.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.209076+0000", 2024-09-13T15:10:57.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.209084+0000", 2024-09-13T15:10:57.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3069999999999997e-06 2024-09-13T15:10:57.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.209105+0000", 2024-09-13T15:10:57.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1129999999999999e-05 2024-09-13T15:10:57.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.209113+0000", 2024-09-13T15:10:57.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5380000000000002e-06 2024-09-13T15:10:57.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.217366+0000", 2024-09-13T15:10:57.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082523049999999997 2024-09-13T15:10:57.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.217422+0000", 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6464e-05 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.238663+0000", 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021240541000000002 2024-09-13T15:10:57.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.238827+0000", 2024-09-13T15:10:57.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016408800000000001 2024-09-13T15:10:57.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.659 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.660 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125925 10.7 e466/448 10:e9810260:::benchmark_data_smithi154_317818_object249:head v 466'7771, mlcod=466'7766)", 2024-09-13T15:10:57.660 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.211990+0000", 2024-09-13T15:10:57.660 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.34548886499999998, 2024-09-13T15:10:57.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026987292, 2024-09-13T15:10:57.660 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.661 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.211990+0000", 2024-09-13T15:10:57.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.211990+0000", 2024-09-13T15:10:57.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.211989+0000", 2024-09-13T15:10:57.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.212009+0000", 2024-09-13T15:10:57.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9816000000000001e-05 2024-09-13T15:10:57.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.212024+0000", 2024-09-13T15:10:57.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5102e-05 2024-09-13T15:10:57.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.212032+0000", 2024-09-13T15:10:57.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6830000000000008e-06 2024-09-13T15:10:57.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.229652+0000", 2024-09-13T15:10:57.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017619645999999999 2024-09-13T15:10:57.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.229722+0000", 2024-09-13T15:10:57.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0495999999999997e-05 2024-09-13T15:10:57.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.238830+0000", 2024-09-13T15:10:57.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091076449999999993 2024-09-13T15:10:57.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.238978+0000", 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014790699999999999 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125959 10.7 e466/448 10:e63ef408:::benchmark_data_smithi154_317818_object283:head v 466'7774, mlcod=466'7771)", 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.250023+0000", 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30745636100000001, 2024-09-13T15:10:57.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035175267000000003, 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.250023+0000", 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.250023+0000", 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.250022+0000", 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.250030+0000", 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7130000000000005e-06 2024-09-13T15:10:57.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.250046+0000", 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6316000000000001e-05 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.250052+0000", 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4339999999999997e-06 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.253061+0000", 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030084230000000001 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.254079+0000", 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001018562 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.285063+0000", 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030983982 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.285198+0000", 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013479599999999999 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:126045 10.a e466/448 10:576955fd:::benchmark_data_smithi154_317818_object369:head v 466'7886, mlcod=466'7883)", 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.364165+0000", 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.193313713, 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028900624999999999, 2024-09-13T15:10:57.674 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364165+0000", 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364165+0000", 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364164+0000", 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364175+0000", 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1239e-05 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364194+0000", 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8101000000000002e-05 2024-09-13T15:10:57.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364202+0000", 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0600000000000008e-06 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364413+0000", 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021121099999999999 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.364456+0000", 2024-09-13T15:10:57.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3643999999999998e-05 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.392917+0000", 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028460642000000001 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.393066+0000", 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148997 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:126089 10.7 e466/448 10:eae6b2e2:::benchmark_data_smithi154_317818_object413:head v 466'7780, mlcod=466'7778)", 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.413254+0000", 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.14422468399999999, 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028650347999999999, 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.413254+0000", 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.413254+0000", 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.413254+0000", 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.413258+0000", 2024-09-13T15:10:57.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2049999999999998e-06 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.413263+0000", 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2989999999999996e-06 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.413267+0000", 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0860000000000004e-06 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.417600+0000", 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043335550000000002 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.417648+0000", 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7571999999999998e-05 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.441778+0000", 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024129757000000002 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:10:57.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.441905+0000", 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012705400000000001 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:10:57.683 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:10:57.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:10:57.730 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:57.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:10:57.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:57.866 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:57.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:57.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:58.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:58.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:58.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:58.151 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:58.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:58.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:58.294 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:58.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:58.396 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:58.428 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:58.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:58.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:58.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:58.472 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:10:58.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:10:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:58.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:58.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:10:58.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:10:58.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:58.777 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:58.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:58.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:58.929 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:58.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:10:59.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:10:59.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:59.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:10:59.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:59.203 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:59.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:59.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:10:59.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:59.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:59.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:59.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:10:59.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:10:59 smithi172 ceph-mon[100696]: pgmap v1486: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 97 B/s rd, 63 MiB/s wr, 1.09k op/s 2024-09-13T15:10:59.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:59.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:10:59.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:10:59.634 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:59.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:10:59.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:10:59 smithi154 ceph-mon[102314]: pgmap v1486: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 97 B/s rd, 63 MiB/s wr, 1.09k op/s 2024-09-13T15:10:59.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:10:59 smithi154 ceph-mon[107038]: pgmap v1486: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 97 B/s rd, 63 MiB/s wr, 1.09k op/s 2024-09-13T15:10:59.727 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:10:59.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:10:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:59.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:10:59.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:10:59.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:10:59.899 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:10:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:10:59.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:00.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:00.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:00.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:00.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:00.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:00.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:00.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:00.335 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:00.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:00.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:00.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:00.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:00.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:00.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:00.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:00.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:00.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:00.779 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:00.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:00.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:00.911 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:00.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:00.933 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:01.013 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:01.047 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:01.090 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:11:01.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:01.179 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:01.220 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3582124236' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:01.282 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:01.339 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:01.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:01.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3582124236' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:01.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3582124236' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:01.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.603 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:01.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-13T15:11:01.703 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:01.729 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:01.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.860 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:01.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:01.997 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:01.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:02.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:02.153 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:02.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:02.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:11:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:11:01] "GET /metrics HTTP/1.1" 200 36640 "" "Prometheus/2.51.0" 2024-09-13T15:11:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3582124236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:11:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[102314]: osdmap e467: 8 total, 8 up, 8 in 2024-09-13T15:11:02.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[102314]: pgmap v1488: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 49 MiB/s wr, 918 op/s 2024-09-13T15:11:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:02.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3582124236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:11:02.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[107038]: osdmap e467: 8 total, 8 up, 8 in 2024-09-13T15:11:02.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:02 smithi154 ceph-mon[107038]: pgmap v1488: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 49 MiB/s wr, 918 op/s 2024-09-13T15:11:02.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:02.304 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:02.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:02.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:02.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:02.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:02 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3582124236' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:11:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:02 smithi172 ceph-mon[100696]: osdmap e467: 8 total, 8 up, 8 in 2024-09-13T15:11:02.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:02 smithi172 ceph-mon[100696]: pgmap v1488: 99 pgs: 99 active+clean; 7.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 49 MiB/s wr, 918 op/s 2024-09-13T15:11:02.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:02.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:02.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:02.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:02.745 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:02.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:02.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:02.887 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:02.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:02.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:02.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:03.018 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:03.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:03.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:03.145 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:03.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:03.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:03.286 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:03.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:03.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:03.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:03.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:03.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:03.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:03 smithi172 ceph-mon[100696]: pgmap v1489: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 40 MiB/s wr, 910 op/s 2024-09-13T15:11:03.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:03.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:03.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:03.634 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:03 smithi154 ceph-mon[102314]: pgmap v1489: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 40 MiB/s wr, 910 op/s 2024-09-13T15:11:03.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:03 smithi154 ceph-mon[107038]: pgmap v1489: 99 pgs: 99 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 102 B/s rd, 40 MiB/s wr, 910 op/s 2024-09-13T15:11:03.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:03.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:03.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:03.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:03.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:03.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:03.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:03.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:04.002 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:04.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:11:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:11:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:131887 10.c 10:380c7b83:::benchmark_data_smithi154_317818_object6211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-09-13T15:11:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:04.020582+0000", 2024-09-13T15:11:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.016992031000000001, 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017024938, 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 131887 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:11:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020582+0000", 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020582+0000", 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020579+0000", 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020588+0000", 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1990000000000006e-06 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020605+0000", 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7787000000000001e-05 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020613+0000", 2024-09-13T15:11:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.661e-06 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020650+0000", 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6729e-05 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020755+0000", 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010497699999999999 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:11:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.020827+0000", 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2356999999999999e-05 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.024984+0000", 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041566179999999999 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.026692+0000", 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017084649999999999 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:11:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:131896 10.8 e467/448 10:19c9a497:::benchmark_data_smithi154_317818_object6220:head v 467'8314, mlcod=467'8311)", 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:04.034696+0000", 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.0028774790000000001, 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029946460000000001, 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:11:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034696+0000", 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034696+0000", 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034695+0000", 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034702+0000", 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5440000000000001e-06 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034716+0000", 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3611000000000001e-05 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034723+0000", 2024-09-13T15:11:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8240000000000002e-06 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034767+0000", 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3713e-05 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:04.034787+0000", 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0632000000000002e-05 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:11:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:11:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:11:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-09-13T15:11:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 2 2024-09-13T15:11:04.049 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:11:04.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:04.139 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:04.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:04.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:04.274 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:04.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:04.305 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:04.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:11:04.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:04.396 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:11:04.396 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:11:04.430 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:11:04.430 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:11:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:04.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:04.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:04.467 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:11:04.468 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:11:04.508 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:11:04.508 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:11:04.542 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:11:04.542 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:11:04.543 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:04 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2748186641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:04.543 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:04 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2748186641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:04.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:04.580 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:11:04.580 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:11:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:04.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:04.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:04.609 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:11:04.610 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:11:04.640 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:11:04.641 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:11:04.689 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:04.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:04 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2748186641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:04.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:04.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:04.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:04.861 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:04.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:04.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:04.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:05.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:11:05.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:05.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:05.233 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:05.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:05.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:05.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:05.415 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:11:05.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:05.557 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:05.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:05.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:05.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:05.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:05.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:05.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:05.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:06.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:06.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:06.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:06.229 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:06.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:06.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:06.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:06.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:06.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:06.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:06 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:06.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:06 smithi154 ceph-mon[102314]: pgmap v1490: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 891 op/s 2024-09-13T15:11:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:06 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:06 smithi154 ceph-mon[107038]: pgmap v1490: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 891 op/s 2024-09-13T15:11:06.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:06 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:06 smithi172 ceph-mon[100696]: pgmap v1490: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 891 op/s 2024-09-13T15:11:06.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:06.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:06.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:06.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:06.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:06.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:06.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:06.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:06.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:06.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:06.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:07.025 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:07.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:07.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:07.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:07.195 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:07.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:07.265 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:11:07.265 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:11:07.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:07.330 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:07.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:07.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:07.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:07 smithi154 ceph-mon[102314]: pgmap v1491: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 12 MiB/s wr, 882 op/s 2024-09-13T15:11:07.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:07 smithi154 ceph-mon[107038]: pgmap v1491: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 12 MiB/s wr, 882 op/s 2024-09-13T15:11:07.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:07.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:07.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:07.612 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:07 smithi172 ceph-mon[100696]: pgmap v1491: 99 pgs: 99 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 12 MiB/s wr, 882 op/s 2024-09-13T15:11:07.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:07.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:07.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:07.778 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:07.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:07.879 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:07.912 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:07.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:07.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:11:08.014 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:08.068 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:08.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:08.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:08.221 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:08.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:08.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:08.330 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:11:08.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:08.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:08.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:08.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:08.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:08.605 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:08.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:08.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:08.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:08.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:08.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:08.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:08.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:08.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:09.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:09.075 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:09.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:09.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:09.205 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:09.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:09.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:09.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:09.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:09.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:09.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:09.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:09 smithi154 ceph-mon[102314]: pgmap v1492: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2.8 MiB/s wr, 868 op/s 2024-09-13T15:11:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:09 smithi154 ceph-mon[107038]: pgmap v1492: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2.8 MiB/s wr, 868 op/s 2024-09-13T15:11:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:09.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:09.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:09.610 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:09.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:09.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:09 smithi172 ceph-mon[100696]: pgmap v1492: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2.8 MiB/s wr, 868 op/s 2024-09-13T15:11:09.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:09.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:09.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:09.877 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:09.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:09.978 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:09.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:10.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:10.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:10.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:10.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:10.272 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:10.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:10.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:10.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-13T15:11:10.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:11:10.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:10.545 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:10.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:10.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:10.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:10.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:10.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:10.992 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:10.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:11.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:11.135 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:11.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:11.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:11.273 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:11.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:11.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:11.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:11.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:11.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:11.555 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:11.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:11.609 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:11 smithi172 ceph-mon[100696]: pgmap v1493: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 576 KiB/s wr, 855 op/s 2024-09-13T15:11:11.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:11.661 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:11:11.661 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:11:11.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:11.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:11.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:11 smithi154 ceph-mon[102314]: pgmap v1493: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 576 KiB/s wr, 855 op/s 2024-09-13T15:11:11.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:11 smithi154 ceph-mon[107038]: pgmap v1493: 99 pgs: 99 active+clean; 7.1 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 576 KiB/s wr, 855 op/s 2024-09-13T15:11:11.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:11.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:11.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:11.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:11.948 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:11.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:12.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:12.075 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:12.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:12.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:12.198 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:12.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:12.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:11:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:11:11] "GET /metrics HTTP/1.1" 200 36640 "" "Prometheus/2.51.0" 2024-09-13T15:11:12.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:12.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:11:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:12.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:12.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:12.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:12.466 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:12.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:12.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:12.602 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:12.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:12.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:12.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:12.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:12.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:12.891 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:12.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:12.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:13.024 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:13.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:13.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:13.162 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:13.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:13.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:13.297 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:13.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:13.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:13.346 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:13.422 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:13.424 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:13.443 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:13.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:13.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:13.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:13.610 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:13.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:13.664 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:13.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:13.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:13.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:13.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:13.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:13.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:13.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:14.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:14.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:14.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:14.178 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:14.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:14.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:14 smithi154 ceph-mon[107038]: pgmap v1494: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 485 KiB/s wr, 842 op/s 2024-09-13T15:11:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:14 smithi154 ceph-mon[102314]: pgmap v1494: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 485 KiB/s wr, 842 op/s 2024-09-13T15:11:14.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:14.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:14.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:14.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:14 smithi172 ceph-mon[100696]: pgmap v1494: 99 pgs: 99 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 485 KiB/s wr, 842 op/s 2024-09-13T15:11:14.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:14.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:14.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:14.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:14.608 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:14.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:14.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:14.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:14.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:14.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:14.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:14.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:14.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:15.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:15.040 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:15.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:15.174 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:15.239 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4115331020' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:15.239 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:15 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:15.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:15.311 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:15.413 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:15.439 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4115331020' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:15.439 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:15 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:15.439 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4115331020' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:15.439 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:15 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:15.446 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:15.492 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:11:15.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:15.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:15.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:15.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:15.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:15.860 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:15.962 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:15.993 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:15.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:16.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:16.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:16.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:16.232 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:16.265 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:16.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:16.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:16.384 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:16 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:11:16.385 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:16 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:11:16.385 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:16 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:11:16.385 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:16 smithi172 ceph-mon[100696]: osdmap e468: 8 total, 8 up, 8 in 2024-09-13T15:11:16.385 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:16 smithi172 ceph-mon[100696]: pgmap v1496: 99 pgs: 99 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 866 op/s 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[102314]: osdmap e468: 8 total, 8 up, 8 in 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[102314]: pgmap v1496: 99 pgs: 99 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 866 op/s 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:11:16.393 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[107038]: osdmap e468: 8 total, 8 up, 8 in 2024-09-13T15:11:16.394 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:16 smithi154 ceph-mon[107038]: pgmap v1496: 99 pgs: 99 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 866 op/s 2024-09-13T15:11:16.398 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:16.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:16.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:16.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:16.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:16.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:16.674 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:16.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:16.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:16.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:16.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:16.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:16.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:16.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:17.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:17.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:17.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:17.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:17.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:17.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:17.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:17.355 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:17.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:17.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:17.500 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:17.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:17.601 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:17.637 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:17.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:17.652 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:17 smithi154 ceph-mon[102314]: pgmap v1497: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-13T15:11:17.652 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:17 smithi154 ceph-mon[107038]: pgmap v1497: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-13T15:11:17.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:17.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:17 smithi172 ceph-mon[100696]: pgmap v1497: 99 pgs: 99 active+clean; 6.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 869 op/s 2024-09-13T15:11:17.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:17.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:17.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:17.916 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:17.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:18.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:18.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:18.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:18.161 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:18.196 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:18.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:18.297 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:18.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:11:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:18.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:18.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:18.439 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:18.473 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:18.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:18.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:18.613 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:18.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:18.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:18.750 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:18.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:18.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:18.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:18.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:18.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:19.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:19.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:19.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:19.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:19.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:19.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:19.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:19.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:19.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:19.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:19.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:19.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:19.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:19.585 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:19.597 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:19 smithi154 ceph-mon[102314]: pgmap v1498: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-09-13T15:11:19.597 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:19 smithi154 ceph-mon[107038]: pgmap v1498: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-09-13T15:11:19.607 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:19.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:19.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:19.741 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:19.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:19.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:19 smithi172 ceph-mon[100696]: pgmap v1498: 99 pgs: 99 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-09-13T15:11:19.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:19.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:19.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:20.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:20.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:20.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:20.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:20.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:20.201 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:20.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:20.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:20.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:20.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:20.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:11:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:11:20.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:11:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:11:20.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:20.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:20.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:20.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:20.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:20.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:11:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:11:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:11:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:20.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:20.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:20.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:20.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:20.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:21.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:21.025 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:21.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:21.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-13T15:11:21.114 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:11:21.115 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:11:21.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:21.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:21.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:21.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:21.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:21.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:21.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:21.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:21.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:21.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:21.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:21.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:21.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:21.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:21 smithi154 ceph-mon[102314]: pgmap v1499: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:11:21.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:21 smithi154 ceph-mon[107038]: pgmap v1499: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:11:21.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:21 smithi172 ceph-mon[100696]: pgmap v1499: 99 pgs: 99 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:11:21.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:21.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:21.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:21.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:21.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:21.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:22.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:22.127 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:22.142 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:11:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:11:21] "GET /metrics HTTP/1.1" 200 36644 "" "Prometheus/2.51.0" 2024-09-13T15:11:22.156 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:22.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:22.289 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:22.316 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:22.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:11:22.391 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:22.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:22.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:22.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:22.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:22.699 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:22.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:22.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:22.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:22.938 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:22.965 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:22.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:23.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:23.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:23.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:23.195 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:11:23.196 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:11:23.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:23.231 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:23.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:23.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:23.234 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:11:23.234 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:11:23.260 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:23 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/896025680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:23.260 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:23 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/896025680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:23.266 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:11:23.266 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:11:23.314 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:11:23.315 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:11:23.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:23.351 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:11:23.351 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:11:23.358 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:23.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:23.388 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:11:23.389 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:11:23.421 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:11:23.421 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:11:23.453 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:11:23.453 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:11:23.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:23.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:23 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/896025680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:23.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:23.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:23.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:23.627 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:23.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:23.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:23.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:23.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:23.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:23.901 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:23.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:24.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:24.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:24.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:24.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:24.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:24.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:24.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:24.314 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:24 smithi154 ceph-mon[102314]: pgmap v1500: 99 pgs: 99 active+clean; 6.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:11:24.314 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:24 smithi154 ceph-mon[107038]: pgmap v1500: 99 pgs: 99 active+clean; 6.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:11:24.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:24.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:24.421 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:24.450 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:24.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:24.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:24 smithi172 ceph-mon[100696]: pgmap v1500: 99 pgs: 99 active+clean; 6.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:11:24.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:24.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:24.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:24.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:24.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:24.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:24.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:24.840 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:24.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:24.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:25.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:25.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:25.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:25.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:25.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:25.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:25.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:25.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:25.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:25.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:25.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:25.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:25.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:25.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:25 smithi154 ceph-mon[102314]: pgmap v1501: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 877 op/s 2024-09-13T15:11:25.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:25 smithi154 ceph-mon[107038]: pgmap v1501: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 877 op/s 2024-09-13T15:11:25.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:25.589 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:25.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:25.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:25.631 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:25 smithi172 ceph-mon[100696]: pgmap v1501: 99 pgs: 99 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 877 op/s 2024-09-13T15:11:25.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:25.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:25.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:25.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:25.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:25.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:25.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.004 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:26.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:26.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:26.138 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:26.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:26.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:26.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:26.315 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:11:26.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:11:26.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:26.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:26.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:26.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:26.573 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:26.669 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3697373550' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:11:26.669 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:26 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:11:26.669 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3697373550' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:11:26.670 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:26 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:11:26.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:26.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3697373550' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:11:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:26 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:11:26.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:26.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:26.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:26.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:26.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:27.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:27.128 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:27.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:27.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:27.271 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:27.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:27.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:27.412 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:27.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:27.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:11:27.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[102314]: osdmap e469: 8 total, 8 up, 8 in 2024-09-13T15:11:27.472 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[102314]: pgmap v1503: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-09-13T15:11:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:11:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[107038]: osdmap e469: 8 total, 8 up, 8 in 2024-09-13T15:11:27.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:27 smithi154 ceph-mon[107038]: pgmap v1503: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-09-13T15:11:27.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:27.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:27.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:27.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:27.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:27.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:27 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:27 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:11:27.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:27 smithi172 ceph-mon[100696]: osdmap e469: 8 total, 8 up, 8 in 2024-09-13T15:11:27.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:27 smithi172 ceph-mon[100696]: pgmap v1503: 99 pgs: 99 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-09-13T15:11:27.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:27.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:27.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:27.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:27.971 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:27.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:28.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:28.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:28.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:28.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:28.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:28.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:28.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:28.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:28.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:28.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:28.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:28.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:28.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:28.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:28.641 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:28.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:28.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:28.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:28.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:28.772 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:28.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:28.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:28.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:28.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:29.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:29.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:29.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:29.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:29.213 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:29.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:29.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:29.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:29.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:29.456 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:29.490 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:29.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:29.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:29.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:29.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:29.698 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:29 smithi154 ceph-mon[102314]: pgmap v1504: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-13T15:11:29.698 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:29 smithi154 ceph-mon[107038]: pgmap v1504: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-13T15:11:29.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:29 smithi172 ceph-mon[100696]: pgmap v1504: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-13T15:11:29.752 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:29.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:29.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:29.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:29.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:29.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:30.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:30.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:30.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:30.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:30.194 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:30.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:30.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:30.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:30.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:30.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:30.464 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:30.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:30.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:30.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:30.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:30.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:30.734 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:30.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:30.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:30.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:30.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:30.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:30.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:11:31.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:31.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:31.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:31.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:31.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:31.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:31.305 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:31.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:31.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:31.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:31.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:31.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:31 smithi154 ceph-mon[102314]: pgmap v1505: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-13T15:11:31.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:31 smithi154 ceph-mon[107038]: pgmap v1505: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-13T15:11:31.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:31.565 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:31.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:31.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:31.713 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:11:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:11:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:11:31.713 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:11:31.713 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:11:31.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:31.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:31 smithi172 ceph-mon[100696]: pgmap v1505: 99 pgs: 99 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-13T15:11:31.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:31.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:31.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:31.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:32.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:32.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:32.130 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:11:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:11:31] "GET /metrics HTTP/1.1" 200 36649 "" "Prometheus/2.51.0" 2024-09-13T15:11:32.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:32.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:32.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:32.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:32.417 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:32.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:32.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:32.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:32.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:32.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:32.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:32.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:32.968 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:32.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:33.070 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:33.106 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:33.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:33.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:33.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:33.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:33.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:33.377 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:33.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:33.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:33.508 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:33.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:33.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:33.621 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:33 smithi154 ceph-mon[107038]: pgmap v1506: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 909 op/s 2024-09-13T15:11:33.621 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:33 smithi154 ceph-mon[102314]: pgmap v1506: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 909 op/s 2024-09-13T15:11:33.652 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:33.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:33.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:11:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:33 smithi172 ceph-mon[100696]: pgmap v1506: 99 pgs: 99 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 909 op/s 2024-09-13T15:11:33.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:33.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:33.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:33.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:33.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:33.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:33.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:34.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:34.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:34.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:34.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:34.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:34.290 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:34.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:34.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:34.451 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:34.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:34.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:34.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:34.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:34.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:34.848 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:34.856 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:34.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:34.989 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:34.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:35.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:35.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:35.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:35.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:35.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:35.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:35.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:35.382 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:35.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:35.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:35.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:35.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:35.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:35 smithi154 ceph-mon[102314]: pgmap v1507: 99 pgs: 99 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-13T15:11:35.561 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:35 smithi154 ceph-mon[107038]: pgmap v1507: 99 pgs: 99 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-13T15:11:35.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:35.656 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:35.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:35 smithi172 ceph-mon[100696]: pgmap v1507: 99 pgs: 99 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 908 op/s 2024-09-13T15:11:35.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:35.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:35.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:35.870 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:35.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:35.924 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:35.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:36.026 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:36.064 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:36.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:36.166 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:36.199 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:36.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:36.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:36.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:36.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:36.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:36.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:36.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:36.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:36.620 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:36.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:36.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3537021916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:36.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3537021916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:36.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3537021916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:36.763 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:36.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:36.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:36.896 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:36.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:36.937 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:11:36.937 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:11:36.937 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:11:36.937 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:11:36.938 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:11:36.939 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:11:36.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:37.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:37.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:37.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:37.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:37.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:37.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:37.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:37.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:37.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:37.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:37.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:37 smithi154 ceph-mon[102314]: pgmap v1508: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 901 op/s 2024-09-13T15:11:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:37 smithi154 ceph-mon[107038]: pgmap v1508: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 901 op/s 2024-09-13T15:11:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:37 smithi172 ceph-mon[100696]: pgmap v1508: 99 pgs: 99 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 901 op/s 2024-09-13T15:11:37.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:37.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:37.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:37.983 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:37.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:38.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:38.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:38.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:38.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:38.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:38.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:38.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:38.397 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:38.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:38.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:38.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:38.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:38.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:38.672 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:38.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:38.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:38.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:38.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:38.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:38.939 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:38.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:39.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:39.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:39.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:39.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:39.220 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:39.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:39.228 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:11:39.228 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:11:39.228 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-13T15:11:39.228 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:11:39.228 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:11:39.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-13T15:11:39.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:39.349 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:39.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:39.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:39.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:39 smithi154 ceph-mon[107038]: pgmap v1509: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 929 op/s 2024-09-13T15:11:39.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:39 smithi154 ceph-mon[102314]: pgmap v1509: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 929 op/s 2024-09-13T15:11:39.479 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:39.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:39.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:39.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:39.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:39.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:39.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:39.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:39 smithi172 ceph-mon[100696]: pgmap v1509: 99 pgs: 99 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 929 op/s 2024-09-13T15:11:39.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:39.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:39.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:39.967 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.003 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:40.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:40.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:40.132 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:40.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:40.267 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:40.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:40.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:40.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.548 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:40.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:40.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:40.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:40.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.814 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:40.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:40.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:40.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:40.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:41.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:41.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:41.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:41.194 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:41.230 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:41.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:41.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:41.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:41.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:41.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:41.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:41 smithi154 ceph-mon[107038]: pgmap v1510: 99 pgs: 99 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 811 op/s 2024-09-13T15:11:41.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:41 smithi154 ceph-mon[102314]: pgmap v1510: 99 pgs: 99 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 811 op/s 2024-09-13T15:11:41.492 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:41.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:41.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:41.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:41.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:41.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:41.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:41 smithi172 ceph-mon[100696]: pgmap v1510: 99 pgs: 99 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 811 op/s 2024-09-13T15:11:41.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:41.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:41.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:41.904 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:41.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:42.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:42.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:42.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:42.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:42.149 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:11:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:11:41] "GET /metrics HTTP/1.1" 200 36649 "" "Prometheus/2.51.0" 2024-09-13T15:11:42.170 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:42.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:42.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:42.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:42.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:42.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:42.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:42.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:42.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:42.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:42.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:42.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:42.748 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:42.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:42.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:42.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:42.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:42.926 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:42.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:43.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:43.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:43.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:43.201 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:43.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:43.219 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:11:43.219 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:11:43.303 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:43.303 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:43.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:43.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:43.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:43.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:43.477 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:43.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:43.580 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:43.614 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:43.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:43.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:43.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:11:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:43.751 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:43.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:43.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:43.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:43.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:43.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:44.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:44.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:44.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:44.086 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:11:44.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:44.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:44.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:44.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:44.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:44.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:44 smithi172 ceph-mon[100696]: pgmap v1511: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 856 op/s 2024-09-13T15:11:44.392 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4063929344' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:44.401 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:44 smithi154 ceph-mon[102314]: pgmap v1511: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 856 op/s 2024-09-13T15:11:44.401 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4063929344' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:44.402 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:44 smithi154 ceph-mon[107038]: pgmap v1511: 99 pgs: 99 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 856 op/s 2024-09-13T15:11:44.403 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4063929344' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:11:44.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:44.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:44.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:44.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:44.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:44.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:44.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:44.737 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:44.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:44.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:44.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:44.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:44.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:45.007 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:45.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:45.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:45.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:45.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:45.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:45.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:45 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:45.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:45 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4063929344' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:11:45.306 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:45 smithi154 ceph-mon[102314]: osdmap e470: 8 total, 8 up, 8 in 2024-09-13T15:11:45.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:45 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:45.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:45 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4063929344' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:11:45.307 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:45 smithi154 ceph-mon[107038]: osdmap e470: 8 total, 8 up, 8 in 2024-09-13T15:11:45.386 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:45.424 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:45 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:45 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4063929344' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:11:45.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:45 smithi172 ceph-mon[100696]: osdmap e470: 8 total, 8 up, 8 in 2024-09-13T15:11:45.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:45.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:45.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:45.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:45.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:45.822 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:45.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:11:45.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:45.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:45.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:46.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:46.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:46.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:46.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:46.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:46.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:46.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:46.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:46.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:46.425 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:46 smithi154 ceph-mon[102314]: pgmap v1513: 99 pgs: 99 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-09-13T15:11:46.426 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:46 smithi154 ceph-mon[107038]: pgmap v1513: 99 pgs: 99 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-09-13T15:11:46.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:46 smithi172 ceph-mon[100696]: pgmap v1513: 99 pgs: 99 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 886 op/s 2024-09-13T15:11:46.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:46.509 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:46.528 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:46.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:46.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:46.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:46.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:46.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:46.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:46.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:46.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:46.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:46.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:47.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:47.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:11:47.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:47.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:47.193 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:47.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:47.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:47.354 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:11:47.354 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T15:11:47.354 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:11:47.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:47.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:47.556 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:47 smithi154 ceph-mon[107038]: pgmap v1514: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 893 op/s 2024-09-13T15:11:47.557 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:47 smithi154 ceph-mon[102314]: pgmap v1514: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 893 op/s 2024-09-13T15:11:47.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:47.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:47.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:47.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:47.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:47.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:47 smithi172 ceph-mon[100696]: pgmap v1514: 99 pgs: 99 active+clean; 5.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 893 op/s 2024-09-13T15:11:47.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:47.856 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:47.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:47.987 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:47.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:48.045 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:48.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:48.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:48.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:48.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:48.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:48.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:48.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:48.397 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:48.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:48.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:48.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:48.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:48.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:48.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:48.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:48.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:48.830 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:48.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:48.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:48.975 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:48.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:49.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:49.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:49.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:49.150 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:49.220 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:49.253 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:49.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:49.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:49.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:49.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:49.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:49 smithi154 ceph-mon[102314]: pgmap v1515: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 890 op/s 2024-09-13T15:11:49.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:49 smithi154 ceph-mon[107038]: pgmap v1515: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 890 op/s 2024-09-13T15:11:49.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:49.530 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:49.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:49.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:49.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:49.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:49 smithi172 ceph-mon[100696]: pgmap v1515: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 890 op/s 2024-09-13T15:11:49.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:49.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:49.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:49.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:49.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:49.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:50.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:50.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:50.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:50.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:50.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:50.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:50.334 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:50.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:50.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:50.424 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:11:50.424 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:11:50.459 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:11:50.460 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:11:50.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:50.491 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:50.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:50.501 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:11:50.501 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:11:50.543 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:11:50.544 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:11:50.576 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:11:50.577 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:11:50.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:50.609 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:11:50.609 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:11:50.618 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:50.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:50.643 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:11:50.643 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:11:50.643 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1098223058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:50.644 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1098223058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:50.678 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:11:50.679 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:11:50.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:50.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1098223058' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:11:50.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:50.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:50.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:50.898 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:50.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:51.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:51.031 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:51.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:51.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:51.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:51.306 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:51.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:51.443 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:51.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:51.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:51.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:51.715 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:51.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:51 smithi154 ceph-mon[102314]: pgmap v1516: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-13T15:11:51.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:51 smithi154 ceph-mon[107038]: pgmap v1516: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-13T15:11:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:51 smithi172 ceph-mon[100696]: pgmap v1516: 99 pgs: 99 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-13T15:11:51.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:51.851 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:51.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:51.988 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:51.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:52.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:52.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:52.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:52.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:11:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:11:51] "GET /metrics HTTP/1.1" 200 36648 "" "Prometheus/2.51.0" 2024-09-13T15:11:52.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:52.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:52.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:52.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:52.386 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:52.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:52.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:52.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:52.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:52.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:52.665 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:52.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:52.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:52.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:52.802 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:52.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:52.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:52.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:52.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:52.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:53.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:53.091 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:53.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:53.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:53.229 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:53.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:53.331 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:53.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:53.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:53.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:53.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:53 smithi154 ceph-mon[102314]: pgmap v1517: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-09-13T15:11:53.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:53 smithi154 ceph-mon[107038]: pgmap v1517: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-09-13T15:11:53.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:53.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:53.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:53.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:53.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:53.744 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:53 smithi172 ceph-mon[100696]: pgmap v1517: 99 pgs: 99 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s 2024-09-13T15:11:53.749 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:53.778 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:53.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:53.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:53.905 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:53.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:53.970 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:11:53.970 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:11:53.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.970 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.36270.0:176155 10.8 e470/448 10:110b6c44:::benchmark_data_smithi154_317818_object50479:head v 470'11075, mlcod=470'11073)", 2024-09-13T15:11:53.970 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:11:53.964309+0000", 2024-09-13T15:11:53.970 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0042386790000000004, 2024-09-13T15:11:53.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042651599999999996, 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964309+0000", 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964309+0000", 2024-09-13T15:11:53.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964310+0000", 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1100000000000005e-07 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964316+0000", 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4110000000000001e-06 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964332+0000", 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6218e-05 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964340+0000", 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1640000000000001e-06 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964645+0000", 2024-09-13T15:11:53.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030463799999999998 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964664+0000", 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8428999999999999e-05 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.36270.0:176160 10.f e470/448 10:f14868e0:::benchmark_data_smithi154_317818_object50484:head v 470'10830, mlcod=470'10828)", 2024-09-13T15:11:53.974 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:11:53.964407+0000", 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0041400999999999999, 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042218749999999999, 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964407+0000", 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964407+0000", 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964408+0000", 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4999999999999998e-07 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964412+0000", 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2309999999999998e-06 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964419+0000", 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3220000000000004e-06 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964425+0000", 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2100000000000001e-06 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:11:53.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964594+0000", 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016937300000000001 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.964617+0000", 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3431999999999999e-05 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:11:53.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:11:53.968609+0000", 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039913129999999998 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 2 2024-09-13T15:11:53.979 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:11:54.007 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:54.035 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:54.063 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:11:54.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:11:54.063 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-13T15:11:54.064 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:11:54.064 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:11:54.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-13T15:11:54.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:54.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:54.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:54.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:54.332 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:11:54.356 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2964757109' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:54.356 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:54 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:54.356 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2964757109' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:54.356 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:54 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:54.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.448 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:54.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:54.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:54.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:54.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.725 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:54.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:54.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2964757109' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:54.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:54 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:11:54.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:11:54.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:54.855 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:54.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:11:54.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:54.985 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:54.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:55.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:55.115 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:55.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:55.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:55.243 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:55.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:55.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:55.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:55.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:55.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:55.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:55.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:55.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:55.661 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:55.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:55.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:55.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:11:55.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[102314]: osdmap e471: 8 total, 8 up, 8 in 2024-09-13T15:11:55.667 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[102314]: pgmap v1519: 99 pgs: 99 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-13T15:11:55.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:55.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:11:55.667 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[107038]: osdmap e471: 8 total, 8 up, 8 in 2024-09-13T15:11:55.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:55 smithi154 ceph-mon[107038]: pgmap v1519: 99 pgs: 99 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-13T15:11:55.715 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:55 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:11:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:55 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:11:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:55 smithi172 ceph-mon[100696]: osdmap e471: 8 total, 8 up, 8 in 2024-09-13T15:11:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:55 smithi172 ceph-mon[100696]: pgmap v1519: 99 pgs: 99 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-13T15:11:55.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:55.766 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:55.790 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:11:55.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:55.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:55.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:55.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:55.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:55.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:56.084 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:56.116 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:56.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:56.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:56.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:56.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:56.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:56.391 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:56.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:56.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:56.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:56.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:56.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:11:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:56.676 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:56.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:11:56.777 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:56.811 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:56.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:56.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:56.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:56.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:57.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:57.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:57.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:57.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:11:57.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:57.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:11:57.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:57.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:57.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:57.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:57.493 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:57.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:57.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:57.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:57.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:57.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:11:57.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:57 smithi154 ceph-mon[102314]: pgmap v1520: 99 pgs: 99 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-13T15:11:57.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:57 smithi154 ceph-mon[107038]: pgmap v1520: 99 pgs: 99 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-13T15:11:57.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:57 smithi172 ceph-mon[100696]: pgmap v1520: 99 pgs: 99 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 884 op/s 2024-09-13T15:11:57.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:57.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:57.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:57.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:57.936 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:57.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:58.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:58.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:58.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:58.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:58.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:58.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:58.323 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:58.357 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:58.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:58.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:11:58.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:58.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:58.485 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:58.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:11:58.588 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:58.622 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:58.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:58.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:11:58.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:58.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:58.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:11:58.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:58.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:11:59.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:59.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:59.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:59.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:59.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:59.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:59.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:59.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:59.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:59.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:11:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:59.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:59.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:11:59.535 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:11:59 smithi154 ceph-mon[102314]: pgmap v1521: 99 pgs: 99 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-09-13T15:11:59.535 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:11:59 smithi154 ceph-mon[107038]: pgmap v1521: 99 pgs: 99 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-09-13T15:11:59.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:11:59.589 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:59.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:11:59.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:11:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:59.723 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:11:59.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:11:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:11:59 smithi172 ceph-mon[100696]: pgmap v1521: 99 pgs: 99 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 895 op/s 2024-09-13T15:11:59.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:11:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:11:59.871 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:11:59.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:11:59.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:00.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:00.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:00.142 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:00.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:00.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:00.333 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:12:00.333 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:12:00.333 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:12:00.334 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:12:00.334 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:12:00.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:00.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:00.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:00.537 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:00.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:00.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:00.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:00.797 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.834 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:00.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:00.935 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:00.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:00.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:01.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:01.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:01.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:01.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:01.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:01.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:01.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:01.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:01.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:01.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:01.514 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:01 smithi154 ceph-mon[102314]: pgmap v1522: 99 pgs: 99 active+clean; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 902 op/s 2024-09-13T15:12:01.514 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:01 smithi154 ceph-mon[107038]: pgmap v1522: 99 pgs: 99 active+clean; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 902 op/s 2024-09-13T15:12:01.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:01.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:01.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:01.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:01.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:01 smithi172 ceph-mon[100696]: pgmap v1522: 99 pgs: 99 active+clean; 4.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 902 op/s 2024-09-13T15:12:01.768 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:12:01.768 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:12:01.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:01.805 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:12:01.806 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:12:01.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:01.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:01.840 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:12:01.840 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:12:01.874 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:12:01.874 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:12:01.910 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:12:01.910 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:12:01.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:01.946 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:12:01.946 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:12:01.956 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:01.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:01.992 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:12:01.992 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:12:02.008 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:12:02.009 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-13T15:12:02.009 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:12:02.019 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:12:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:12:01] "GET /metrics HTTP/1.1" 200 36649 "" "Prometheus/2.51.0" 2024-09-13T15:12:02.039 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:12:02.039 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:12:02.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:02.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:02.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:02.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:02.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:02.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:02.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:02.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:02.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:02.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:02.561 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:02.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:02.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:02.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:02.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3763926994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3763926994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3763926994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:02.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:02.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:02.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:02.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:02.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:02.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:03.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:03.108 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:03.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:03.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:03.247 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:03.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:03.349 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:03.385 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:03.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:03.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:03.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:03.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:03.544 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:03 smithi154 ceph-mon[102314]: pgmap v1523: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 909 op/s 2024-09-13T15:12:03.545 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:03 smithi154 ceph-mon[107038]: pgmap v1523: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 909 op/s 2024-09-13T15:12:03.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:03.669 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:03.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:03 smithi172 ceph-mon[100696]: pgmap v1523: 99 pgs: 99 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 909 op/s 2024-09-13T15:12:03.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:03.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:03.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:03.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:03.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:03.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:04.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:04.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:04.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:04.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:04.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:04.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:04.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:04.355 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:04.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:04.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:04.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:04.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:04.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:04.625 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:04.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:04.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:04.758 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:04.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:04.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:04.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:12:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:04.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:04.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:05.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:05.029 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:05.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:05.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:05.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:05.321 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:05.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:05.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:05.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:05.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:12:05.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:05.614 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:05 smithi154 ceph-mon[102314]: pgmap v1524: 99 pgs: 99 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T15:12:05.615 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:05 smithi154 ceph-mon[107038]: pgmap v1524: 99 pgs: 99 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T15:12:05.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.726 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:05.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:05 smithi172 ceph-mon[100696]: pgmap v1524: 99 pgs: 99 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 898 op/s 2024-09-13T15:12:05.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:05.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:05.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:05.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:05.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:06.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:06.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:06.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:06.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:06.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:06.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:06.355 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:06.394 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:06.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:06.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:06.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:06.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:06.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:06.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:06.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:06.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:06.797 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:06.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:06.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:06.936 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:06.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:07.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:07.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:07.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:07.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:07.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:07.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:07.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:07.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:07.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:07.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:07.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:07.608 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:07.640 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:07.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:07.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:07 smithi154 ceph-mon[102314]: pgmap v1525: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:12:07.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:07 smithi154 ceph-mon[107038]: pgmap v1525: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:12:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:07 smithi172 ceph-mon[100696]: pgmap v1525: 99 pgs: 99 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 861 op/s 2024-09-13T15:12:07.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:07.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:07.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:07.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:07.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:08.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:08.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:08.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:08.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:08.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:08.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:08.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:08.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:08.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:08.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:08.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:08.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:08.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:08.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:08.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:08.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:08.735 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:08.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:08.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:08.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:08.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:08.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.021 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:09.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:09.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:09.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:09.252 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:09.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:09.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:09.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:09.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:09.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:09.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.669 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:09.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:09.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:09 smithi154 ceph-mon[102314]: pgmap v1526: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-09-13T15:12:09.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:09 smithi154 ceph-mon[107038]: pgmap v1526: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-09-13T15:12:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:09 smithi172 ceph-mon[100696]: pgmap v1526: 99 pgs: 99 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 933 op/s 2024-09-13T15:12:09.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:09.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:09.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:09.937 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:09.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:10.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:10.065 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:10.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:10.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:10.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:10.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:10.306 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:10.336 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:10.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:10.437 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:10.469 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:10.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:10.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:10.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:10.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:10.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:10.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:10.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:10.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:10.885 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:10.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:10.986 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:11.016 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:11.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:11.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:11.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:11.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:11.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:11.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:11.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:11.393 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:11.424 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:11.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:11.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:11.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:11.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:11.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:11.685 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:11 smithi154 ceph-mon[102314]: pgmap v1527: 99 pgs: 99 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-13T15:12:11.685 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:11 smithi154 ceph-mon[107038]: pgmap v1527: 99 pgs: 99 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-13T15:12:11.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:11.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:11 smithi172 ceph-mon[100696]: pgmap v1527: 99 pgs: 99 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-13T15:12:11.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:11.823 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:11.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:11.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:11.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:12:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:12:11] "GET /metrics HTTP/1.1" 200 36649 "" "Prometheus/2.51.0" 2024-09-13T15:12:11.971 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:11.986 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:12.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:12.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:12.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:12.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:12.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:12.326 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:12:12.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:12.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:12.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/423183461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/423183461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:12.528 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.562 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:12.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:12.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:12.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:12.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/423183461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:12.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:12.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.823 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:12.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:12.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:12.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:12.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:13.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:13.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:13.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:13.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:13.221 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:13.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:13.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:13.359 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:13.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:13.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/423183461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[102314]: osdmap e472: 8 total, 8 up, 8 in 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[102314]: pgmap v1529: 99 pgs: 99 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-13T15:12:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:12:13.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:12:13.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/423183461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:12:13.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[107038]: osdmap e472: 8 total, 8 up, 8 in 2024-09-13T15:12:13.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:13.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:13.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:13 smithi154 ceph-mon[107038]: pgmap v1529: 99 pgs: 99 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-13T15:12:13.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:13.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:13.596 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:13.628 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:13.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:13.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:13 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:12:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:13 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:12:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/423183461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:12:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:13 smithi172 ceph-mon[100696]: osdmap e472: 8 total, 8 up, 8 in 2024-09-13T15:12:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:13 smithi172 ceph-mon[100696]: pgmap v1529: 99 pgs: 99 active+clean; 3.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-13T15:12:13.761 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:13.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:13.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:13.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:13.896 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:13.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:13.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:14.023 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:14.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:14.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:14.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.291 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:14.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:14.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:14.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:14.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:14.553 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:14.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.693 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:14.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:14.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:14.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:14.885 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:12:14.885 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:12:14.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:14.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:14.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:15.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:15.111 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:15.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:15.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:15.247 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:15.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:15.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:15.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:15.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:15.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:15.520 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:15.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:15.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:12:15.612 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:15 smithi154 ceph-mon[102314]: pgmap v1530: 99 pgs: 99 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 899 op/s 2024-09-13T15:12:15.613 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:15 smithi154 ceph-mon[107038]: pgmap v1530: 99 pgs: 99 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 899 op/s 2024-09-13T15:12:15.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:15.650 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:15.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:15 smithi172 ceph-mon[100696]: pgmap v1530: 99 pgs: 99 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 899 op/s 2024-09-13T15:12:15.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:15.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:15.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:15.922 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:15.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:15.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:16.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:16.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:16.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:16.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:16.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:16.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:16.331 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:12:16.332 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:12:16.332 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T15:12:16.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:12:16.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:16.359 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:16.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:16.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:16.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:16.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:16.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:16.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:16.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:16.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:16.776 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:16.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:16.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:16.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:16.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:16.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:16.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-13T15:12:17.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:17.046 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:17.148 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.176 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:17.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:17.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:17.304 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:17.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:17.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:17.543 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:17 smithi154 ceph-mon[102314]: pgmap v1531: 99 pgs: 99 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-13T15:12:17.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:17 smithi154 ceph-mon[107038]: pgmap v1531: 99 pgs: 99 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-13T15:12:17.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:17.560 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:12:17.560 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:12:17.571 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:17.595 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:12:17.595 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:12:17.641 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:12:17.641 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:12:17.649 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:17.672 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:17.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:17.677 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:12:17.678 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:12:17.692 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:17.696 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:17.711 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:12:17.712 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:12:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:17 smithi172 ceph-mon[100696]: pgmap v1531: 99 pgs: 99 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-13T15:12:17.748 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:12:17.748 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:12:17.782 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:12:17.783 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:12:17.798 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:17.815 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:12:17.815 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:12:17.824 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:17.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:17.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:17.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:18.062 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:18.089 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:18.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:18.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:18.215 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:18.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:18.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:18.354 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:18.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:18.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:18.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:18.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:18.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:18.565 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3025630937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:18.565 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3025630937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:18.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:18.620 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:18.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:18.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3025630937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:18.750 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:18.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:18.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:18.879 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:18.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:18.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:19.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:19.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:19.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:19.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.292 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:19.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:19.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:19.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:19.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.561 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:19.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:19.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.689 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:19.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:19.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:19 smithi154 ceph-mon[102314]: pgmap v1532: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-13T15:12:19.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:19 smithi154 ceph-mon[107038]: pgmap v1532: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-13T15:12:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:19 smithi172 ceph-mon[100696]: pgmap v1532: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 885 op/s 2024-09-13T15:12:19.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.828 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:19.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:19.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:19.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:19.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:20.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:20.108 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:20.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:20.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:20.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:20.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:20.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:20.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:20.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:20.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:20.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:20.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:20.613 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:12:20.613 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:12:20.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:20.656 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:20.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:20.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:12:20.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:20.778 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:20.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:20.880 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:20.902 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:20.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:21.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:21.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:21.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:21.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:21.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:21.178 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:21.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:21.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:21.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:21.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:21.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:21.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:21.449 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:21.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:21.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:21 smithi154 ceph-mon[102314]: pgmap v1533: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-13T15:12:21.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:21 smithi154 ceph-mon[107038]: pgmap v1533: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-13T15:12:21.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:21.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:21.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:21.661 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:21 smithi172 ceph-mon[100696]: pgmap v1533: 99 pgs: 99 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 894 op/s 2024-09-13T15:12:21.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:21.743 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:21.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:21.846 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:21.881 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:21.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:21.971 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:12:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:12:21] "GET /metrics HTTP/1.1" 200 36644 "" "Prometheus/2.51.0" 2024-09-13T15:12:21.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:22.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:22.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:22.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:22.177 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:22.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:22.279 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:22.322 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:22.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:22.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:12:22.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:22.460 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:22.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:22.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:22.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:22.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:22.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:22.770 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:22.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:22.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:22.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:22.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:23.012 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:23.053 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:23.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:23.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:23.214 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:23.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:23.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:23.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:23.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:23.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:23.490 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:23.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:23.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:23.596 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:23 smithi172 ceph-mon[100696]: pgmap v1534: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:12:23.616 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:23.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:23.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:23.619 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:23 smithi154 ceph-mon[102314]: pgmap v1534: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:12:23.619 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:23 smithi154 ceph-mon[107038]: pgmap v1534: 99 pgs: 99 active+clean; 3.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 804 op/s 2024-09-13T15:12:23.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:23.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:23.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:23.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:23.893 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:23.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:23.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:24.044 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:24.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:24.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:24.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.325 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:24.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:24.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:24.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:24.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:24.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:24.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.710 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:24.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:24.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:24.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:24.936 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:24.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:24.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:25.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:25.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:25.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:202536 10.b 10:dc96b526:::benchmark_data_smithi154_317818_object76860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.040375+0000", 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0017774760000000001, 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001806782, 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:12:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202536 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.040375+0000", 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.040375+0000", 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.040377+0000", 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.316e-06 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.040382+0000", 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8329999999999999e-06 2024-09-13T15:12:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.040398+0000", 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5481999999999998e-05 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.040405+0000", 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3679999999999996e-06 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.041896+0000", 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014909439999999999 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 1 2024-09-13T15:12:25.126 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:12:25.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:25.241 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:25.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:25.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:25.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:25.371 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:25.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:25.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:25 smithi154 ceph-mon[102314]: pgmap v1535: 99 pgs: 99 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-09-13T15:12:25.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:25 smithi154 ceph-mon[107038]: pgmap v1535: 99 pgs: 99 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-09-13T15:12:25.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:25.505 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:25.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:25.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:25.648 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:25.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:25.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:25 smithi172 ceph-mon[100696]: pgmap v1535: 99 pgs: 99 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 965 op/s 2024-09-13T15:12:25.748 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:25.782 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:25.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:25.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:25.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:25.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:26.022 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:26.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:26.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:26.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:26.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:26.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:26.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:26.363 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:12:26.365 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:12:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:26.378 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:26.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:26.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:26.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:26.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:26.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:26.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:26.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:26.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:26.794 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:26.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:26.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:26.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:26.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:26.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:27.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:27.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:27.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:27.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:27.227 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:27.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:27.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:27.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:27.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:27.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:27.483 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:27.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:27.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:27.619 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:27.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:27.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:27.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:27.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:27.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:27.882 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:27.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:27.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:28.007 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:28.027 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1535,"stamp":"2024-09-13T15:12:25.238793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3232045053,"num_objects":49273,"num_object_clones":0,"num_object_copies":147819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49273,"num_whiteouts":0,"num_read":5942,"num_read_kb":12955,"num_write":203011,"num_write_kb":8055698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":36635,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134275,"ondisk_log_size":134275,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15248276,"kb_used_data":9598804,"kb_used_omap":40,"kb_used_meta":5649303,"kb_avail":734712940,"statfs":{"total":767960285184,"available":752346050560,"internally_reserved":0,"allocated":9829175296,"data_stored":9808567471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":5784886388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,3,4,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-759365632,"num_objects":-11587,"num_object_clones":0,"num_object_copies":-34761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11587,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005699"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":168,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.352643+0000","last_change":"2024-09-13T15:08:31.190995+0000","last_active":"2024-09-13T15:12:12.352643+0000","last_peered":"2024-09-13T15:12:12.352643+0000","last_clean":"2024-09-13T15:12:12.352643+0000","last_became_active":"2024-09-13T15:08:31.190610+0000","last_became_peered":"2024-09-13T15:08:31.190610+0000","last_unstale":"2024-09-13T15:12:12.352643+0000","last_undegraded":"2024-09-13T15:12:12.352643+0000","last_fullsized":"2024-09-13T15:12:12.352643+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:42.124757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":53,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334203+0000","last_change":"2024-09-13T15:08:31.191585+0000","last_active":"2024-09-13T15:11:54.334203+0000","last_peered":"2024-09-13T15:11:54.334203+0000","last_clean":"2024-09-13T15:11:54.334203+0000","last_became_active":"2024-09-13T15:08:31.184700+0000","last_became_peered":"2024-09-13T15:08:31.184700+0000","last_unstale":"2024-09-13T15:11:54.334203+0000","last_undegraded":"2024-09-13T15:11:54.334203+0000","last_fullsized":"2024-09-13T15:11:54.334203+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:14.376834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":329,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326961+0000","last_change":"2024-09-13T15:08:46.080613+0000","last_active":"2024-09-13T15:11:54.326961+0000","last_peered":"2024-09-13T15:11:54.326961+0000","last_clean":"2024-09-13T15:11:54.326961+0000","last_became_active":"2024-09-13T15:08:46.080320+0000","last_became_peered":"2024-09-13T15:08:46.080320+0000","last_unstale":"2024-09-13T15:11:54.326961+0000","last_undegraded":"2024-09-13T15:11:54.326961+0000","last_fullsized":"2024-09-13T15:11:54.326961+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:09.205609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":784,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.338233+0000","last_change":"2024-09-13T15:08:31.196449+0000","last_active":"2024-09-13T15:12:12.338233+0000","last_peered":"2024-09-13T15:12:12.338233+0000","last_clean":"2024-09-13T15:12:12.338233+0000","last_became_active":"2024-09-13T15:08:31.196143+0000","last_became_peered":"2024-09-13T15:08:31.196143+0000","last_unstale":"2024-09-13T15:12:12.338233+0000","last_undegraded":"2024-09-13T15:12:12.338233+0000","last_fullsized":"2024-09-13T15:12:12.338233+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:53.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":788,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342780+0000","last_change":"2024-09-13T15:08:31.195499+0000","last_active":"2024-09-13T15:12:12.342780+0000","last_peered":"2024-09-13T15:12:12.342780+0000","last_clean":"2024-09-13T15:12:12.342780+0000","last_became_active":"2024-09-13T15:08:31.194928+0000","last_became_peered":"2024-09-13T15:08:31.194928+0000","last_unstale":"2024-09-13T15:12:12.342780+0000","last_undegraded":"2024-09-13T15:12:12.342780+0000","last_fullsized":"2024-09-13T15:12:12.342780+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.112396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":47,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334325+0000","last_change":"2024-09-13T15:08:31.185658+0000","last_active":"2024-09-13T15:11:54.334325+0000","last_peered":"2024-09-13T15:11:54.334325+0000","last_clean":"2024-09-13T15:11:54.334325+0000","last_became_active":"2024-09-13T15:08:31.185202+0000","last_became_peered":"2024-09-13T15:08:31.185202+0000","last_unstale":"2024-09-13T15:11:54.334325+0000","last_undegraded":"2024-09-13T15:11:54.334325+0000","last_fullsized":"2024-09-13T15:11:54.334325+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:04.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":288,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.346063+0000","last_change":"2024-09-13T15:08:31.199682+0000","last_active":"2024-09-13T15:12:12.346063+0000","last_peered":"2024-09-13T15:12:12.346063+0000","last_clean":"2024-09-13T15:12:12.346063+0000","last_became_active":"2024-09-13T15:08:31.199110+0000","last_became_peered":"2024-09-13T15:08:31.199110+0000","last_unstale":"2024-09-13T15:12:12.346063+0000","last_undegraded":"2024-09-13T15:12:12.346063+0000","last_fullsized":"2024-09-13T15:12:12.346063+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:30.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":394,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.337747+0000","last_change":"2024-09-13T15:04:05.021292+0000","last_active":"2024-09-13T15:12:12.337747+0000","last_peered":"2024-09-13T15:12:12.337747+0000","last_clean":"2024-09-13T15:12:12.337747+0000","last_became_active":"2024-09-13T15:04:05.011146+0000","last_became_peered":"2024-09-13T15:04:05.011146+0000","last_unstale":"2024-09-13T15:12:12.337747+0000","last_undegraded":"2024-09-13T15:12:12.337747+0000","last_fullsized":"2024-09-13T15:12:12.337747+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:00.137694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"472'12607","reported_seq":12641,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:24.075089+0000","last_change":"2024-09-13T15:09:16.042122+0000","last_active":"2024-09-13T15:12:24.075089+0000","last_peered":"2024-09-13T15:12:24.075089+0000","last_clean":"2024-09-13T15:12:24.075089+0000","last_became_active":"2024-09-13T15:09:16.041789+0000","last_became_peered":"2024-09-13T15:09:16.041789+0000","last_unstale":"2024-09-13T15:12:24.075089+0000","last_undegraded":"2024-09-13T15:12:24.075089+0000","last_fullsized":"2024-09-13T15:12:24.075089+0000","mapping_epoch":448,"log_start":"453'2600","ondisk_log_start":"453'2600","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:50.955816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195231744,"num_objects":2979,"num_object_clones":0,"num_object_copies":8937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12607,"num_write_kb":498752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":51,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334504+0000","last_change":"2024-09-13T15:08:31.351504+0000","last_active":"2024-09-13T15:11:54.334504+0000","last_peered":"2024-09-13T15:11:54.334504+0000","last_clean":"2024-09-13T15:11:54.334504+0000","last_became_active":"2024-09-13T15:08:31.320950+0000","last_became_peered":"2024-09-13T15:08:31.320950+0000","last_unstale":"2024-09-13T15:11:54.334504+0000","last_undegraded":"2024-09-13T15:11:54.334504+0000","last_fullsized":"2024-09-13T15:11:54.334504+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:38.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":375,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326474+0000","last_change":"2024-09-13T15:08:46.078296+0000","last_active":"2024-09-13T15:11:54.326474+0000","last_peered":"2024-09-13T15:11:54.326474+0000","last_clean":"2024-09-13T15:11:54.326474+0000","last_became_active":"2024-09-13T15:08:46.078037+0000","last_became_peered":"2024-09-13T15:08:46.078037+0000","last_unstale":"2024-09-13T15:11:54.326474+0000","last_undegraded":"2024-09-13T15:11:54.326474+0000","last_fullsized":"2024-09-13T15:11:54.326474+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:49.959361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"472'12411","reported_seq":12470,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:21.346957+0000","last_change":"2024-09-13T15:09:16.070197+0000","last_active":"2024-09-13T15:12:21.346957+0000","last_peered":"2024-09-13T15:12:21.346957+0000","last_clean":"2024-09-13T15:12:21.346957+0000","last_became_active":"2024-09-13T15:09:16.069805+0000","last_became_peered":"2024-09-13T15:09:16.069805+0000","last_unstale":"2024-09-13T15:12:21.346957+0000","last_undegraded":"2024-09-13T15:12:21.346957+0000","last_fullsized":"2024-09-13T15:12:21.346957+0000","mapping_epoch":448,"log_start":"457'4900","ondisk_log_start":"457'4900","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:17.465317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202702848,"num_objects":3093,"num_object_clones":0,"num_object_copies":9279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12411,"num_write_kb":496128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":51,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334367+0000","last_change":"2024-09-13T15:08:31.201939+0000","last_active":"2024-09-13T15:11:54.334367+0000","last_peered":"2024-09-13T15:11:54.334367+0000","last_clean":"2024-09-13T15:11:54.334367+0000","last_became_active":"2024-09-13T15:08:31.191089+0000","last_became_peered":"2024-09-13T15:08:31.191089+0000","last_unstale":"2024-09-13T15:11:54.334367+0000","last_undegraded":"2024-09-13T15:11:54.334367+0000","last_fullsized":"2024-09-13T15:11:54.334367+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:06.249945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":539,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326564+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:11:54.326564+0000","last_peered":"2024-09-13T15:11:54.326564+0000","last_clean":"2024-09-13T15:11:54.326564+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:11:54.326564+0000","last_undegraded":"2024-09-13T15:11:54.326564+0000","last_fullsized":"2024-09-13T15:11:54.326564+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"472'12855","reported_seq":12915,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:22.209275+0000","last_change":"2024-09-13T15:09:16.042793+0000","last_active":"2024-09-13T15:12:22.209275+0000","last_peered":"2024-09-13T15:12:22.209275+0000","last_clean":"2024-09-13T15:12:22.209275+0000","last_became_active":"2024-09-13T15:09:16.042016+0000","last_became_peered":"2024-09-13T15:09:16.042016+0000","last_unstale":"2024-09-13T15:12:22.209275+0000","last_undegraded":"2024-09-13T15:12:22.209275+0000","last_fullsized":"2024-09-13T15:12:22.209275+0000","mapping_epoch":448,"log_start":"460'5500","ondisk_log_start":"460'5500","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7355,"log_dups_size":0,"ondisk_log_size":7355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:40.268202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":9639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12855,"num_write_kb":514176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":51,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334416+0000","last_change":"2024-09-13T15:08:31.199736+0000","last_active":"2024-09-13T15:11:54.334416+0000","last_peered":"2024-09-13T15:11:54.334416+0000","last_clean":"2024-09-13T15:11:54.334416+0000","last_became_active":"2024-09-13T15:08:31.186018+0000","last_became_peered":"2024-09-13T15:08:31.186018+0000","last_unstale":"2024-09-13T15:11:54.334416+0000","last_undegraded":"2024-09-13T15:11:54.334416+0000","last_fullsized":"2024-09-13T15:11:54.334416+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:26.855187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":524,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342630+0000","last_change":"2024-09-13T15:08:31.193478+0000","last_active":"2024-09-13T15:12:12.342630+0000","last_peered":"2024-09-13T15:12:12.342630+0000","last_clean":"2024-09-13T15:12:12.342630+0000","last_became_active":"2024-09-13T15:08:31.193038+0000","last_became_peered":"2024-09-13T15:08:31.193038+0000","last_unstale":"2024-09-13T15:12:12.342630+0000","last_undegraded":"2024-09-13T15:12:12.342630+0000","last_fullsized":"2024-09-13T15:12:12.342630+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:05.600387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"472'12683","reported_seq":12743,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:24.155551+0000","last_change":"2024-09-13T15:09:16.043416+0000","last_active":"2024-09-13T15:12:24.155551+0000","last_peered":"2024-09-13T15:12:24.155551+0000","last_clean":"2024-09-13T15:12:24.155551+0000","last_became_active":"2024-09-13T15:09:16.043131+0000","last_became_peered":"2024-09-13T15:09:16.043131+0000","last_unstale":"2024-09-13T15:12:24.155551+0000","last_undegraded":"2024-09-13T15:12:24.155551+0000","last_fullsized":"2024-09-13T15:12:24.155551+0000","mapping_epoch":448,"log_start":"453'3300","ondisk_log_start":"453'3300","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":9383,"log_dups_size":0,"ondisk_log_size":9383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:21.376669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201261056,"num_objects":3071,"num_object_clones":0,"num_object_copies":9213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12683,"num_write_kb":504128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":346,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327279+0000","last_change":"2024-09-13T15:08:46.073352+0000","last_active":"2024-09-13T15:11:54.327279+0000","last_peered":"2024-09-13T15:11:54.327279+0000","last_clean":"2024-09-13T15:11:54.327279+0000","last_became_active":"2024-09-13T15:08:46.073048+0000","last_became_peered":"2024-09-13T15:08:46.073048+0000","last_unstale":"2024-09-13T15:11:54.327279+0000","last_undegraded":"2024-09-13T15:11:54.327279+0000","last_fullsized":"2024-09-13T15:11:54.327279+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:57.277685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":696,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.339994+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:12:12.339994+0000","last_peered":"2024-09-13T15:12:12.339994+0000","last_clean":"2024-09-13T15:12:12.339994+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:12:12.339994+0000","last_undegraded":"2024-09-13T15:12:12.339994+0000","last_fullsized":"2024-09-13T15:12:12.339994+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"89'149","reported_seq":853,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.338916+0000","last_change":"2024-09-13T15:10:26.504570+0000","last_active":"2024-09-13T15:12:12.338916+0000","last_peered":"2024-09-13T15:12:12.338916+0000","last_clean":"2024-09-13T15:12:12.338916+0000","last_became_active":"2024-09-13T15:10:19.782654+0000","last_became_peered":"2024-09-13T15:10:19.782654+0000","last_unstale":"2024-09-13T15:12:12.338916+0000","last_undegraded":"2024-09-13T15:12:12.338916+0000","last_fullsized":"2024-09-13T15:12:12.338916+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:10:26.504369+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:57.248804+0000","scrub_duration":0.013934767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"472'12555","reported_seq":12615,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:25.024923+0000","last_change":"2024-09-13T15:09:16.042156+0000","last_active":"2024-09-13T15:12:25.024923+0000","last_peered":"2024-09-13T15:12:25.024923+0000","last_clean":"2024-09-13T15:12:25.024923+0000","last_became_active":"2024-09-13T15:09:16.041712+0000","last_became_peered":"2024-09-13T15:09:16.041712+0000","last_unstale":"2024-09-13T15:12:25.024923+0000","last_undegraded":"2024-09-13T15:12:25.024923+0000","last_fullsized":"2024-09-13T15:12:25.024923+0000","mapping_epoch":448,"log_start":"453'3700","ondisk_log_start":"453'3700","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:57.243483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12555,"num_write_kb":499392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":822,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.349813+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:12:12.349813+0000","last_peered":"2024-09-13T15:12:12.349813+0000","last_clean":"2024-09-13T15:12:12.349813+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:12:12.349813+0000","last_undegraded":"2024-09-13T15:12:12.349813+0000","last_fullsized":"2024-09-13T15:12:12.349813+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":481,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326854+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:11:54.326854+0000","last_peered":"2024-09-13T15:11:54.326854+0000","last_clean":"2024-09-13T15:11:54.326854+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:11:54.326854+0000","last_undegraded":"2024-09-13T15:11:54.326854+0000","last_fullsized":"2024-09-13T15:11:54.326854+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"89'149","reported_seq":829,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326293+0000","last_change":"2024-09-13T15:10:20.069094+0000","last_active":"2024-09-13T15:11:54.326293+0000","last_peered":"2024-09-13T15:11:54.326293+0000","last_clean":"2024-09-13T15:11:54.326293+0000","last_became_active":"2024-09-13T15:10:19.782890+0000","last_became_peered":"2024-09-13T15:10:19.782890+0000","last_unstale":"2024-09-13T15:11:54.326293+0000","last_undegraded":"2024-09-13T15:11:54.326293+0000","last_fullsized":"2024-09-13T15:11:54.326293+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:20.068973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:20.068973+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:30.570922+0000","scrub_duration":0.011313818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.3","version":"472'12733","reported_seq":12793,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:24.162399+0000","last_change":"2024-09-13T15:09:16.042855+0000","last_active":"2024-09-13T15:12:24.162399+0000","last_peered":"2024-09-13T15:12:24.162399+0000","last_clean":"2024-09-13T15:12:24.162399+0000","last_became_active":"2024-09-13T15:09:16.042443+0000","last_became_peered":"2024-09-13T15:09:16.042443+0000","last_unstale":"2024-09-13T15:12:24.162399+0000","last_undegraded":"2024-09-13T15:12:24.162399+0000","last_fullsized":"2024-09-13T15:12:24.162399+0000","mapping_epoch":448,"log_start":"453'3300","ondisk_log_start":"453'3300","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":9433,"log_dups_size":0,"ondisk_log_size":9433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:28.644770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199163904,"num_objects":3039,"num_object_clones":0,"num_object_copies":9117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12733,"num_write_kb":504704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":279,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327625+0000","last_change":"2024-09-13T15:04:05.013503+0000","last_active":"2024-09-13T15:11:54.327625+0000","last_peered":"2024-09-13T15:11:54.327625+0000","last_clean":"2024-09-13T15:11:54.327625+0000","last_became_active":"2024-09-13T15:04:05.013204+0000","last_became_peered":"2024-09-13T15:04:05.013204+0000","last_unstale":"2024-09-13T15:11:54.327625+0000","last_undegraded":"2024-09-13T15:11:54.327625+0000","last_fullsized":"2024-09-13T15:11:54.327625+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:17.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":475,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327287+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:11:54.327287+0000","last_peered":"2024-09-13T15:11:54.327287+0000","last_clean":"2024-09-13T15:11:54.327287+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:11:54.327287+0000","last_undegraded":"2024-09-13T15:11:54.327287+0000","last_fullsized":"2024-09-13T15:11:54.327287+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"472'12855","reported_seq":12915,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:25.055315+0000","last_change":"2024-09-13T15:09:16.070224+0000","last_active":"2024-09-13T15:12:25.055315+0000","last_peered":"2024-09-13T15:12:25.055315+0000","last_clean":"2024-09-13T15:12:25.055315+0000","last_became_active":"2024-09-13T15:09:16.069845+0000","last_became_peered":"2024-09-13T15:09:16.069845+0000","last_unstale":"2024-09-13T15:12:25.055315+0000","last_undegraded":"2024-09-13T15:12:25.055315+0000","last_fullsized":"2024-09-13T15:12:25.055315+0000","mapping_epoch":448,"log_start":"454'4000","ondisk_log_start":"454'4000","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:44.584670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195887104,"num_objects":2989,"num_object_clones":0,"num_object_copies":8967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12855,"num_write_kb":507008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":500,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326683+0000","last_change":"2024-09-13T15:08:31.184973+0000","last_active":"2024-09-13T15:11:54.326683+0000","last_peered":"2024-09-13T15:11:54.326683+0000","last_clean":"2024-09-13T15:11:54.326683+0000","last_became_active":"2024-09-13T15:08:31.184600+0000","last_became_peered":"2024-09-13T15:08:31.184600+0000","last_unstale":"2024-09-13T15:11:54.326683+0000","last_undegraded":"2024-09-13T15:11:54.326683+0000","last_fullsized":"2024-09-13T15:11:54.326683+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:09.452895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":508,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327276+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:11:54.327276+0000","last_peered":"2024-09-13T15:11:54.327276+0000","last_clean":"2024-09-13T15:11:54.327276+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:11:54.327276+0000","last_undegraded":"2024-09-13T15:11:54.327276+0000","last_fullsized":"2024-09-13T15:11:54.327276+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"89'149","reported_seq":829,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326623+0000","last_change":"2024-09-13T15:10:22.099331+0000","last_active":"2024-09-13T15:11:54.326623+0000","last_peered":"2024-09-13T15:11:54.326623+0000","last_clean":"2024-09-13T15:11:54.326623+0000","last_became_active":"2024-09-13T15:10:19.785103+0000","last_became_peered":"2024-09-13T15:10:19.785103+0000","last_unstale":"2024-09-13T15:11:54.326623+0000","last_undegraded":"2024-09-13T15:11:54.326623+0000","last_fullsized":"2024-09-13T15:11:54.326623+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:22.099204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:22.099204+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:34.804057+0000","scrub_duration":0.0069277289999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1","version":"472'12516","reported_seq":12550,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:23.882653+0000","last_change":"2024-09-13T15:09:16.040707+0000","last_active":"2024-09-13T15:12:23.882653+0000","last_peered":"2024-09-13T15:12:23.882653+0000","last_clean":"2024-09-13T15:12:23.882653+0000","last_became_active":"2024-09-13T15:09:16.040411+0000","last_became_peered":"2024-09-13T15:09:16.040411+0000","last_unstale":"2024-09-13T15:12:23.882653+0000","last_undegraded":"2024-09-13T15:12:23.882653+0000","last_fullsized":"2024-09-13T15:12:23.882653+0000","mapping_epoch":448,"log_start":"462'5500","ondisk_log_start":"462'5500","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:13.485945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197656576,"num_objects":3016,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12516,"num_write_kb":497024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":289,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.341340+0000","last_change":"2024-09-13T15:08:31.198602+0000","last_active":"2024-09-13T15:12:12.341340+0000","last_peered":"2024-09-13T15:12:12.341340+0000","last_clean":"2024-09-13T15:12:12.341340+0000","last_became_active":"2024-09-13T15:08:31.198289+0000","last_became_peered":"2024-09-13T15:08:31.198289+0000","last_unstale":"2024-09-13T15:12:12.341340+0000","last_undegraded":"2024-09-13T15:12:12.341340+0000","last_fullsized":"2024-09-13T15:12:12.341340+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:36.803833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":568,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.353423+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:12:12.353423+0000","last_peered":"2024-09-13T15:12:12.353423+0000","last_clean":"2024-09-13T15:12:12.353423+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:12:12.353423+0000","last_undegraded":"2024-09-13T15:12:12.353423+0000","last_fullsized":"2024-09-13T15:12:12.353423+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"472'12875","reported_seq":12935,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:25.012808+0000","last_change":"2024-09-13T15:09:16.069640+0000","last_active":"2024-09-13T15:12:25.012808+0000","last_peered":"2024-09-13T15:12:25.012808+0000","last_clean":"2024-09-13T15:12:25.012808+0000","last_became_active":"2024-09-13T15:09:16.069319+0000","last_became_peered":"2024-09-13T15:09:16.069319+0000","last_unstale":"2024-09-13T15:12:25.012808+0000","last_undegraded":"2024-09-13T15:12:25.012808+0000","last_fullsized":"2024-09-13T15:12:25.012808+0000","mapping_epoch":448,"log_start":"454'4000","ondisk_log_start":"454'4000","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":8875,"log_dups_size":0,"ondisk_log_size":8875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:52.471344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199294976,"num_objects":3041,"num_object_clones":0,"num_object_copies":9123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12875,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":836,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334318+0000","last_change":"2024-09-13T15:10:16.461469+0000","last_active":"2024-09-13T15:11:54.334318+0000","last_peered":"2024-09-13T15:11:54.334318+0000","last_clean":"2024-09-13T15:11:54.334318+0000","last_became_active":"2024-09-13T15:10:10.356111+0000","last_became_peered":"2024-09-13T15:10:10.356111+0000","last_unstale":"2024-09-13T15:11:54.334318+0000","last_undegraded":"2024-09-13T15:11:54.334318+0000","last_fullsized":"2024-09-13T15:11:54.334318+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:16.460943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:16.460943+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:12:14.909595+0000","scrub_duration":0.013345770999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1142,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.330422+0000","last_change":"2024-09-13T15:08:57.728783+0000","last_active":"2024-09-13T15:11:54.330422+0000","last_peered":"2024-09-13T15:11:54.330422+0000","last_clean":"2024-09-13T15:11:54.330422+0000","last_became_active":"2024-09-13T15:08:47.077688+0000","last_became_peered":"2024-09-13T15:08:47.077688+0000","last_unstale":"2024-09-13T15:11:54.330422+0000","last_undegraded":"2024-09-13T15:11:54.330422+0000","last_fullsized":"2024-09-13T15:11:54.330422+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:52.710051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":784,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342001+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:12:12.342001+0000","last_peered":"2024-09-13T15:12:12.342001+0000","last_clean":"2024-09-13T15:12:12.342001+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:12:12.342001+0000","last_undegraded":"2024-09-13T15:12:12.342001+0000","last_fullsized":"2024-09-13T15:12:12.342001+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":327,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327176+0000","last_change":"2024-09-13T15:08:31.187510+0000","last_active":"2024-09-13T15:11:54.327176+0000","last_peered":"2024-09-13T15:11:54.327176+0000","last_clean":"2024-09-13T15:11:54.327176+0000","last_became_active":"2024-09-13T15:08:31.187108+0000","last_became_peered":"2024-09-13T15:08:31.187108+0000","last_unstale":"2024-09-13T15:11:54.327176+0000","last_undegraded":"2024-09-13T15:11:54.327176+0000","last_fullsized":"2024-09-13T15:11:54.327176+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:25.096543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"89'149","reported_seq":835,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.343146+0000","last_change":"2024-09-13T15:10:20.593635+0000","last_active":"2024-09-13T15:12:12.343146+0000","last_peered":"2024-09-13T15:12:12.343146+0000","last_clean":"2024-09-13T15:12:12.343146+0000","last_became_active":"2024-09-13T15:10:19.868381+0000","last_became_peered":"2024-09-13T15:10:19.868381+0000","last_unstale":"2024-09-13T15:12:12.343146+0000","last_undegraded":"2024-09-13T15:12:12.343146+0000","last_fullsized":"2024-09-13T15:12:12.343146+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:20.593564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:20.593564+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:08.556697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"472'12644","reported_seq":12704,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:22.178888+0000","last_change":"2024-09-13T15:09:16.042809+0000","last_active":"2024-09-13T15:12:22.178888+0000","last_peered":"2024-09-13T15:12:22.178888+0000","last_clean":"2024-09-13T15:12:22.178888+0000","last_became_active":"2024-09-13T15:09:16.041824+0000","last_became_peered":"2024-09-13T15:09:16.041824+0000","last_unstale":"2024-09-13T15:12:22.178888+0000","last_undegraded":"2024-09-13T15:12:22.178888+0000","last_fullsized":"2024-09-13T15:12:22.178888+0000","mapping_epoch":448,"log_start":"459'5300","ondisk_log_start":"459'5300","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7344,"log_dups_size":0,"ondisk_log_size":7344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:31:05.692268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209453056,"num_objects":3196,"num_object_clones":0,"num_object_copies":9588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12644,"num_write_kb":506880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":65,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334479+0000","last_change":"2024-09-13T15:08:31.218115+0000","last_active":"2024-09-13T15:11:54.334479+0000","last_peered":"2024-09-13T15:11:54.334479+0000","last_clean":"2024-09-13T15:11:54.334479+0000","last_became_active":"2024-09-13T15:08:31.196118+0000","last_became_peered":"2024-09-13T15:08:31.196118+0000","last_unstale":"2024-09-13T15:11:54.334479+0000","last_undegraded":"2024-09-13T15:11:54.334479+0000","last_fullsized":"2024-09-13T15:11:54.334479+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:08.721976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":771,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.338537+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:12:12.338537+0000","last_peered":"2024-09-13T15:12:12.338537+0000","last_clean":"2024-09-13T15:12:12.338537+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:12:12.338537+0000","last_undegraded":"2024-09-13T15:12:12.338537+0000","last_fullsized":"2024-09-13T15:12:12.338537+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":765,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342605+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:12:12.342605+0000","last_peered":"2024-09-13T15:12:12.342605+0000","last_clean":"2024-09-13T15:12:12.342605+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:12:12.342605+0000","last_undegraded":"2024-09-13T15:12:12.342605+0000","last_fullsized":"2024-09-13T15:12:12.342605+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"89'149","reported_seq":853,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.338710+0000","last_change":"2024-09-13T15:10:25.509321+0000","last_active":"2024-09-13T15:12:12.338710+0000","last_peered":"2024-09-13T15:12:12.338710+0000","last_clean":"2024-09-13T15:12:12.338710+0000","last_became_active":"2024-09-13T15:10:19.782838+0000","last_became_peered":"2024-09-13T15:10:19.782838+0000","last_unstale":"2024-09-13T15:12:12.338710+0000","last_undegraded":"2024-09-13T15:12:12.338710+0000","last_fullsized":"2024-09-13T15:12:12.338710+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:10:25.509161+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:41.280899+0000","scrub_duration":0.0086533800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"472'12652","reported_seq":12686,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:24.013237+0000","last_change":"2024-09-13T15:09:16.040901+0000","last_active":"2024-09-13T15:12:24.013237+0000","last_peered":"2024-09-13T15:12:24.013237+0000","last_clean":"2024-09-13T15:12:24.013237+0000","last_became_active":"2024-09-13T15:09:16.040576+0000","last_became_peered":"2024-09-13T15:09:16.040576+0000","last_unstale":"2024-09-13T15:12:24.013237+0000","last_undegraded":"2024-09-13T15:12:24.013237+0000","last_fullsized":"2024-09-13T15:12:24.013237+0000","mapping_epoch":448,"log_start":"453'2600","ondisk_log_start":"453'2600","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:43.937156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206045184,"num_objects":3144,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12652,"num_write_kb":505472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1205,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.362885+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:12:12.362885+0000","last_peered":"2024-09-13T15:12:12.362885+0000","last_clean":"2024-09-13T15:12:12.362885+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:12:12.362885+0000","last_undegraded":"2024-09-13T15:12:12.362885+0000","last_fullsized":"2024-09-13T15:12:12.362885+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":834,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342439+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:12:12.342439+0000","last_peered":"2024-09-13T15:12:12.342439+0000","last_clean":"2024-09-13T15:12:12.342439+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:12:12.342439+0000","last_undegraded":"2024-09-13T15:12:12.342439+0000","last_fullsized":"2024-09-13T15:12:12.342439+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":572,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.345170+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:12:12.345170+0000","last_peered":"2024-09-13T15:12:12.345170+0000","last_clean":"2024-09-13T15:12:12.345170+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:12:12.345170+0000","last_undegraded":"2024-09-13T15:12:12.345170+0000","last_fullsized":"2024-09-13T15:12:12.345170+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"89'149","reported_seq":830,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.336832+0000","last_change":"2024-09-13T15:10:28.100953+0000","last_active":"2024-09-13T15:12:12.336832+0000","last_peered":"2024-09-13T15:12:12.336832+0000","last_clean":"2024-09-13T15:12:12.336832+0000","last_became_active":"2024-09-13T15:10:19.782915+0000","last_became_peered":"2024-09-13T15:10:19.782915+0000","last_unstale":"2024-09-13T15:12:12.336832+0000","last_undegraded":"2024-09-13T15:12:12.336832+0000","last_fullsized":"2024-09-13T15:12:12.336832+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:28.100874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:28.100874+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:46.155201+0000","scrub_duration":0.0092040070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"472'12624","reported_seq":12684,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:22.195980+0000","last_change":"2024-09-13T15:09:16.069575+0000","last_active":"2024-09-13T15:12:22.195980+0000","last_peered":"2024-09-13T15:12:22.195980+0000","last_clean":"2024-09-13T15:12:22.195980+0000","last_became_active":"2024-09-13T15:09:16.069234+0000","last_became_peered":"2024-09-13T15:09:16.069234+0000","last_unstale":"2024-09-13T15:12:22.195980+0000","last_undegraded":"2024-09-13T15:12:22.195980+0000","last_fullsized":"2024-09-13T15:12:22.195980+0000","mapping_epoch":448,"log_start":"458'5300","ondisk_log_start":"458'5300","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:43.686408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204210176,"num_objects":3116,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12624,"num_write_kb":503680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":396,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326852+0000","last_change":"2024-09-13T15:08:46.074249+0000","last_active":"2024-09-13T15:11:54.326852+0000","last_peered":"2024-09-13T15:11:54.326852+0000","last_clean":"2024-09-13T15:11:54.326852+0000","last_became_active":"2024-09-13T15:08:46.073842+0000","last_became_peered":"2024-09-13T15:08:46.073842+0000","last_unstale":"2024-09-13T15:11:54.326852+0000","last_undegraded":"2024-09-13T15:11:54.326852+0000","last_fullsized":"2024-09-13T15:11:54.326852+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:44.854067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:12:03.966554+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:12:03.966554+0000","last_peered":"2024-09-13T15:12:03.966554+0000","last_clean":"2024-09-13T15:12:03.966554+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:12:03.966554+0000","last_undegraded":"2024-09-13T15:12:03.966554+0000","last_fullsized":"2024-09-13T15:12:03.966554+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":313,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326629+0000","last_change":"2024-09-13T15:08:46.075360+0000","last_active":"2024-09-13T15:11:54.326629+0000","last_peered":"2024-09-13T15:11:54.326629+0000","last_clean":"2024-09-13T15:11:54.326629+0000","last_became_active":"2024-09-13T15:08:46.075046+0000","last_became_peered":"2024-09-13T15:08:46.075046+0000","last_unstale":"2024-09-13T15:11:54.326629+0000","last_undegraded":"2024-09-13T15:11:54.326629+0000","last_fullsized":"2024-09-13T15:11:54.326629+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.044150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"89'149","reported_seq":834,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.345953+0000","last_change":"2024-09-13T15:10:21.244600+0000","last_active":"2024-09-13T15:12:12.345953+0000","last_peered":"2024-09-13T15:12:12.345953+0000","last_clean":"2024-09-13T15:12:12.345953+0000","last_became_active":"2024-09-13T15:10:19.784559+0000","last_became_peered":"2024-09-13T15:10:19.784559+0000","last_unstale":"2024-09-13T15:12:12.345953+0000","last_undegraded":"2024-09-13T15:12:12.345953+0000","last_fullsized":"2024-09-13T15:12:12.345953+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:21.244441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:21.244441+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:57.154783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"472'12572","reported_seq":12606,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:23.826112+0000","last_change":"2024-09-13T15:09:16.042846+0000","last_active":"2024-09-13T15:12:23.826112+0000","last_peered":"2024-09-13T15:12:23.826112+0000","last_clean":"2024-09-13T15:12:23.826112+0000","last_became_active":"2024-09-13T15:09:16.042571+0000","last_became_peered":"2024-09-13T15:09:16.042571+0000","last_unstale":"2024-09-13T15:12:23.826112+0000","last_undegraded":"2024-09-13T15:12:23.826112+0000","last_fullsized":"2024-09-13T15:12:23.826112+0000","mapping_epoch":448,"log_start":"454'4600","ondisk_log_start":"454'4600","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7972,"log_dups_size":0,"ondisk_log_size":7972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:16.701976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193986560,"num_objects":2960,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12572,"num_write_kb":497024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":346,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326888+0000","last_change":"2024-09-13T15:08:46.079918+0000","last_active":"2024-09-13T15:11:54.326888+0000","last_peered":"2024-09-13T15:11:54.326888+0000","last_clean":"2024-09-13T15:11:54.326888+0000","last_became_active":"2024-09-13T15:08:46.075051+0000","last_became_peered":"2024-09-13T15:08:46.075051+0000","last_unstale":"2024-09-13T15:11:54.326888+0000","last_undegraded":"2024-09-13T15:11:54.326888+0000","last_fullsized":"2024-09-13T15:11:54.326888+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:54.485416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1486,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342688+0000","last_change":"2024-09-13T15:08:59.341688+0000","last_active":"2024-09-13T15:12:12.342688+0000","last_peered":"2024-09-13T15:12:12.342688+0000","last_clean":"2024-09-13T15:12:12.342688+0000","last_became_active":"2024-09-13T15:08:59.340925+0000","last_became_peered":"2024-09-13T15:08:59.340925+0000","last_unstale":"2024-09-13T15:12:12.342688+0000","last_undegraded":"2024-09-13T15:12:12.342688+0000","last_fullsized":"2024-09-13T15:12:12.342688+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:22.966119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":416,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.338148+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:12:12.338148+0000","last_peered":"2024-09-13T15:12:12.338148+0000","last_clean":"2024-09-13T15:12:12.338148+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:12:12.338148+0000","last_undegraded":"2024-09-13T15:12:12.338148+0000","last_fullsized":"2024-09-13T15:12:12.338148+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"89'149","reported_seq":840,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.338215+0000","last_change":"2024-09-13T15:10:19.802154+0000","last_active":"2024-09-13T15:12:12.338215+0000","last_peered":"2024-09-13T15:12:12.338215+0000","last_clean":"2024-09-13T15:12:12.338215+0000","last_became_active":"2024-09-13T15:10:19.784455+0000","last_became_peered":"2024-09-13T15:10:19.784455+0000","last_unstale":"2024-09-13T15:12:12.338215+0000","last_undegraded":"2024-09-13T15:12:12.338215+0000","last_fullsized":"2024-09-13T15:12:12.338215+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:19.802110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:19.802110+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:31.531320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"472'12533","reported_seq":12568,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:23.962736+0000","last_change":"2024-09-13T15:09:16.069726+0000","last_active":"2024-09-13T15:12:23.962736+0000","last_peered":"2024-09-13T15:12:23.962736+0000","last_clean":"2024-09-13T15:12:23.962736+0000","last_became_active":"2024-09-13T15:09:16.069375+0000","last_became_peered":"2024-09-13T15:09:16.069375+0000","last_unstale":"2024-09-13T15:12:23.962736+0000","last_undegraded":"2024-09-13T15:12:23.962736+0000","last_fullsized":"2024-09-13T15:12:23.962736+0000","mapping_epoch":448,"log_start":"462'5500","ondisk_log_start":"462'5500","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7033,"log_dups_size":0,"ondisk_log_size":7033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:29.701410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202375192,"num_objects":3089,"num_object_clones":0,"num_object_copies":9267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12533,"num_write_kb":499841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":826,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342542+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:12:12.342542+0000","last_peered":"2024-09-13T15:12:12.342542+0000","last_clean":"2024-09-13T15:12:12.342542+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:12:12.342542+0000","last_undegraded":"2024-09-13T15:12:12.342542+0000","last_fullsized":"2024-09-13T15:12:12.342542+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":65,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334133+0000","last_change":"2024-09-13T15:08:31.209459+0000","last_active":"2024-09-13T15:11:54.334133+0000","last_peered":"2024-09-13T15:11:54.334133+0000","last_clean":"2024-09-13T15:11:54.334133+0000","last_became_active":"2024-09-13T15:08:31.188811+0000","last_became_peered":"2024-09-13T15:08:31.188811+0000","last_unstale":"2024-09-13T15:11:54.334133+0000","last_undegraded":"2024-09-13T15:11:54.334133+0000","last_fullsized":"2024-09-13T15:11:54.334133+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.667223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":775,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.340242+0000","last_change":"2024-09-13T15:08:31.197894+0000","last_active":"2024-09-13T15:12:12.340242+0000","last_peered":"2024-09-13T15:12:12.340242+0000","last_clean":"2024-09-13T15:12:12.340242+0000","last_became_active":"2024-09-13T15:08:31.197516+0000","last_became_peered":"2024-09-13T15:08:31.197516+0000","last_unstale":"2024-09-13T15:12:12.340242+0000","last_undegraded":"2024-09-13T15:12:12.340242+0000","last_fullsized":"2024-09-13T15:12:12.340242+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:42.650595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"89'149","reported_seq":830,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.333177+0000","last_change":"2024-09-13T15:10:25.007758+0000","last_active":"2024-09-13T15:12:12.333177+0000","last_peered":"2024-09-13T15:12:12.333177+0000","last_clean":"2024-09-13T15:12:12.333177+0000","last_became_active":"2024-09-13T15:10:19.785150+0000","last_became_peered":"2024-09-13T15:10:19.785150+0000","last_unstale":"2024-09-13T15:12:12.333177+0000","last_undegraded":"2024-09-13T15:12:12.333177+0000","last_fullsized":"2024-09-13T15:12:12.333177+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:25.007498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:25.007498+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:10.893248+0000","scrub_duration":0.0037146029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.e","version":"472'12725","reported_seq":12759,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:23.933884+0000","last_change":"2024-09-13T15:09:16.069586+0000","last_active":"2024-09-13T15:12:23.933884+0000","last_peered":"2024-09-13T15:12:23.933884+0000","last_clean":"2024-09-13T15:12:23.933884+0000","last_became_active":"2024-09-13T15:09:16.069287+0000","last_became_peered":"2024-09-13T15:09:16.069287+0000","last_unstale":"2024-09-13T15:12:23.933884+0000","last_undegraded":"2024-09-13T15:12:23.933884+0000","last_fullsized":"2024-09-13T15:12:23.933884+0000","mapping_epoch":448,"log_start":"463'5700","ondisk_log_start":"463'5700","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:30.843370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198770688,"num_objects":3033,"num_object_clones":0,"num_object_copies":9099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12725,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":878,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.346275+0000","last_change":"2024-09-13T15:08:31.202795+0000","last_active":"2024-09-13T15:12:12.346275+0000","last_peered":"2024-09-13T15:12:12.346275+0000","last_clean":"2024-09-13T15:12:12.346275+0000","last_became_active":"2024-09-13T15:08:31.199963+0000","last_became_peered":"2024-09-13T15:08:31.199963+0000","last_unstale":"2024-09-13T15:12:12.346275+0000","last_undegraded":"2024-09-13T15:12:12.346275+0000","last_fullsized":"2024-09-13T15:12:12.346275+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:08.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1312,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.345608+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:12:12.345608+0000","last_peered":"2024-09-13T15:12:12.345608+0000","last_clean":"2024-09-13T15:12:12.345608+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:12:12.345608+0000","last_undegraded":"2024-09-13T15:12:12.345608+0000","last_fullsized":"2024-09-13T15:12:12.345608+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":582,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.341426+0000","last_change":"2024-09-13T15:08:31.198729+0000","last_active":"2024-09-13T15:12:12.341426+0000","last_peered":"2024-09-13T15:12:12.341426+0000","last_clean":"2024-09-13T15:12:12.341426+0000","last_became_active":"2024-09-13T15:08:31.198423+0000","last_became_peered":"2024-09-13T15:08:31.198423+0000","last_unstale":"2024-09-13T15:12:12.341426+0000","last_undegraded":"2024-09-13T15:12:12.341426+0000","last_fullsized":"2024-09-13T15:12:12.341426+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:58.362919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"89'149","reported_seq":835,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342575+0000","last_change":"2024-09-13T15:10:20.231220+0000","last_active":"2024-09-13T15:12:12.342575+0000","last_peered":"2024-09-13T15:12:12.342575+0000","last_clean":"2024-09-13T15:12:12.342575+0000","last_became_active":"2024-09-13T15:10:19.782705+0000","last_became_peered":"2024-09-13T15:10:19.782705+0000","last_unstale":"2024-09-13T15:12:12.342575+0000","last_undegraded":"2024-09-13T15:12:12.342575+0000","last_fullsized":"2024-09-13T15:12:12.342575+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:20.231025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:20.231025+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:44.195293+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"472'12263","reported_seq":12323,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:21.320775+0000","last_change":"2024-09-13T15:09:16.070043+0000","last_active":"2024-09-13T15:12:21.320775+0000","last_peered":"2024-09-13T15:12:21.320775+0000","last_clean":"2024-09-13T15:12:21.320775+0000","last_became_active":"2024-09-13T15:09:16.069704+0000","last_became_peered":"2024-09-13T15:09:16.069704+0000","last_unstale":"2024-09-13T15:12:21.320775+0000","last_undegraded":"2024-09-13T15:12:21.320775+0000","last_fullsized":"2024-09-13T15:12:21.320775+0000","mapping_epoch":448,"log_start":"456'4700","ondisk_log_start":"456'4700","created":448,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:09:15.007807+0000","last_clean_scrub_stamp":"2024-09-13T15:09:15.007807+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:58:30.439736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12263,"num_write_kb":495296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":701,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342873+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:12:12.342873+0000","last_peered":"2024-09-13T15:12:12.342873+0000","last_clean":"2024-09-13T15:12:12.342873+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:12:12.342873+0000","last_undegraded":"2024-09-13T15:12:12.342873+0000","last_fullsized":"2024-09-13T15:12:12.342873+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":904,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334536+0000","last_change":"2024-09-13T15:08:59.333844+0000","last_active":"2024-09-13T15:11:54.334536+0000","last_peered":"2024-09-13T15:11:54.334536+0000","last_clean":"2024-09-13T15:11:54.334536+0000","last_became_active":"2024-09-13T15:08:59.332369+0000","last_became_peered":"2024-09-13T15:08:59.332369+0000","last_unstale":"2024-09-13T15:11:54.334536+0000","last_undegraded":"2024-09-13T15:11:54.334536+0000","last_fullsized":"2024-09-13T15:11:54.334536+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:36.490753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":497,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:58.961604+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:11:58.961604+0000","last_peered":"2024-09-13T15:11:58.961604+0000","last_clean":"2024-09-13T15:11:58.961604+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:11:58.961604+0000","last_undegraded":"2024-09-13T15:11:58.961604+0000","last_fullsized":"2024-09-13T15:11:58.961604+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":480,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327193+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:11:54.327193+0000","last_peered":"2024-09-13T15:11:54.327193+0000","last_clean":"2024-09-13T15:11:54.327193+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:11:54.327193+0000","last_undegraded":"2024-09-13T15:11:54.327193+0000","last_fullsized":"2024-09-13T15:11:54.327193+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":612,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327425+0000","last_change":"2024-09-13T15:08:31.193477+0000","last_active":"2024-09-13T15:11:54.327425+0000","last_peered":"2024-09-13T15:11:54.327425+0000","last_clean":"2024-09-13T15:11:54.327425+0000","last_became_active":"2024-09-13T15:08:31.181861+0000","last_became_peered":"2024-09-13T15:08:31.181861+0000","last_unstale":"2024-09-13T15:11:54.327425+0000","last_undegraded":"2024-09-13T15:11:54.327425+0000","last_fullsized":"2024-09-13T15:11:54.327425+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:02.424164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":33292,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":837,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342785+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:12:12.342785+0000","last_peered":"2024-09-13T15:12:12.342785+0000","last_clean":"2024-09-13T15:12:12.342785+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:12:12.342785+0000","last_undegraded":"2024-09-13T15:12:12.342785+0000","last_fullsized":"2024-09-13T15:12:12.342785+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":53,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.334277+0000","last_change":"2024-09-13T15:08:31.204593+0000","last_active":"2024-09-13T15:11:54.334277+0000","last_peered":"2024-09-13T15:11:54.334277+0000","last_clean":"2024-09-13T15:11:54.334277+0000","last_became_active":"2024-09-13T15:08:31.198883+0000","last_became_peered":"2024-09-13T15:08:31.198883+0000","last_unstale":"2024-09-13T15:11:54.334277+0000","last_undegraded":"2024-09-13T15:11:54.334277+0000","last_fullsized":"2024-09-13T15:11:54.334277+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:08.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":369,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327440+0000","last_change":"2024-09-13T15:08:31.188790+0000","last_active":"2024-09-13T15:11:54.327440+0000","last_peered":"2024-09-13T15:11:54.327440+0000","last_clean":"2024-09-13T15:11:54.327440+0000","last_became_active":"2024-09-13T15:08:31.188441+0000","last_became_peered":"2024-09-13T15:08:31.188441+0000","last_unstale":"2024-09-13T15:11:54.327440+0000","last_undegraded":"2024-09-13T15:11:54.327440+0000","last_fullsized":"2024-09-13T15:11:54.327440+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:52.934163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":825,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.341716+0000","last_change":"2024-09-13T15:08:31.197112+0000","last_active":"2024-09-13T15:12:12.341716+0000","last_peered":"2024-09-13T15:12:12.341716+0000","last_clean":"2024-09-13T15:12:12.341716+0000","last_became_active":"2024-09-13T15:08:31.196687+0000","last_became_peered":"2024-09-13T15:08:31.196687+0000","last_unstale":"2024-09-13T15:12:12.341716+0000","last_undegraded":"2024-09-13T15:12:12.341716+0000","last_fullsized":"2024-09-13T15:12:12.341716+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:47.532912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":793,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.338060+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:12:12.338060+0000","last_peered":"2024-09-13T15:12:12.338060+0000","last_clean":"2024-09-13T15:12:12.338060+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:12:12.338060+0000","last_undegraded":"2024-09-13T15:12:12.338060+0000","last_fullsized":"2024-09-13T15:12:12.338060+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":855,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.341497+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:12:12.341497+0000","last_peered":"2024-09-13T15:12:12.341497+0000","last_clean":"2024-09-13T15:12:12.341497+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:12:12.341497+0000","last_undegraded":"2024-09-13T15:12:12.341497+0000","last_fullsized":"2024-09-13T15:12:12.341497+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":275,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327776+0000","last_change":"2024-09-13T15:08:46.075112+0000","last_active":"2024-09-13T15:11:54.327776+0000","last_peered":"2024-09-13T15:11:54.327776+0000","last_clean":"2024-09-13T15:11:54.327776+0000","last_became_active":"2024-09-13T15:08:46.074818+0000","last_became_peered":"2024-09-13T15:08:46.074818+0000","last_unstale":"2024-09-13T15:11:54.327776+0000","last_undegraded":"2024-09-13T15:11:54.327776+0000","last_fullsized":"2024-09-13T15:11:54.327776+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:56.288263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":818,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.341566+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:12:12.341566+0000","last_peered":"2024-09-13T15:12:12.341566+0000","last_clean":"2024-09-13T15:12:12.341566+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:12:12.341566+0000","last_undegraded":"2024-09-13T15:12:12.341566+0000","last_fullsized":"2024-09-13T15:12:12.341566+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":790,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342697+0000","last_change":"2024-09-13T15:08:31.195215+0000","last_active":"2024-09-13T15:12:12.342697+0000","last_peered":"2024-09-13T15:12:12.342697+0000","last_clean":"2024-09-13T15:12:12.342697+0000","last_became_active":"2024-09-13T15:08:31.194526+0000","last_became_peered":"2024-09-13T15:08:31.194526+0000","last_unstale":"2024-09-13T15:12:12.342697+0000","last_undegraded":"2024-09-13T15:12:12.342697+0000","last_fullsized":"2024-09-13T15:12:12.342697+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:56.358764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":792,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342337+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:12:12.342337+0000","last_peered":"2024-09-13T15:12:12.342337+0000","last_clean":"2024-09-13T15:12:12.342337+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:12:12.342337+0000","last_undegraded":"2024-09-13T15:12:12.342337+0000","last_fullsized":"2024-09-13T15:12:12.342337+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":656,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.343865+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:12:12.343865+0000","last_peered":"2024-09-13T15:12:12.343865+0000","last_clean":"2024-09-13T15:12:12.343865+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:12:12.343865+0000","last_undegraded":"2024-09-13T15:12:12.343865+0000","last_fullsized":"2024-09-13T15:12:12.343865+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":475,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327720+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:11:54.327720+0000","last_peered":"2024-09-13T15:11:54.327720+0000","last_clean":"2024-09-13T15:11:54.327720+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:11:54.327720+0000","last_undegraded":"2024-09-13T15:11:54.327720+0000","last_fullsized":"2024-09-13T15:11:54.327720+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":407,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.352899+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:12:12.352899+0000","last_peered":"2024-09-13T15:12:12.352899+0000","last_clean":"2024-09-13T15:12:12.352899+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:12:12.352899+0000","last_undegraded":"2024-09-13T15:12:12.352899+0000","last_fullsized":"2024-09-13T15:12:12.352899+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":778,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.340340+0000","last_change":"2024-09-13T15:08:31.194912+0000","last_active":"2024-09-13T15:12:12.340340+0000","last_peered":"2024-09-13T15:12:12.340340+0000","last_clean":"2024-09-13T15:12:12.340340+0000","last_became_active":"2024-09-13T15:08:31.194553+0000","last_became_peered":"2024-09-13T15:08:31.194553+0000","last_unstale":"2024-09-13T15:12:12.340340+0000","last_undegraded":"2024-09-13T15:12:12.340340+0000","last_fullsized":"2024-09-13T15:12:12.340340+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:17.457690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":485,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327146+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:11:54.327146+0000","last_peered":"2024-09-13T15:11:54.327146+0000","last_clean":"2024-09-13T15:11:54.327146+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:11:54.327146+0000","last_undegraded":"2024-09-13T15:11:54.327146+0000","last_fullsized":"2024-09-13T15:11:54.327146+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":313,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327198+0000","last_change":"2024-09-13T15:08:46.073232+0000","last_active":"2024-09-13T15:11:54.327198+0000","last_peered":"2024-09-13T15:11:54.327198+0000","last_clean":"2024-09-13T15:11:54.327198+0000","last_became_active":"2024-09-13T15:08:46.072807+0000","last_became_peered":"2024-09-13T15:08:46.072807+0000","last_unstale":"2024-09-13T15:11:54.327198+0000","last_undegraded":"2024-09-13T15:11:54.327198+0000","last_fullsized":"2024-09-13T15:11:54.327198+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:11.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":473,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327636+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:11:54.327636+0000","last_peered":"2024-09-13T15:11:54.327636+0000","last_clean":"2024-09-13T15:11:54.327636+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:11:54.327636+0000","last_undegraded":"2024-09-13T15:11:54.327636+0000","last_fullsized":"2024-09-13T15:11:54.327636+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":341,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.327047+0000","last_change":"2024-09-13T15:08:46.079600+0000","last_active":"2024-09-13T15:11:54.327047+0000","last_peered":"2024-09-13T15:11:54.327047+0000","last_clean":"2024-09-13T15:11:54.327047+0000","last_became_active":"2024-09-13T15:08:46.075458+0000","last_became_peered":"2024-09-13T15:08:46.075458+0000","last_unstale":"2024-09-13T15:11:54.327047+0000","last_undegraded":"2024-09-13T15:11:54.327047+0000","last_fullsized":"2024-09-13T15:11:54.327047+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:41.707452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":828,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.341918+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:12:12.341918+0000","last_peered":"2024-09-13T15:12:12.341918+0000","last_clean":"2024-09-13T15:12:12.341918+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:12:12.341918+0000","last_undegraded":"2024-09-13T15:12:12.341918+0000","last_fullsized":"2024-09-13T15:12:12.341918+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":851,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.342320+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:12:12.342320+0000","last_peered":"2024-09-13T15:12:12.342320+0000","last_clean":"2024-09-13T15:12:12.342320+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:12:12.342320+0000","last_undegraded":"2024-09-13T15:12:12.342320+0000","last_fullsized":"2024-09-13T15:12:12.342320+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":507,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-13T15:11:54.326232+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:11:54.326232+0000","last_peered":"2024-09-13T15:11:54.326232+0000","last_clean":"2024-09-13T15:11:54.326232+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:11:54.326232+0000","last_undegraded":"2024-09-13T15:11:54.326232+0000","last_fullsized":"2024-09-13T15:11:54.326232+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":818,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-09-13T15:12:12.337846+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:12:12.337846+0000","last_peered":"2024-09-13T15:12:12.337846+0000","last_clean":"2024-09-13T15:12:12.337846+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:12:12.337846+0000","last_undegraded":"2024-09-13T15:12:12.337846+0000","last_fullsized":"2024-09-13T15:12:12.337846+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3227254808,"num_objects":49245,"num_object_clones":0,"num_object_copies":147735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202103,"num_write_kb":8043073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9706811392,"data_stored":9706799176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131603,"ondisk_log_size":131603,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":36578,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234540,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299748,"kb_used_data":795292,"kb_used_omap":1,"kb_used_meta":504446,"kb_avail":92445404,"statfs":{"total":95995035648,"available":94664093696,"internally_reserved":0,"allocated":814379008,"data_stored":811795785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":516553163},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496176,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2086044,"kb_used_data":1387156,"kb_used_omap":1,"kb_used_meta":698878,"kb_avail":91659108,"statfs":{"total":95995035648,"available":93858926592,"internally_reserved":0,"allocated":1420447744,"data_stored":1417872337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":715651527},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757811,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1816536,"kb_used_data":977360,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":91928616,"statfs":{"total":95995035648,"available":94134902784,"internally_reserved":0,"allocated":1000816640,"data_stored":998242006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":859306439},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019451,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2400228,"kb_used_data":1570908,"kb_used_omap":1,"kb_used_meta":829310,"kb_avail":91344924,"statfs":{"total":95995035648,"available":93537202176,"internally_reserved":0,"allocated":1608609792,"data_stored":1606026832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":849213894},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468335,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1991380,"kb_used_data":1043468,"kb_used_omap":4,"kb_used_meta":947899,"kb_avail":91753772,"statfs":{"total":95995035648,"available":93955862528,"internally_reserved":0,"allocated":1068511232,"data_stored":1065940398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":970648740},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273235,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1605212,"kb_used_data":1196500,"kb_used_omap":12,"kb_used_meta":408627,"kb_avail":92139940,"statfs":{"total":95995035648,"available":94351298560,"internally_reserved":0,"allocated":1225216000,"data_stored":1222637522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":418434944},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795388,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1465744,"kb_used_data":591368,"kb_used_omap":7,"kb_used_meta":874360,"kb_avail":92279408,"statfs":{"total":95995035648,"available":94494113792,"internally_reserved":0,"allocated":605560832,"data_stored":602993479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":895345266},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502169,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583384,"kb_used_data":2036752,"kb_used_omap":10,"kb_used_meta":546613,"kb_avail":91161768,"statfs":{"total":95995035648,"available":93349650432,"internally_reserved":0,"allocated":2085634048,"data_stored":2083059112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":559732415},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2065436672,"data_stored":2065432600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":594870272,"data_stored":594870272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1207369728,"data_stored":1207369728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1049821184,"data_stored":1049821184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1593380864,"data_stored":1593376792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":993460224,"data_stored":993460224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1400508416,"data_stored":1400504344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":801964032,"data_stored":801964032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:12:28.033 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T15:12:28.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-13T15:12:28.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:28.133 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[102314]: from='client.46346 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:28.133 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:28.133 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:28.133 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[102314]: pgmap v1536: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-09-13T15:12:28.133 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2239051127' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:12:28.133 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:12:28.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[107038]: from='client.46346 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:28.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:28.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:28.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[107038]: pgmap v1536: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-09-13T15:12:28.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2239051127' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:12:28.134 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:27 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:12:28.142 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:27 smithi172 ceph-mon[100696]: from='client.46346 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:12:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:27 smithi172 ceph-mon[100696]: pgmap v1536: 99 pgs: 99 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-09-13T15:12:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2239051127' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:12:28.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:27 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:12:28.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:28.251 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:12:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.276 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:28.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:28.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:28.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:28.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:28.543 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:28.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.684 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:28.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:28.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:28.802 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:28.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-13T15:12:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.844 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:28.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:28.945 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:28.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:28.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:29.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:29.125 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:29.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:29.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[102314]: osdmap e473: 8 total, 8 up, 8 in 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[107038]: osdmap e473: 8 total, 8 up, 8 in 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:29.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:29.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:28 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:12:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:28 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:12:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:28 smithi172 ceph-mon[100696]: osdmap e473: 8 total, 8 up, 8 in 2024-09-13T15:12:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:29.254 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:29.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:29.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:29.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:29.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:29.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:29.534 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:29.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:29.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:29.670 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:29.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:29.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:29.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:29.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:29.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:29.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:29.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:30.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:30.065 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:30.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:30.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:30.113 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:29 smithi154 ceph-mon[102314]: pgmap v1538: 99 pgs: 99 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:12:30.114 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:29 smithi154 ceph-mon[107038]: pgmap v1538: 99 pgs: 99 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:12:30.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:30.239 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:30.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:29 smithi172 ceph-mon[100696]: pgmap v1538: 99 pgs: 99 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 815 op/s 2024-09-13T15:12:30.341 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:30.377 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:30.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:30.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:30.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:30.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:30.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:30.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:30.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:30.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:30.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:30.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:30.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:30.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:30.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:31.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:31.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:31.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:31.155 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:31.184 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:31.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:31.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:12:31.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:31.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:31.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:31.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:31.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:31.500 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:31.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:31.551 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:31 smithi154 ceph-mon[102314]: pgmap v1539: 99 pgs: 99 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-13T15:12:31.552 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:31 smithi154 ceph-mon[107038]: pgmap v1539: 99 pgs: 99 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-13T15:12:31.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:31.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:31.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:31 smithi172 ceph-mon[100696]: pgmap v1539: 99 pgs: 99 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-13T15:12:31.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:31.780 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:31.787 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:31.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:31.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:31.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:31.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:32.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:32.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:32.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:32.125 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:12:32.125 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:12:32.158 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:12:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:12:31] "GET /metrics HTTP/1.1" 200 36645 "" "Prometheus/2.51.0" 2024-09-13T15:12:32.163 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:12:32.163 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:12:32.189 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:32.205 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:12:32.205 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:12:32.217 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:32.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:32.245 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:12:32.246 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:12:32.276 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:12:32.277 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:12:32.306 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:12:32.307 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:12:32.319 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:32.345 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:12:32.345 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:12:32.348 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:32.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:32.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:32.384 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:12:32.385 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:12:32.413 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1326164031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:32.413 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1326164031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:32.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:32.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:32.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:32.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:32.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:32.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:32.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1326164031' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:32.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:32.773 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:32.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:32.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:32.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:32.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:33.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:33.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:33.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:33.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:33.180 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:33.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:33.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:33.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:33.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:33.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:33.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:33.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:33.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:33.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:33.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:33.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:33.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:33.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:33 smithi154 ceph-mon[102314]: pgmap v1540: 99 pgs: 99 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:12:33.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:33 smithi154 ceph-mon[107038]: pgmap v1540: 99 pgs: 99 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:12:33.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:33.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:33 smithi172 ceph-mon[100696]: pgmap v1540: 99 pgs: 99 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:12:33.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:33.872 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:33.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:33.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:34.011 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:34.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:34.113 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:34.150 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:34.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:34.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:34.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.418 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:34.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:34.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:34.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:34.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:34.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:34.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:34.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:34.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:34.976 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:34.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:35.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:35.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:35.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:35.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:35.246 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:35.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:35.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:35.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:35.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:35.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:35 smithi154 ceph-mon[102314]: pgmap v1541: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-13T15:12:35.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:35 smithi154 ceph-mon[107038]: pgmap v1541: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-13T15:12:35.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:35.520 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:35.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:35.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:35.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:35.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:35 smithi172 ceph-mon[100696]: pgmap v1541: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 828 op/s 2024-09-13T15:12:35.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:35.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:35.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:35.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:35.926 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:35.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:36.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:36.054 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:36.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:36.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:36.187 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:36.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:36.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:36.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:36.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:36.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:36.474 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:36.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:36.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:36.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:36.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:36.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:36.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:36.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:36.856 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:36.884 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 1 pg_num to 21 2024-09-13T15:12:36.889 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:36.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:36.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:37.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:37.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:37.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2989489474' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T15:12:37.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:36 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T15:12:37.141 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2989489474' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T15:12:37.141 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:36 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T15:12:37.163 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2989489474' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T15:12:37.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:36 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-13T15:12:37.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:37.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:37.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:37.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:37.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:37.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:37.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:37.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:37.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:12:37.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:37.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:37.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:37.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:37.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:38.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:38.117 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T15:12:38.117 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[102314]: osdmap e474: 8 total, 8 up, 8 in 2024-09-13T15:12:38.117 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[102314]: pgmap v1543: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:12:38.118 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T15:12:38.118 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T15:12:38.118 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[107038]: osdmap e474: 8 total, 8 up, 8 in 2024-09-13T15:12:38.118 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[107038]: pgmap v1543: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:12:38.118 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:37 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T15:12:38.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:38.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:38.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:38.158 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:12:38.158 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:12:38.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:37 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-13T15:12:38.250 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:37 smithi172 ceph-mon[100696]: osdmap e474: 8 total, 8 up, 8 in 2024-09-13T15:12:38.251 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:37 smithi172 ceph-mon[100696]: pgmap v1543: 99 pgs: 99 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-13T15:12:38.251 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:37 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-13T15:12:38.251 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:38.282 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:38.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:38.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:38.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:38.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:38.535 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:38.567 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:38.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:38.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:38.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:38.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:38.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:38.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:38.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-13T15:12:38.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:38.974 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:38.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:39.076 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:39.098 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:39.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:39.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:39.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:38 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T15:12:39.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:38 smithi154 ceph-mon[102314]: osdmap e475: 8 total, 8 up, 8 in 2024-09-13T15:12:39.222 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:38 smithi154 ceph-mon[102314]: osdmap e476: 8 total, 8 up, 8 in 2024-09-13T15:12:39.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:38 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T15:12:39.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:38 smithi154 ceph-mon[107038]: osdmap e475: 8 total, 8 up, 8 in 2024-09-13T15:12:39.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:38 smithi154 ceph-mon[107038]: osdmap e476: 8 total, 8 up, 8 in 2024-09-13T15:12:39.235 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:39.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:38 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-13T15:12:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:38 smithi172 ceph-mon[100696]: osdmap e475: 8 total, 8 up, 8 in 2024-09-13T15:12:39.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:38 smithi172 ceph-mon[100696]: osdmap e476: 8 total, 8 up, 8 in 2024-09-13T15:12:39.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:39.374 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:39.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:39.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:39.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:39.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:39.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:39.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:39.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:39.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:39.792 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:39.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:39.894 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:39.936 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:39.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:40.037 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:40.072 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:40.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:40.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:40.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:40.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:40.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:39 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:12:40.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:39 smithi154 ceph-mon[102314]: 1.3 scrub starts 2024-09-13T15:12:40.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:39 smithi154 ceph-mon[102314]: pgmap v1546: 109 pgs: 2 peering, 8 unknown, 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T15:12:40.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:39 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:12:40.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:39 smithi154 ceph-mon[107038]: 1.3 scrub starts 2024-09-13T15:12:40.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:39 smithi154 ceph-mon[107038]: pgmap v1546: 109 pgs: 2 peering, 8 unknown, 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T15:12:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:39 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:12:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:39 smithi172 ceph-mon[100696]: 1.3 scrub starts 2024-09-13T15:12:40.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:39 smithi172 ceph-mon[100696]: pgmap v1546: 109 pgs: 2 peering, 8 unknown, 99 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-13T15:12:40.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:40.350 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:40.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:40.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:40.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:40.490 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:40.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:40.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:40.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:40.623 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:40.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:40.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:40.763 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:40.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:40.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:40.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:40.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:41.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:41.038 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:41.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:41.167 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:41.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:40 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:12:41.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:40 smithi154 ceph-mon[102314]: 1.e scrub starts 2024-09-13T15:12:41.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:40 smithi154 ceph-mon[102314]: 1.13 scrub starts 2024-09-13T15:12:41.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:40 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:12:41.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:40 smithi154 ceph-mon[107038]: 1.e scrub starts 2024-09-13T15:12:41.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:40 smithi154 ceph-mon[107038]: 1.13 scrub starts 2024-09-13T15:12:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:40 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:12:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:40 smithi172 ceph-mon[100696]: 1.e scrub starts 2024-09-13T15:12:41.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:40 smithi172 ceph-mon[100696]: 1.13 scrub starts 2024-09-13T15:12:41.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:41.303 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:41.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:41.435 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:41.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:41.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:41.676 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:41.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:41.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:41.840 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:41.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:41.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:41.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:42.035 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:42.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:42.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:42.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:42.173 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:41 smithi154 ceph-mon[102314]: 1.14 scrub starts 2024-09-13T15:12:42.174 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:41 smithi154 ceph-mon[102314]: pgmap v1547: 109 pgs: 1 active+clean+scrubbing, 2 peering, 6 unknown, 100 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-13T15:12:42.174 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:41 smithi154 ceph-mon[102314]: 1.b scrub starts 2024-09-13T15:12:42.174 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:12:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:12:41] "GET /metrics HTTP/1.1" 200 36645 "" "Prometheus/2.51.0" 2024-09-13T15:12:42.174 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:41 smithi154 ceph-mon[107038]: 1.14 scrub starts 2024-09-13T15:12:42.174 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:41 smithi154 ceph-mon[107038]: pgmap v1547: 109 pgs: 1 active+clean+scrubbing, 2 peering, 6 unknown, 100 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-13T15:12:42.174 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:41 smithi154 ceph-mon[107038]: 1.b scrub starts 2024-09-13T15:12:42.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:41 smithi172 ceph-mon[100696]: 1.14 scrub starts 2024-09-13T15:12:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:41 smithi172 ceph-mon[100696]: pgmap v1547: 109 pgs: 1 active+clean+scrubbing, 2 peering, 6 unknown, 100 active+clean; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-13T15:12:42.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:41 smithi172 ceph-mon[100696]: 1.b scrub starts 2024-09-13T15:12:42.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:42.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:42.373 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:42.385 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:42.414 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:42.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:42.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:42.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:42.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:42.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:42.702 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:42.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:42.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:42.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:42.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:42.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:42.851 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:12:42.851 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:12:42.852 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-13T15:12:42.852 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T15:12:42.852 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T15:12:42.852 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-13T15:12:42.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:42.977 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:42.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:43.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:43.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:43.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:43.176 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:12:43.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:43.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:42 smithi154 ceph-mon[102314]: 1.f scrub starts 2024-09-13T15:12:43.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:42 smithi154 ceph-mon[107038]: 1.f scrub starts 2024-09-13T15:12:43.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:42 smithi172 ceph-mon[100696]: 1.f scrub starts 2024-09-13T15:12:43.245 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:43.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:43.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:43.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:43.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:43.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:43.524 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:43.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:43.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:43.662 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:43.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:43.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:43.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:43.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:43.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:43.941 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:43.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:44.036 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:44.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:44.106 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:44.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:44.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:44.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:43 smithi154 ceph-mon[102314]: 1.4 scrub starts 2024-09-13T15:12:44.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:44.223 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:43 smithi154 ceph-mon[102314]: pgmap v1548: 109 pgs: 3 active+clean+scrubbing, 2 peering, 2 unknown, 102 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 827 op/s 2024-09-13T15:12:44.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:43 smithi154 ceph-mon[107038]: 1.4 scrub starts 2024-09-13T15:12:44.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:44.223 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:43 smithi154 ceph-mon[107038]: pgmap v1548: 109 pgs: 3 active+clean+scrubbing, 2 peering, 2 unknown, 102 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 827 op/s 2024-09-13T15:12:44.240 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:44.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:44.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:43 smithi172 ceph-mon[100696]: 1.4 scrub starts 2024-09-13T15:12:44.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:44.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:43 smithi172 ceph-mon[100696]: pgmap v1548: 109 pgs: 3 active+clean+scrubbing, 2 peering, 2 unknown, 102 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 827 op/s 2024-09-13T15:12:44.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:44.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:44.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:44.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:44.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:44.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:44.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:44.690 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:44.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:44.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:44.823 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:44.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:44.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:44.955 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:44.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:45.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:45.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:45.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[102314]: 1.c scrub starts 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[102314]: 1.3 scrub starts 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[107038]: 1.c scrub starts 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:45.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:12:45.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:45.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:45 smithi154 ceph-mon[107038]: 1.3 scrub starts 2024-09-13T15:12:45.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:45.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:45.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:45.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:45.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:45 smithi172 ceph-mon[100696]: 1.c scrub starts 2024-09-13T15:12:45.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:45.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:45 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:12:45.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:45 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:45.389 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:45 smithi172 ceph-mon[100696]: 1.3 scrub starts 2024-09-13T15:12:45.402 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:45.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:45.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:45.546 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:45.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:45.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:45.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:45.677 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:45.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:45.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:45.810 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:45.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:45.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:45.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:45.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:46.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:46.084 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:46.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:46.119 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:12:46.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:46.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:46.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:46.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:46.383 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:46.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:46.454 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:46 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:12:46.454 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:46 smithi172 ceph-mon[100696]: pgmap v1549: 109 pgs: 5 active+clean+scrubbing, 104 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T15:12:46.454 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:46 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T15:12:46.454 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:46 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1140329001' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:46.454 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:46 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:46.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:12:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[102314]: pgmap v1549: 109 pgs: 5 active+clean+scrubbing, 104 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T15:12:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T15:12:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1140329001' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:46.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:12:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[107038]: pgmap v1549: 109 pgs: 5 active+clean+scrubbing, 104 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-13T15:12:46.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-13T15:12:46.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1140329001' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:46.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:46 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:12:46.485 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:46.519 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:46.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:46.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:46.664 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:46.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:46.765 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:46.807 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:46.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:46.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:46.947 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:46.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:46.977 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:12:46.977 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "" 2024-09-13T15:12:46.977 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:12:47.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:47.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:47.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:47.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[102314]: 1.14 scrub starts 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[102314]: 1.e scrub starts 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[102314]: osdmap e477: 8 total, 8 up, 8 in 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[102314]: 1.3 scrub starts 2024-09-13T15:12:47.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[107038]: 1.14 scrub starts 2024-09-13T15:12:47.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:12:47.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[107038]: 1.e scrub starts 2024-09-13T15:12:47.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T15:12:47.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:12:47.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[107038]: osdmap e477: 8 total, 8 up, 8 in 2024-09-13T15:12:47.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:47 smithi154 ceph-mon[107038]: 1.3 scrub starts 2024-09-13T15:12:47.232 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:47.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:47 smithi172 ceph-mon[100696]: 1.14 scrub starts 2024-09-13T15:12:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:47 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:12:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:47 smithi172 ceph-mon[100696]: 1.e scrub starts 2024-09-13T15:12:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:47 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-13T15:12:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:47 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:12:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:47 smithi172 ceph-mon[100696]: osdmap e477: 8 total, 8 up, 8 in 2024-09-13T15:12:47.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:47 smithi172 ceph-mon[100696]: 1.3 scrub starts 2024-09-13T15:12:47.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:47.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:47.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:47.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:47.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:47.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:47.616 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:47.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:47.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:47.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:47.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:47.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:47.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:47.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:47.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:48.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:48.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:48.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:48.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:48.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:48.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:48.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:48.351 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[102314]: 1.4 scrub starts 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[102314]: osdmap e478: 8 total, 8 up, 8 in 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[102314]: pgmap v1552: 109 pgs: 1 peering, 4 active+clean+scrubbing, 104 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[107038]: 1.4 scrub starts 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[107038]: osdmap e478: 8 total, 8 up, 8 in 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:12:48.352 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[107038]: pgmap v1552: 109 pgs: 1 peering, 4 active+clean+scrubbing, 104 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T15:12:48.353 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:48.353 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:48 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:48.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:48.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:48.468 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:48 smithi172 ceph-mon[100696]: 1.4 scrub starts 2024-09-13T15:12:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:48 smithi172 ceph-mon[100696]: osdmap e478: 8 total, 8 up, 8 in 2024-09-13T15:12:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:48 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:12:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:48 smithi172 ceph-mon[100696]: pgmap v1552: 109 pgs: 1 peering, 4 active+clean+scrubbing, 104 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-13T15:12:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:48 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:48 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:12:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:48.497 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:48.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:48.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:48.635 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:48.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:48.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:48.766 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:48.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:48.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:48.909 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:48.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:49.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:49.041 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:49.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:49.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:49.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:49 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:12:49.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:49 smithi154 ceph-mon[102314]: osdmap e479: 8 total, 8 up, 8 in 2024-09-13T15:12:49.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:49 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:12:49.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:49 smithi154 ceph-mon[107038]: osdmap e479: 8 total, 8 up, 8 in 2024-09-13T15:12:49.288 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.323 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:49.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:49.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:49.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:49 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:12:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:49 smithi172 ceph-mon[100696]: osdmap e479: 8 total, 8 up, 8 in 2024-09-13T15:12:49.534 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:12:49.534 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:12:49.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:49.591 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:49.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:49.720 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:49.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:49.822 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:49.845 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:49.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:49.975 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:49.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:50.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:50.105 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:50.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:50.207 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:50.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-13T15:12:50.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:50.232 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:50.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:50.334 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:50.360 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:50.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:50.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:12:50.452 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:50 smithi154 ceph-mon[102314]: osdmap e480: 8 total, 8 up, 8 in 2024-09-13T15:12:50.452 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:50 smithi154 ceph-mon[102314]: pgmap v1555: 109 pgs: 4 remapped+peering, 2 peering, 4 active+clean+scrubbing, 99 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 28 KiB/s rd, 82 KiB/s wr, 935 op/s; 138 KiB/s, 0 objects/s recovering 2024-09-13T15:12:50.453 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:50 smithi154 ceph-mon[107038]: osdmap e480: 8 total, 8 up, 8 in 2024-09-13T15:12:50.453 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:50 smithi154 ceph-mon[107038]: pgmap v1555: 109 pgs: 4 remapped+peering, 2 peering, 4 active+clean+scrubbing, 99 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 28 KiB/s rd, 82 KiB/s wr, 935 op/s; 138 KiB/s, 0 objects/s recovering 2024-09-13T15:12:50.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:50 smithi172 ceph-mon[100696]: osdmap e480: 8 total, 8 up, 8 in 2024-09-13T15:12:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:50 smithi172 ceph-mon[100696]: pgmap v1555: 109 pgs: 4 remapped+peering, 2 peering, 4 active+clean+scrubbing, 99 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 28 KiB/s rd, 82 KiB/s wr, 935 op/s; 138 KiB/s, 0 objects/s recovering 2024-09-13T15:12:50.509 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:50.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:50.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:50.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:50.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:50.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:50.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:50.780 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:50.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:50.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:50.904 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:12:50.905 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:12:50.951 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:12:50.951 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:12:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:50.957 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:50.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:50.985 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:12:50.986 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:12:51.034 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:12:51.034 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:12:51.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:51.072 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:12:51.072 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:12:51.102 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:12:51.102 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:12:51.104 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:51.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:51.135 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:12:51.135 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:12:51.168 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:12:51.168 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:12:51.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:51.234 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:51.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:51.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:12:51.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[102314]: osdmap e481: 8 total, 8 up, 8 in 2024-09-13T15:12:51.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[102314]: 1.b scrub starts 2024-09-13T15:12:51.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[102314]: 1.11 scrub starts 2024-09-13T15:12:51.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3445966535' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:51.266 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[102314]: 1.14 scrub starts 2024-09-13T15:12:51.266 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:12:51.267 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[107038]: osdmap e481: 8 total, 8 up, 8 in 2024-09-13T15:12:51.267 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[107038]: 1.b scrub starts 2024-09-13T15:12:51.267 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[107038]: 1.11 scrub starts 2024-09-13T15:12:51.267 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3445966535' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:51.267 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-mon[107038]: 1.14 scrub starts 2024-09-13T15:12:51.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:51.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:51.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:51.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:51 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-13T15:12:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:51 smithi172 ceph-mon[100696]: osdmap e481: 8 total, 8 up, 8 in 2024-09-13T15:12:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:51 smithi172 ceph-mon[100696]: 1.b scrub starts 2024-09-13T15:12:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:51 smithi172 ceph-mon[100696]: 1.11 scrub starts 2024-09-13T15:12:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3445966535' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:12:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:51 smithi172 ceph-mon[100696]: 1.14 scrub starts 2024-09-13T15:12:51.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:51.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:51.528 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:51.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:51.644 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:51.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:51.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:51.777 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:51.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:51.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:51.909 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:51.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:51.967 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:52.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:52.049 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:52.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:52.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:52.165 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:12:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:12:51] "GET /metrics HTTP/1.1" 200 36646 "" "Prometheus/2.51.0" 2024-09-13T15:12:52.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:52 smithi154 ceph-mon[107038]: 1.c scrub starts 2024-09-13T15:12:52.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:52 smithi154 ceph-mon[107038]: 1.13 scrub starts 2024-09-13T15:12:52.165 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:52 smithi154 ceph-mon[107038]: pgmap v1557: 109 pgs: 4 remapped+peering, 4 peering, 1 active+clean+scrubbing, 100 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 27 KiB/s rd, 80 KiB/s wr, 966 op/s; 133 KiB/s, 0 objects/s recovering 2024-09-13T15:12:52.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:52.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:52.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:52.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:52.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:52.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:52.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:52.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:52 smithi154 ceph-mon[102314]: 1.c scrub starts 2024-09-13T15:12:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:52 smithi154 ceph-mon[102314]: 1.13 scrub starts 2024-09-13T15:12:52.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:52 smithi154 ceph-mon[102314]: pgmap v1557: 109 pgs: 4 remapped+peering, 4 peering, 1 active+clean+scrubbing, 100 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 27 KiB/s rd, 80 KiB/s wr, 966 op/s; 133 KiB/s, 0 objects/s recovering 2024-09-13T15:12:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:52 smithi172 ceph-mon[100696]: 1.c scrub starts 2024-09-13T15:12:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:52 smithi172 ceph-mon[100696]: 1.13 scrub starts 2024-09-13T15:12:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:52 smithi172 ceph-mon[100696]: pgmap v1557: 109 pgs: 4 remapped+peering, 4 peering, 1 active+clean+scrubbing, 100 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 27 KiB/s rd, 80 KiB/s wr, 966 op/s; 133 KiB/s, 0 objects/s recovering 2024-09-13T15:12:52.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:52.598 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:52.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:52.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 3, 4, 5, 6, 7, 0, 2] 2024-09-13T15:12:52.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T15:12:52.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:52.730 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:52.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:52.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:52.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:52.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:52.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:53.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:53.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:53.157 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:53.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:53.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:53 smithi154 ceph-mon[102314]: 1.4 scrub starts 2024-09-13T15:12:53.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:53 smithi154 ceph-mon[102314]: 1.3 scrub starts 2024-09-13T15:12:53.263 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:53 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:12:53.263 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:53 smithi154 ceph-mon[107038]: 1.4 scrub starts 2024-09-13T15:12:53.263 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:53 smithi154 ceph-mon[107038]: 1.3 scrub starts 2024-09-13T15:12:53.263 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:53 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:12:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.291 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:53.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:53.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:53.426 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:53 smithi172 ceph-mon[100696]: 1.4 scrub starts 2024-09-13T15:12:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:53 smithi172 ceph-mon[100696]: 1.3 scrub starts 2024-09-13T15:12:53.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:53 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:12:53.529 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:53.558 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:53.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:53.694 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:53.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:53.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:53.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:53.985 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:53.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:54.087 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:54.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:54.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:54.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:54.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:54 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:12:54.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:54 smithi154 ceph-mon[102314]: pgmap v1558: 109 pgs: 3 active+clean+scrubbing, 3 remapped+peering, 3 peering, 100 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 903 op/s; 145 KiB/s, 0 objects/s recovering 2024-09-13T15:12:54.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:54 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:12:54.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:54 smithi154 ceph-mon[107038]: pgmap v1558: 109 pgs: 3 active+clean+scrubbing, 3 remapped+peering, 3 peering, 100 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 903 op/s; 145 KiB/s, 0 objects/s recovering 2024-09-13T15:12:54.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:54.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:54.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:54.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:54.421 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:54.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:54 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:12:54.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:54 smithi172 ceph-mon[100696]: pgmap v1558: 109 pgs: 3 active+clean+scrubbing, 3 remapped+peering, 3 peering, 100 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 903 op/s; 145 KiB/s, 0 objects/s recovering 2024-09-13T15:12:54.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:54.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:12:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:54.549 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:54.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:54.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:54.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:54.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:54.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:54.825 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:54.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:54.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:54.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:54.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:55.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:55.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:55.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:55.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:55.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:55.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:55.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:55.387 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:55.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:55.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:55.517 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:55.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:55.565 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:55 smithi154 ceph-mon[102314]: pgmap v1559: 109 pgs: 7 active+clean+scrubbing, 102 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 5.3 KiB/s rd, 15 KiB/s wr, 1.08k op/s; 71 KiB/s, 0 objects/s recovering 2024-09-13T15:12:55.565 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:55 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T15:12:55.565 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:55 smithi154 ceph-mon[107038]: pgmap v1559: 109 pgs: 7 active+clean+scrubbing, 102 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 5.3 KiB/s rd, 15 KiB/s wr, 1.08k op/s; 71 KiB/s, 0 objects/s recovering 2024-09-13T15:12:55.565 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:55 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T15:12:55.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:55.657 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:55.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:55 smithi172 ceph-mon[100696]: pgmap v1559: 109 pgs: 7 active+clean+scrubbing, 102 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 5.3 KiB/s rd, 15 KiB/s wr, 1.08k op/s; 71 KiB/s, 0 objects/s recovering 2024-09-13T15:12:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:55 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-13T15:12:55.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:55.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:55.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:55.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:55.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:55.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:56.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:56.058 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:56.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:56.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:56.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:56.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:56.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:56.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:56.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:56.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:56.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:56.457 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:56.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:56 smithi154 ceph-mon[102314]: 1.14 scrub starts 2024-09-13T15:12:56.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:56 smithi154 ceph-mon[102314]: 1.c scrub starts 2024-09-13T15:12:56.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:56 smithi154 ceph-mon[107038]: 1.14 scrub starts 2024-09-13T15:12:56.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:56 smithi154 ceph-mon[107038]: 1.c scrub starts 2024-09-13T15:12:56.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:56.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:56.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:56.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:56.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:56.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:56 smithi172 ceph-mon[100696]: 1.14 scrub starts 2024-09-13T15:12:56.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:56 smithi172 ceph-mon[100696]: 1.c scrub starts 2024-09-13T15:12:56.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:56.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:56.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:56.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:57.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:57.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:57.146 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:57.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:57.277 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:57.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:57.379 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.407 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:57.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:57.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[102314]: 1.11 scrub starts 2024-09-13T15:12:57.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[102314]: 1.b scrub starts 2024-09-13T15:12:57.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[102314]: 1.4 scrub starts 2024-09-13T15:12:57.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[102314]: pgmap v1560: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 4.7 KiB/s rd, 14 KiB/s wr, 815 op/s; 62 KiB/s, 0 objects/s recovering 2024-09-13T15:12:57.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[107038]: 1.11 scrub starts 2024-09-13T15:12:57.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[107038]: 1.b scrub starts 2024-09-13T15:12:57.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[107038]: 1.4 scrub starts 2024-09-13T15:12:57.472 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:57 smithi154 ceph-mon[107038]: pgmap v1560: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 4.7 KiB/s rd, 14 KiB/s wr, 815 op/s; 62 KiB/s, 0 objects/s recovering 2024-09-13T15:12:57.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:57.538 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:57.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:57.667 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:57 smithi172 ceph-mon[100696]: 1.11 scrub starts 2024-09-13T15:12:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:57 smithi172 ceph-mon[100696]: 1.b scrub starts 2024-09-13T15:12:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:57 smithi172 ceph-mon[100696]: 1.4 scrub starts 2024-09-13T15:12:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:57 smithi172 ceph-mon[100696]: pgmap v1560: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 4.7 KiB/s rd, 14 KiB/s wr, 815 op/s; 62 KiB/s, 0 objects/s recovering 2024-09-13T15:12:57.770 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:57.799 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:57.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:57.957 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:57.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:58.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:58.087 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:58.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:58.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:12:58.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:58.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:12:58.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:58.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:58.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:58.473 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:12:58.502 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:58.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:12:58.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:12:58.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:58 smithi154 ceph-mon[102314]: 1.13 scrub starts 2024-09-13T15:12:58.637 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:58.637 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:58 smithi154 ceph-mon[107038]: 1.13 scrub starts 2024-09-13T15:12:58.637 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:58.647 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:58.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:12:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:58 smithi172 ceph-mon[100696]: 1.13 scrub starts 2024-09-13T15:12:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:12:58.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:58.749 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:12:58.807 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:12:58.807 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:12:58.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.807 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:233425 10.7 10:ebd7e66b:::benchmark_data_smithi154_317818_object107749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T15:12:58.807 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:58.770331+0000", 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.032980561999999998, 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033011667000000001, 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 233425 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:58.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.770331+0000", 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.770331+0000", 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.770333+0000", 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2419999999999999e-06 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.770341+0000", 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7570000000000004e-06 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.770364+0000", 2024-09-13T15:12:58.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3346e-05 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.770374+0000", 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6970000000000005e-06 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.774215+0000", 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038412279999999999 2024-09-13T15:12:58.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.774330+0000", 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114763 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.774441+0000", 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0001111 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.788340+0000", 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013898915 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.794401+0000", 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060605090000000004 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:233431 10.f 10:f0598f18:::benchmark_data_smithi154_317818_object107755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:58.780943+0000", 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.022368312000000001, 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022491690000000002, 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 233431 2024-09-13T15:12:58.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.780943+0000", 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.780943+0000", 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:58.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.780945+0000", 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.193e-06 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.780952+0000", 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3559999999999998e-06 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.780970+0000", 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7924e-05 2024-09-13T15:12:58.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.780979+0000", 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6919999999999997e-06 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781117+0000", 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137856 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:12:58.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781244+0000", 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012700400000000001 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781364+0000", 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011937100000000001 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.794583+0000", 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013219876 2024-09-13T15:12:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.794615+0000", 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1164000000000002e-05 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:233432 10.f 10:ff229d60:::benchmark_data_smithi154_317818_object107756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:58.781001+0000", 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.022310177, 2024-09-13T15:12:58.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022509946999999999, 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 233432 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781001+0000", 2024-09-13T15:12:58.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781001+0000", 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781002+0000", 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8500000000000001e-07 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781004+0000", 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4219999999999999e-06 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781011+0000", 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5289999999999997e-06 2024-09-13T15:12:58.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.781016+0000", 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8389999999999997e-06 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.794643+0000", 2024-09-13T15:12:58.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013627425 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.794782+0000", 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000139228 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.794923+0000", 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014060099999999999 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.824 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:233436 10.f 10:fbdc7999:::benchmark_data_smithi154_317818_object107760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:58.783218+0000", 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.020093284999999999, 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020359805000000002, 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:12:58.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 233436 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.783218+0000", 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.783218+0000", 2024-09-13T15:12:58.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.783220+0000", 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.44e-06 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.783225+0000", 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0699999999999997e-06 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:58.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.783240+0000", 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5182e-05 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.783251+0000", 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0947e-05 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.798890+0000", 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015639184 2024-09-13T15:12:58.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799185+0000", 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029516600000000003 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799317+0000", 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000131676 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(client.36270.0:233432 10.f e481/448)", 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:58.799179+0000", 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0041324689999999997, 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044587120000000001, 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:58.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799179+0000", 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799179+0000", 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799180+0000", 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.16e-07 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799189+0000", 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9369999999999993e-06 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799201+0000", 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2088e-05 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:58.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.799213+0000", 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.155e-05 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:233448 10.7 10.168c087 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e481)", 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:58.801952+0000", 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.001359814, 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00172573, 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:58.833 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 233448 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.801952+0000", 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:58.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.801952+0000", 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.801954+0000", 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8190000000000001e-06 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.801960+0000", 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8639999999999998e-06 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.801979+0000", 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8800999999999998e-05 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.801988+0000", 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6880000000000005e-06 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.36270.0:233443 10.3 e481/448)", 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:58.802033+0000", 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.0012781470000000001, 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001687518, 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.802033+0000", 2024-09-13T15:12:58.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.802033+0000", 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.802034+0000", 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2099999999999996e-07 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:12:58.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.802038+0000", 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9879999999999998e-06 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.802046+0000", 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2160000000000006e-06 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:58.802051+0000", 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0479999999999998e-06 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.839 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:12:58.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:12:58.840 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T15:12:58.840 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 7 2024-09-13T15:12:58.840 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:12:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:58.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:58.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:58.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:12:58.987 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:58.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:12:59.089 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:12:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:59.121 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:59.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:12:59.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:59.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:59.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:12:59.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:59.392 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:59.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:59.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:59.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:59.521 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:12:59.521 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[102314]: 1.3 scrub starts 2024-09-13T15:12:59.522 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:12:59.522 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[102314]: pgmap v1561: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 873 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T15:12:59.522 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:12:59.523 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[107038]: 1.3 scrub starts 2024-09-13T15:12:59.523 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:12:59.523 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:12:59 smithi154 ceph-mon[107038]: pgmap v1561: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 873 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T15:12:59.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:59.524 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:59.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:59.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:12:59.654 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:12:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:59.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:12:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:59 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:12:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:59 smithi172 ceph-mon[100696]: 1.3 scrub starts 2024-09-13T15:12:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:59 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:12:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:12:59 smithi172 ceph-mon[100696]: pgmap v1561: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 873 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T15:12:59.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:12:59.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:59.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:12:59.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:12:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:12:59.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:12:59.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:00.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:00.074 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:00.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:00.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:00.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:00.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:00.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:00.341 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:00.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:00.378 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T15:13:00.379 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:13:00.403 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2038838436' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:13:00.403 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3888383036' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:00.403 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:00 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:00.404 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2038838436' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:13:00.404 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3888383036' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:00.404 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:00 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:00.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:00.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:00.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:00.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:00.622 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:00.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:00.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2038838436' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:13:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3888383036' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:00.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:00 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:00.757 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:00.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:00.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:00.896 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:00.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:00.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:01.033 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:01.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:01.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:01.274 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:01.303 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:01.405 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:01.437 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:01.496 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:01.496 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2038838436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:13:01.497 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:13:01.497 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: osdmap e482: 8 total, 8 up, 8 in 2024-09-13T15:13:01.497 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: 1.5 scrub starts 2024-09-13T15:13:01.497 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: 1.5 scrub ok 2024-09-13T15:13:01.497 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: 1.14 scrub starts 2024-09-13T15:13:01.497 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: 1.14 scrub ok 2024-09-13T15:13:01.497 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[107038]: pgmap v1563: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 939 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2038838436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: osdmap e482: 8 total, 8 up, 8 in 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: 1.5 scrub starts 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: 1.5 scrub ok 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: 1.14 scrub starts 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: 1.14 scrub ok 2024-09-13T15:13:01.498 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-mon[102314]: pgmap v1563: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 939 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T15:13:01.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-13T15:13:01.538 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.568 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:01.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:01.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:01.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2038838436' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:13:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:13:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: osdmap e482: 8 total, 8 up, 8 in 2024-09-13T15:13:01.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: 1.5 scrub starts 2024-09-13T15:13:01.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: 1.5 scrub ok 2024-09-13T15:13:01.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: 1.14 scrub starts 2024-09-13T15:13:01.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: 1.14 scrub ok 2024-09-13T15:13:01.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:01 smithi172 ceph-mon[100696]: pgmap v1563: 109 pgs: 9 active+clean+scrubbing, 100 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 939 op/s; 50 KiB/s, 0 objects/s recovering 2024-09-13T15:13:01.814 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:01.845 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:01.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:01.987 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:01.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:02.088 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:02.090 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:13:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:13:01] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T15:13:02.122 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:02.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:02.224 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:02.258 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:02.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:02.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:02.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:02.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:02.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:02.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:02.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:02.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:02.649 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:02.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[107038]: 1.11 scrub starts 2024-09-13T15:13:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[107038]: 1.11 scrub ok 2024-09-13T15:13:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[107038]: 1.e scrub starts 2024-09-13T15:13:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[107038]: 1.e scrub ok 2024-09-13T15:13:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[107038]: 1.c scrub starts 2024-09-13T15:13:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[107038]: 1.c scrub ok 2024-09-13T15:13:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[102314]: 1.11 scrub starts 2024-09-13T15:13:02.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[102314]: 1.11 scrub ok 2024-09-13T15:13:02.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[102314]: 1.e scrub starts 2024-09-13T15:13:02.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[102314]: 1.e scrub ok 2024-09-13T15:13:02.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[102314]: 1.c scrub starts 2024-09-13T15:13:02.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:02 smithi154 ceph-mon[102314]: 1.c scrub ok 2024-09-13T15:13:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:02 smithi172 ceph-mon[100696]: 1.11 scrub starts 2024-09-13T15:13:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:02 smithi172 ceph-mon[100696]: 1.11 scrub ok 2024-09-13T15:13:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:02 smithi172 ceph-mon[100696]: 1.e scrub starts 2024-09-13T15:13:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:02 smithi172 ceph-mon[100696]: 1.e scrub ok 2024-09-13T15:13:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:02 smithi172 ceph-mon[100696]: 1.c scrub starts 2024-09-13T15:13:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:02 smithi172 ceph-mon[100696]: 1.c scrub ok 2024-09-13T15:13:02.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:02.788 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:02.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:02.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:02.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:02.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:02.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:03.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:03.066 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:03.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:03.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:03.191 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:03.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:03.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:03.318 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:03.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:03.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:03.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:13:03.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:03.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:03.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:03.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:03.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:03.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:03.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:03.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[107038]: 1.b scrub starts 2024-09-13T15:13:03.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[107038]: 1.b scrub ok 2024-09-13T15:13:03.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[107038]: 1.4 scrub starts 2024-09-13T15:13:03.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[107038]: 1.4 scrub ok 2024-09-13T15:13:03.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[107038]: pgmap v1564: 109 pgs: 4 active+clean+scrubbing, 105 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 937 op/s 2024-09-13T15:13:03.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[102314]: 1.b scrub starts 2024-09-13T15:13:03.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[102314]: 1.b scrub ok 2024-09-13T15:13:03.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[102314]: 1.4 scrub starts 2024-09-13T15:13:03.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[102314]: 1.4 scrub ok 2024-09-13T15:13:03.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:03 smithi154 ceph-mon[102314]: pgmap v1564: 109 pgs: 4 active+clean+scrubbing, 105 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 937 op/s 2024-09-13T15:13:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:03.724 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:03.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:03 smithi172 ceph-mon[100696]: 1.b scrub starts 2024-09-13T15:13:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:03 smithi172 ceph-mon[100696]: 1.b scrub ok 2024-09-13T15:13:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:03 smithi172 ceph-mon[100696]: 1.4 scrub starts 2024-09-13T15:13:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:03 smithi172 ceph-mon[100696]: 1.4 scrub ok 2024-09-13T15:13:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:03 smithi172 ceph-mon[100696]: pgmap v1564: 109 pgs: 4 active+clean+scrubbing, 105 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 937 op/s 2024-09-13T15:13:03.825 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:03.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:03.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:03.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:04.000 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:04.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:04.137 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:04.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:04.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:04.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:04.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:04.512 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:04.549 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:04.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:04.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:04.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.1 scrub starts 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.13 scrub starts 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.13 scrub ok 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.1 scrub ok 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.6 scrub ok 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[107038]: 1.7 scrub ok 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.1 scrub starts 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.13 scrub starts 2024-09-13T15:13:04.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.13 scrub ok 2024-09-13T15:13:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.1 scrub ok 2024-09-13T15:13:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:13:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.6 scrub ok 2024-09-13T15:13:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:13:04.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:04 smithi154 ceph-mon[102314]: 1.7 scrub ok 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.1 scrub starts 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.13 scrub starts 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.13 scrub ok 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.1 scrub ok 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.6 scrub ok 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:13:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:04 smithi172 ceph-mon[100696]: 1.7 scrub ok 2024-09-13T15:13:04.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:04.824 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:04.926 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:04.958 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:04.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:05.059 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:05.100 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:05.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:05.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:05.228 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:05.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:05.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:05.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:05.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:05.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:05.503 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:05.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:05.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:05.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:05.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:05.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:05 smithi154 ceph-mon[102314]: 1.3 scrub starts 2024-09-13T15:13:05.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:05 smithi154 ceph-mon[102314]: 1.3 scrub ok 2024-09-13T15:13:05.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:05 smithi154 ceph-mon[102314]: pgmap v1565: 109 pgs: 109 active+clean; 860 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 911 op/s 2024-09-13T15:13:05.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:05 smithi154 ceph-mon[107038]: 1.3 scrub starts 2024-09-13T15:13:05.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:05 smithi154 ceph-mon[107038]: 1.3 scrub ok 2024-09-13T15:13:05.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:05 smithi154 ceph-mon[107038]: pgmap v1565: 109 pgs: 109 active+clean; 860 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 911 op/s 2024-09-13T15:13:05.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:05 smithi172 ceph-mon[100696]: 1.3 scrub starts 2024-09-13T15:13:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:05 smithi172 ceph-mon[100696]: 1.3 scrub ok 2024-09-13T15:13:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:05 smithi172 ceph-mon[100696]: pgmap v1565: 109 pgs: 109 active+clean; 860 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 911 op/s 2024-09-13T15:13:05.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:05.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:05.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:05.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:05.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:06.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:06.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:06.149 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.174 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:06.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:06.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:06.296 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:06.398 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:06.425 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:06.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:06.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:06.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:06.705 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:06.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:06 smithi154 ceph-mon[102314]: 1.2 scrub starts 2024-09-13T15:13:06.726 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:06 smithi154 ceph-mon[102314]: 1.2 scrub ok 2024-09-13T15:13:06.726 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:06 smithi154 ceph-mon[107038]: 1.2 scrub starts 2024-09-13T15:13:06.727 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:06 smithi154 ceph-mon[107038]: 1.2 scrub ok 2024-09-13T15:13:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:06 smithi172 ceph-mon[100696]: 1.2 scrub starts 2024-09-13T15:13:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:06 smithi172 ceph-mon[100696]: 1.2 scrub ok 2024-09-13T15:13:06.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:06.838 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:06.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:06.974 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:06.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:07.075 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:07.104 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:07.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:07.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:07.249 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:07.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:07.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:07.376 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:07.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:07.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:07.511 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:07.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:07.613 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:07.646 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:07.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:07.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:07 smithi154 ceph-mon[102314]: 1.f scrub starts 2024-09-13T15:13:07.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:07 smithi154 ceph-mon[102314]: 1.f scrub ok 2024-09-13T15:13:07.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:07 smithi154 ceph-mon[102314]: pgmap v1566: 109 pgs: 109 active+clean; 830 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-09-13T15:13:07.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:07 smithi154 ceph-mon[107038]: 1.f scrub starts 2024-09-13T15:13:07.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:07 smithi154 ceph-mon[107038]: 1.f scrub ok 2024-09-13T15:13:07.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:07 smithi154 ceph-mon[107038]: pgmap v1566: 109 pgs: 109 active+clean; 830 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-09-13T15:13:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:07 smithi172 ceph-mon[100696]: 1.f scrub starts 2024-09-13T15:13:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:07 smithi172 ceph-mon[100696]: 1.f scrub ok 2024-09-13T15:13:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:07 smithi172 ceph-mon[100696]: pgmap v1566: 109 pgs: 109 active+clean; 830 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 903 op/s 2024-09-13T15:13:07.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:07.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:07.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:07.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:07.925 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:07.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:07.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:08.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:08.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T15:13:08.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:08.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:08.217 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:08.244 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:08.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:08.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:08.380 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:08.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:08.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:08.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:08.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:08.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:08.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:08.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:08.760 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:08.790 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:08.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:08.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:08.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:08.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:09.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:09.101 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:09.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:09.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:09.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:09.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:09.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:09.369 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:09.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:09.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:09.476 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:09.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:09.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:09.530 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:09.563 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:13:09.564 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:13:09.613 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:13:09.613 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:13:09.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:09.651 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:09.652 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:13:09.653 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:13:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:09.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:09.687 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:13:09.687 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:13:09.718 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:13:09.718 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:13:09.749 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:13:09.750 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:13:09.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:09.781 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:09.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:09.788 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:13:09.789 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:13:09.824 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:13:09.825 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:13:09.882 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:09.915 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:09.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:09.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[102314]: pgmap v1567: 109 pgs: 109 active+clean; 677 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3021842269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:09.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:13:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:13:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[107038]: pgmap v1567: 109 pgs: 109 active+clean; 677 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:13:09.972 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:09 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3021842269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:10.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:10.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:10.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:10.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:10.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:10.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:10.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:10.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:10.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:13:10.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:13:10.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:10.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:09 smithi172 ceph-mon[100696]: pgmap v1567: 109 pgs: 109 active+clean; 677 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 879 op/s 2024-09-13T15:13:10.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:09 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3021842269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:10.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:10.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:10.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:10.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:10.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:10.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:10.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:10.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:10.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:10.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:10.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:10.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:10.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:10.906 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:10.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:11.008 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:11.042 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:11.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:11.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:11.181 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:11.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:10 smithi154 ceph-mon[102314]: 1.10 scrub starts 2024-09-13T15:13:11.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:10 smithi154 ceph-mon[102314]: 1.10 scrub ok 2024-09-13T15:13:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:10 smithi154 ceph-mon[107038]: 1.10 scrub starts 2024-09-13T15:13:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:10 smithi154 ceph-mon[107038]: 1.10 scrub ok 2024-09-13T15:13:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:10 smithi172 ceph-mon[100696]: 1.10 scrub starts 2024-09-13T15:13:11.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:10 smithi172 ceph-mon[100696]: 1.10 scrub ok 2024-09-13T15:13:11.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:11.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:11.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:11.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:11.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:11.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:11.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:11.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:11.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:11.706 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:11.732 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:11.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:11.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:11.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:11.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:11.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.018 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:12.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:12.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:12.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:12.189 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:11 smithi154 ceph-mon[102314]: pgmap v1568: 109 pgs: 109 active+clean; 589 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-13T15:13:12.191 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:13:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:13:11] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-13T15:13:12.192 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:11 smithi154 ceph-mon[107038]: pgmap v1568: 109 pgs: 109 active+clean; 589 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-13T15:13:12.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:11 smithi172 ceph-mon[100696]: pgmap v1568: 109 pgs: 109 active+clean; 589 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-13T15:13:12.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:12.278 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:12.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:12.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:12.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:12.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:12.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:12.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.727 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:12.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:12.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:12.859 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:12.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:12.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:12.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:13.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:13.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:13.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[102314]: 1.12 scrub starts 2024-09-13T15:13:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[102314]: 1.12 scrub ok 2024-09-13T15:13:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[102314]: 1.0 scrub starts 2024-09-13T15:13:13.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[102314]: 1.0 scrub ok 2024-09-13T15:13:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[107038]: 1.12 scrub starts 2024-09-13T15:13:13.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[107038]: 1.12 scrub ok 2024-09-13T15:13:13.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[107038]: 1.0 scrub starts 2024-09-13T15:13:13.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:12 smithi154 ceph-mon[107038]: 1.0 scrub ok 2024-09-13T15:13:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:12 smithi172 ceph-mon[100696]: 1.12 scrub starts 2024-09-13T15:13:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:12 smithi172 ceph-mon[100696]: 1.12 scrub ok 2024-09-13T15:13:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:12 smithi172 ceph-mon[100696]: 1.0 scrub starts 2024-09-13T15:13:13.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:12 smithi172 ceph-mon[100696]: 1.0 scrub ok 2024-09-13T15:13:13.245 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:13.279 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:13.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:13.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:13:13.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:13.406 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:13.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:13.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:13.537 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:13.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:13.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:13.672 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:13.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:13.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:13.808 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:13.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:13.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:13.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:13.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:14.052 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:14.058 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:14.058 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:13 smithi154 ceph-mon[102314]: pgmap v1569: 109 pgs: 109 active+clean; 494 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 852 op/s 2024-09-13T15:13:14.059 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:14.059 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:13 smithi154 ceph-mon[107038]: pgmap v1569: 109 pgs: 109 active+clean; 494 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 852 op/s 2024-09-13T15:13:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:14.081 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:14.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:14.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:14.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:14.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:14.224 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T15:13:14.225 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:13:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:14.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:13 smithi172 ceph-mon[100696]: pgmap v1569: 109 pgs: 109 active+clean; 494 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 852 op/s 2024-09-13T15:13:14.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:14.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:14.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:14.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:14.507 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:14.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:14.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:14.636 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:14.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:14.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:14.768 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:14.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:14.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:14.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:14.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:14.996 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:13:15.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:15.033 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:15.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:15.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-13T15:13:15.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:15.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:15.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:15.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/373692221' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:15.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:15.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:15.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:15.396 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/373692221' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:15.396 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/373692221' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:15.423 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:15.456 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:15.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:15.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:15.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:15.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:15.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:15.734 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:15.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:15.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:15.862 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:15.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:15.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:16.001 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:16.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.136 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:16.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:16.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/373692221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[102314]: osdmap e483: 8 total, 8 up, 8 in 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[102314]: pgmap v1571: 109 pgs: 109 active+clean; 288 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 912 op/s 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[102314]: 1.d scrub starts 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[102314]: 1.d scrub ok 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:13:16.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/373692221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:16.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[107038]: osdmap e483: 8 total, 8 up, 8 in 2024-09-13T15:13:16.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[107038]: pgmap v1571: 109 pgs: 109 active+clean; 288 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 912 op/s 2024-09-13T15:13:16.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[107038]: 1.d scrub starts 2024-09-13T15:13:16.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:15 smithi154 ceph-mon[107038]: 1.d scrub ok 2024-09-13T15:13:16.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:15 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:13:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:15 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:13:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/373692221' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:15 smithi172 ceph-mon[100696]: osdmap e483: 8 total, 8 up, 8 in 2024-09-13T15:13:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:15 smithi172 ceph-mon[100696]: pgmap v1571: 109 pgs: 109 active+clean; 288 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 912 op/s 2024-09-13T15:13:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:15 smithi172 ceph-mon[100696]: 1.d scrub starts 2024-09-13T15:13:16.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:15 smithi172 ceph-mon[100696]: 1.d scrub ok 2024-09-13T15:13:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:16.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:16.368 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:16.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:16.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:16.543 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.578 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:16.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:16.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:16.713 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:16.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:16.852 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:16.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:16.987 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:16.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:17.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:17.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:17.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:17.174 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:17.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:17.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:17.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:17.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:17.401 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:17.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:17.502 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:17.531 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:17.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:17.633 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:17.666 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:17.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:17.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:17 smithi154 ceph-mon[107038]: pgmap v1572: 109 pgs: 109 active+clean; 250 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-09-13T15:13:17.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:17 smithi154 ceph-mon[102314]: pgmap v1572: 109 pgs: 109 active+clean; 250 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-09-13T15:13:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:17 smithi172 ceph-mon[100696]: pgmap v1572: 109 pgs: 109 active+clean; 250 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 924 op/s 2024-09-13T15:13:17.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:17.809 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:17.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:17.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:17.930 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:17.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:18.032 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:18.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:18.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:18.201 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:18.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:18.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set noscrub 2024-09-13T15:13:18.340 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:18.353 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T15:13:18.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:18.472 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:18.528 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Removed 125673 objects 2024-09-13T15:13:18.529 INFO:tasks.radosbench.radosbench.0.smithi154.stdout:Clean up completed and total clean up time :141.563 2024-09-13T15:13:18.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:18.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:18.601 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2919455899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:13:18.601 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2919455899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:13:18.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:18.721 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:18.738 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-13T15:13:18.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-13T15:13:18.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2919455899' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:13:18.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:18.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:18.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:18.969 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:18.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:19.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:19.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:19.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:19.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:19.283 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:13:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=cleanup t=2024-09-13T15:13:18.822567456Z level=info msg="Completed cleanup jobs" duration=12.422424ms 2024-09-13T15:13:19.283 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:13:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=plugins.update.checker t=2024-09-13T15:13:18.945768705Z level=info msg="Update check succeeded" duration=59.6665ms 2024-09-13T15:13:19.283 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:13:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana.update.checker t=2024-09-13T15:13:18.963022591Z level=info msg="Update check succeeded" duration=67.109567ms 2024-09-13T15:13:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:19.305 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:19.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:19.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:13:19.406 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:19.430 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:19.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:19.531 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:19.552 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:19.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:19.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:19.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:19 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2919455899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:13:19.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:19 smithi154 ceph-mon[102314]: osdmap e484: 8 total, 8 up, 8 in 2024-09-13T15:13:19.657 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:19 smithi154 ceph-mon[102314]: pgmap v1574: 109 pgs: 109 active+clean; 96 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 980 op/s 2024-09-13T15:13:19.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:19 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2919455899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:13:19.657 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:19 smithi154 ceph-mon[107038]: osdmap e484: 8 total, 8 up, 8 in 2024-09-13T15:13:19.658 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:19 smithi154 ceph-mon[107038]: pgmap v1574: 109 pgs: 109 active+clean; 96 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 980 op/s 2024-09-13T15:13:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:19.679 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:19.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:19 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2919455899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:13:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:19 smithi172 ceph-mon[100696]: osdmap e484: 8 total, 8 up, 8 in 2024-09-13T15:13:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:19 smithi172 ceph-mon[100696]: pgmap v1574: 109 pgs: 109 active+clean; 96 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 980 op/s 2024-09-13T15:13:19.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:19.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:19.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:19.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:19.928 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:19.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:20.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.052 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:20.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:20.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:20.177 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:20.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:20.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:20.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:20.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:20.527 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:20.551 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:20.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.678 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:20.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:20.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:20.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:20.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:20.931 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:20.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:21.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:21.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:21.060 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:13:21.060 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:13:21.071 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:13:21.071 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:13:21.082 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:21.094 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:13:21.095 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:13:21.121 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:13:21.122 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:13:21.129 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:21.142 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:13:21.143 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:13:21.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:21.160 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:13:21.161 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:13:21.177 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:13:21.177 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:13:21.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:21.194 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:13:21.194 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:13:21.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:21.309 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:21.360 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2992529394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:21.360 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2992529394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:21.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.438 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:21.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:21.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2992529394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:21.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:21.564 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:21.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:21.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:21.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:21.817 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:21.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:21.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:21.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:22.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:22.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:22.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:22.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:22.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:22.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:22.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:22.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:22 smithi154 ceph-mon[102314]: pgmap v1575: 109 pgs: 109 active+clean; 31 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 921 op/s 2024-09-13T15:13:22.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:22 smithi154 ceph-mon[107038]: pgmap v1575: 109 pgs: 109 active+clean; 31 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 921 op/s 2024-09-13T15:13:22.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:13:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:13:21] "GET /metrics HTTP/1.1" 200 36648 "" "Prometheus/2.51.0" 2024-09-13T15:13:22.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:22.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:22.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:22.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:22.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:22.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:22.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:22 smithi172 ceph-mon[100696]: pgmap v1575: 109 pgs: 109 active+clean; 31 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 921 op/s 2024-09-13T15:13:22.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:22.563 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:22.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:22.624 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18374 9.c 9:3ff0438f:::benchmark_data_smithi154_291684_object18373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.169900+0000", 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 486.41817395300001, 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0942832500000002, 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.625 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18374 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169900+0000", 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169900+0000", 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169901+0000", 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.635e-06 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169926+0000", 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4556999999999999e-05 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169939+0000", 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3517e-05 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.169948+0000", 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3920000000000003e-06 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.170069+0000", 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121261 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.170079+0000", 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0298e-05 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.156239+0000", 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98616019200000005 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.156252+0000", 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2894999999999999e-05 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.258169+0000", 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1019164889999999 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.258339+0000", 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017045799999999999 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.258511+0000", 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000171965 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.261168+0000", 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002657066 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.261210+0000", 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2236e-05 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.262996+0000", 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001785888 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.263061+0000", 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4194000000000002e-05 2024-09-13T15:13:22.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.264183+0000", 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011222070000000001 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:134987 9.2 9:467b600d:::benchmark_data_smithi154_291684_object34526:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e408)", 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.280008+0000", 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 361.30806562399999, 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.075537991, 2024-09-13T15:13:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 134987 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280008+0000", 2024-09-13T15:13:22.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280008+0000", 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280009+0000", 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5489999999999999e-06 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280013+0000", 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7560000000000001e-06 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280022+0000", 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5010000000000004e-06 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280030+0000", 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3739999999999998e-06 2024-09-13T15:13:22.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.282923+0000", 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028925449999999998 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.282929+0000", 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7669999999999998e-06 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.739226+0000", 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.45629631799999998 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.739234+0000", 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1810000000000001e-06 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.346576+0000", 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.60734178299999997 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.346682+0000", 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010657100000000001 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.355485+0000", 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088032649999999994 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.355546+0000", 2024-09-13T15:13:22.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0380999999999997e-05 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:134997 9.12 9:49076328:::benchmark_data_smithi154_291684_object34536:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e408)", 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.280088+0000", 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 361.30798593100002, 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0973005330000001, 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:13:22.641 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 134997 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280088+0000", 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280088+0000", 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280088+0000", 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280090+0000", 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2989999999999999e-06 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280096+0000", 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1260000000000003e-06 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280101+0000", 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8289999999999998e-06 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.740648+0000", 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.46054710799999998 2024-09-13T15:13:22.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.740657+0000", 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1449999999999997e-06 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.365721+0000", 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.62506350200000005 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T15:13:22.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.365811+0000", 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9590000000000001e-05 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.377334+0000", 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011522991999999999 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.377388+0000", 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4527999999999997e-05 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.646 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:135004 9.1 9:8219fd83:::benchmark_data_smithi154_291684_object34543:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e408)", 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.280108+0000", 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 361.30796570299998, 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0910305149999999, 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.647 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 135004 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280108+0000", 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280108+0000", 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280108+0000", 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7e-07 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280110+0000", 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7430000000000001e-06 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280115+0000", 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0189999999999997e-06 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280119+0000", 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5090000000000004e-06 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.824926+0000", 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.54480658299999996 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.824939+0000", 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2733e-05 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.344628+0000", 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.51968938399999998 2024-09-13T15:13:22.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.344784+0000", 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015630599999999999 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.371068+0000", 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026284016 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.371138+0000", 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9851999999999995e-05 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:135005 9.8 9:1083822a:::benchmark_data_smithi154_291684_object34544:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e408)", 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.280134+0000", 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 361.30793984399997, 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.045486337, 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.653 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 135005 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280134+0000", 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280134+0000", 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280134+0000", 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.44e-07 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280136+0000", 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.052e-06 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280142+0000", 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280148+0000", 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9540000000000001e-06 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280352+0000", 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000204005 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.280356+0000", 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0740000000000003e-06 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.669409+0000", 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.38905345099999999 2024-09-13T15:13:22.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.669430+0000", 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0530000000000002e-05 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.299345+0000", 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.62991517399999997 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for missing object", 2024-09-13T15:13:22.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.299457+0000", 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011202 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.325547+0000", 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026090229999999999 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:22.325620+0000", 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2546000000000006e-05 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.659 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141340 9.11 9:88608592:::benchmark_data_smithi154_291684_object40879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.601917+0000", 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.98615683200001, 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5281782349999999, 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141340 2024-09-13T15:13:22.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.601917+0000", 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.601917+0000", 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.601919+0000", 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1050000000000002e-06 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.601924+0000", 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4419999999999997e-06 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.601937+0000", 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3035e-05 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.601945+0000", 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5989999999999996e-06 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.602126+0000", 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000181495 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.602137+0000", 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0609e-05 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072357+0000", 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.470219738 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072368+0000", 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1060000000000001e-05 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.125905+0000", 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.053537588 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.126007+0000", 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101548 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.126237+0000", 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000229726 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.129820+0000", 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035838570000000002 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.129829+0000", 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9220000000000003e-06 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.129867+0000", 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7827000000000002e-05 2024-09-13T15:13:22.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.129916+0000", 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8390999999999998e-05 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.129964+0000", 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8841999999999998e-05 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.130095+0000", 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013045099999999999 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141362 9.11 9:8f8101c7:::benchmark_data_smithi154_291684_object40901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.618870+0000", 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.96920383899999, 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.51534575, 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.668 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141362 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.618870+0000", 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.618870+0000", 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.618870+0000", 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.92e-07 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.618875+0000", 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1289999999999998e-06 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.618885+0000", 2024-09-13T15:13:22.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0487e-05 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.618892+0000", 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3329999999999999e-06 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.619007+0000", 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114227 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.619017+0000", 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9299999999999998e-06 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072393+0000", 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.45337670200000002 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072402+0000", 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4700000000000002e-06 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.127792+0000", 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.055389887 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.127879+0000", 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7300999999999996e-05 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.128085+0000", 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020619400000000001 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.130270+0000", 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002184479 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.130328+0000", 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8928999999999998e-05 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.130404+0000", 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5647000000000003e-05 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134052+0000", 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036476270000000001 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134109+0000", 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7179999999999998e-05 2024-09-13T15:13:22.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134215+0000", 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106436 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141385 9.11 9:8f06d450:::benchmark_data_smithi154_291684_object40924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.644295+0000", 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.94377856400001, 2024-09-13T15:13:22.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4902774599999999, 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141385 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644295+0000", 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644295+0000", 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644296+0000", 2024-09-13T15:13:22.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.096e-06 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644299+0000", 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.428e-06 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644309+0000", 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4900000000000006e-06 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644315+0000", 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4670000000000001e-06 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.646071+0000", 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001755599 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.646088+0000", 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6926999999999999e-05 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072427+0000", 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42633939799999998 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072434+0000", 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6810000000000001e-06 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.128889+0000", 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.056455436 2024-09-13T15:13:22.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.128985+0000", 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5414999999999994e-05 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.129110+0000", 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012518300000000001 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.133980+0000", 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004870012 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.133996+0000", 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5914e-05 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134274+0000", 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027808899999999997 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134346+0000", 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2008999999999994e-05 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134435+0000", 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9441e-05 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134572+0000", 2024-09-13T15:13:22.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013687500000000001 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141386 9.11 9:8833430a:::benchmark_data_smithi154_291684_object40925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.644325+0000", 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.94374807499997, 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4974338409999999, 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.685 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141386 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644325+0000", 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644325+0000", 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644326+0000", 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4099999999999999e-07 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644328+0000", 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7099999999999999e-06 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644333+0000", 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8050000000000003e-06 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.644338+0000", 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4900000000000002e-06 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.646113+0000", 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00177546 2024-09-13T15:13:22.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.646120+0000", 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3500000000000002e-06 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072456+0000", 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.426336671 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072464+0000", 2024-09-13T15:13:22.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4109999999999999e-06 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.130433+0000", 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.057968821 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.130544+0000", 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011112000000000001 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.130805+0000", 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026153299999999999 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134294+0000", 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034886840000000001 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.135200+0000", 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090631700000000002 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.135260+0000", 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0257999999999998e-05 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.141486+0000", 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062258330000000001 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.141645+0000", 2024-09-13T15:13:22.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015846100000000001 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.141759+0000", 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011447600000000001 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141420 9.11 9:8c780cbd:::benchmark_data_smithi154_291684_object40959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.677547+0000", 2024-09-13T15:13:22.693 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.91052687299998, 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4646128940000001, 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141420 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677547+0000", 2024-09-13T15:13:22.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677547+0000", 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677548+0000", 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.674e-06 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677555+0000", 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4749999999999998e-06 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677573+0000", 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7757999999999998e-05 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677581+0000", 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7469999999999994e-06 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677649+0000", 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8233999999999998e-05 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.677659+0000", 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7450000000000007e-06 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072489+0000", 2024-09-13T15:13:22.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.39482963300000001 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072496+0000", 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1010000000000003e-06 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.132091+0000", 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0595948230000001 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.132203+0000", 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112249 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.132411+0000", 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000207603 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.134750+0000", 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002339215 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.135300+0000", 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055006200000000003 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.135355+0000", 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5098e-05 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.141904+0000", 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006549053 2024-09-13T15:13:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.142056+0000", 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015162699999999999 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.142159+0000", 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103797 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141425 9.11 9:8fcd5c38:::benchmark_data_smithi154_291684_object40964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.683813+0000", 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.904260007, 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4683109999999999, 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141425 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.702 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683813+0000", 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683813+0000", 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683816+0000", 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0779999999999998e-06 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683820+0000", 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6970000000000001e-06 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683834+0000", 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3822000000000001e-05 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683842+0000", 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8229999999999995e-06 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683961+0000", 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119022 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.683971+0000", 2024-09-13T15:13:22.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7550000000000006e-06 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072519+0000", 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.38854872299999998 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072526+0000", 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1029999999999996e-06 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.135390+0000", 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0628634830000001 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.135526+0000", 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013644300000000001 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.135842+0000", 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031608 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.141802+0000", 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059591369999999998 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.142204+0000", 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000402622 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.147874+0000", 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056695019999999999 2024-09-13T15:13:22.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.151912+0000", 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040380210000000001 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152004+0000", 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2281999999999995e-05 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152124+0000", 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120407 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141458 9.11 9:8c309f3b:::benchmark_data_smithi154_291684_object40997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.725183+0000", 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.86289046299999, 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4275213339999999, 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.710 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141458 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.725183+0000", 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.725183+0000", 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.725185+0000", 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5e-06 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.725190+0000", 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2209999999999996e-06 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.725202+0000", 2024-09-13T15:13:22.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1823e-05 2024-09-13T15:13:22.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.725209+0000", 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6159999999999996e-06 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.727436+0000", 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022270039999999999 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.727446+0000", 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3430000000000002e-06 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072548+0000", 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.34510227300000001 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072555+0000", 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9829999999999999e-06 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.137372+0000", 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.064817535 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.137494+0000", 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121432 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.137717+0000", 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022322999999999999 2024-09-13T15:13:22.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.141835+0000", 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041176219999999996 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.142249+0000", 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041435099999999998 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.147916+0000", 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056670100000000001 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152262+0000", 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043459529999999996 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152390+0000", 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012776700000000001 2024-09-13T15:13:22.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152704+0000", 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031467099999999998 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141516 9.11 9:891afe5e:::benchmark_data_smithi154_291684_object41055:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.811547+0000", 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.77652614900001, 2024-09-13T15:13:22.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3415160830000001, 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141516 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811547+0000", 2024-09-13T15:13:22.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811547+0000", 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811549+0000", 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.632e-06 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811554+0000", 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6260000000000003e-06 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811567+0000", 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3271999999999999e-05 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811576+0000", 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0319999999999992e-06 2024-09-13T15:13:22.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811680+0000", 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103639 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.811690+0000", 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0837e-05 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072579+0000", 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.26088856199999999 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072905+0000", 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000326089 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.139256+0000", 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.066350503 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.139411+0000", 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000155184 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.139658+0000", 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024738499999999998 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.141873+0000", 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022150429999999999 2024-09-13T15:13:22.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.147957+0000", 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060836110000000001 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152246+0000", 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042895549999999996 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152716+0000", 2024-09-13T15:13:22.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047012099999999999 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152877+0000", 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016017399999999999 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153063+0000", 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000186818 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.727 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141629 9.11 9:8b8cebae:::benchmark_data_smithi154_291684_object41168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:29.972735+0000", 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.61533894299998, 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1806378580000001, 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141629 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972735+0000", 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972735+0000", 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972736+0000", 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.066e-06 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972739+0000", 2024-09-13T15:13:22.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8929999999999998e-06 2024-09-13T15:13:22.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972752+0000", 2024-09-13T15:13:22.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2055000000000001e-05 2024-09-13T15:13:22.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972759+0000", 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1949999999999997e-06 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972824+0000", 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4797999999999999e-05 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:29.972833+0000", 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0759999999999992e-06 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072980+0000", 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.100147599 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.072990+0000", 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6630000000000005e-06 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.142283+0000", 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0692929369999999 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.142414+0000", 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013154799999999999 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.142711+0000", 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000296459 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.148003+0000", 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052920980000000003 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.148044+0000", 2024-09-13T15:13:22.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1034e-05 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153075+0000", 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050309739999999997 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153116+0000", 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1309e-05 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153262+0000", 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014574799999999999 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153372+0000", 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110406 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141723 9.11 9:8c6da84c:::benchmark_data_smithi154_291684_object41262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:30.105918+0000", 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.48215506499997, 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0531963989999999, 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141723 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.739 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.105918+0000", 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.105918+0000", 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.105920+0000", 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3489999999999999e-06 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.105924+0000", 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6480000000000002e-06 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.105935+0000", 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0740000000000001e-05 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.105942+0000", 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0520000000000004e-06 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.106015+0000", 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2438e-05 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.106025+0000", 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8849999999999994e-06 2024-09-13T15:13:22.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.145636+0000", 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0396117739999999 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.145764+0000", 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000127921 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.146179+0000", 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041464900000000001 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.152183+0000", 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060043470000000002 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153386+0000", 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012026949999999999 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153425+0000", 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9384000000000001e-05 2024-09-13T15:13:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.158962+0000", 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055363610000000001 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.159014+0000", 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2488999999999998e-05 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.159115+0000", 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100667 2024-09-13T15:13:22.747 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141728 9.11 9:886a0326:::benchmark_data_smithi154_291684_object41267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:30.111245+0000", 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.47682861999999, 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.05982037, 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.748 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141728 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111245+0000", 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111245+0000", 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111247+0000", 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8810000000000001e-06 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111253+0000", 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6220000000000003e-06 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111270+0000", 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6286999999999998e-05 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111277+0000", 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2080000000000003e-06 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111342+0000", 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5550999999999996e-05 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.111352+0000", 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5810000000000003e-06 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.148088+0000", 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0367364370000001 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.148198+0000", 2024-09-13T15:13:22.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010969800000000001 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.148501+0000", 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030248599999999998 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153465+0000", 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049645829999999998 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153542+0000", 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6970000000000003e-05 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.159186+0000", 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056439569999999998 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.170911+0000", 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011724668000000001 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.170961+0000", 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0358000000000001e-05 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.171065+0000", 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104083 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.755 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:141756 9.11 9:8d391f54:::benchmark_data_smithi154_291684_object41295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:30.148237+0000", 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 352.43983625700002, 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0229698519999999, 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 141756 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148237+0000", 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148237+0000", 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148238+0000", 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9200000000000001e-07 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148241+0000", 2024-09-13T15:13:22.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.174e-06 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148249+0000", 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2970000000000004e-06 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148256+0000", 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2500000000000003e-06 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148326+0000", 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0128000000000003e-05 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:30.148337+0000", 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1178999999999999e-05 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.150092+0000", 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0017552599999999 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.150287+0000", 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019521499999999999 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,4,6", 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.150494+0000", 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020625 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.153503+0000", 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003009421 2024-09-13T15:13:22.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.159222+0000", 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057189390000000001 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.159259+0000", 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7314000000000002e-05 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.171074+0000", 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011814715999999999 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.171114+0000", 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0488999999999998e-05 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:31.171207+0000", 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2627999999999995e-05 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:178595 9.c 9:3b4141c8:::benchmark_data_smithi154_291684_object78134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:23.213437+0000", 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 299.374636802, 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.34651794800000002, 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:22.764 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 178595 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.213437+0000", 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.213437+0000", 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.213438+0000", 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5880000000000001e-06 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.213445+0000", 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1430000000000001e-06 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.213457+0000", 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1649e-05 2024-09-13T15:13:22.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.213466+0000", 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7299999999999994e-06 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.216027+0000", 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025607189999999999 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.216125+0000", 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8414999999999999e-05 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.216225+0000", 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100197 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.220610+0000", 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043845400000000001 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.222727+0000", 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021174100000000001 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.559847+0000", 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.337119536 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.559912+0000", 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5326000000000004e-05 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:23.559955+0000", 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2694999999999997e-05 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:202978 10.7 10:ed2a9c37:::benchmark_data_smithi154_317818_object77302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.564691+0000", 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.023382695999999, 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.34343146499999999, 2024-09-13T15:13:22.770 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202978 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564691+0000", 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564691+0000", 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564693+0000", 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8589999999999999e-06 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564698+0000", 2024-09-13T15:13:22.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5629999999999999e-06 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564712+0000", 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3448000000000001e-05 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564719+0000", 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6879999999999999e-06 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564776+0000", 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6441999999999999e-05 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564892+0000", 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116323 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,6", 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.564995+0000", 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103077 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.901348+0000", 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.33635295300000001 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.901410+0000", 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1994000000000003e-05 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.908052+0000", 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066413899999999996 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.908088+0000", 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6511999999999999e-05 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.908122+0000", 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4215999999999999e-05 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.36270.0:202979 10.5 e472/448 10:a05f3667:::benchmark_data_smithi154_317818_object77303:head v 472'12701, mlcod=472'12699)", 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.566430+0000", 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.021643003999998, 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.35760386999999999, 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.566430+0000", 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.566430+0000", 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.566432+0000", 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3039999999999999e-06 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.566437+0000", 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.592e-06 2024-09-13T15:13:22.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.897293+0000", 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.33085561400000002 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.897302+0000", 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2769999999999994e-06 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.897360+0000", 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7568000000000002e-05 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.897386+0000", 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5703000000000001e-05 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.924003+0000", 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026617294999999999 2024-09-13T15:13:22.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.924034+0000", 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1517000000000003e-05 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:13:22.781 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:13:22.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:22.783 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:22.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:22.884 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:22.910 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:22.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:23.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:23.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:23.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:23.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:23.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:23.284 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:23.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:23.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:23.510 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:23.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:23.617 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:23 smithi154 ceph-mon[102314]: pgmap v1576: 109 pgs: 109 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 652 op/s 2024-09-13T15:13:23.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:23 smithi154 ceph-mon[107038]: pgmap v1576: 109 pgs: 109 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 652 op/s 2024-09-13T15:13:23.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:23.660 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:23 smithi172 ceph-mon[100696]: pgmap v1576: 109 pgs: 109 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 652 op/s 2024-09-13T15:13:23.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:23.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:23.888 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:23.914 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:23.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:24.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:24.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:24.140 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:24.166 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:24.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:24.293 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:24.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:24.418 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:24.520 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:24.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:24.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:24.666 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:24.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:24.794 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:24.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:24.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:24.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:25.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:25.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:25.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:25.171 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:25.273 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:25.298 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:25.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-13T15:13:25.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:25.459 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:25.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:25.585 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:25.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:25.615 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:25.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:25.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:25.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:25.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:25 smithi154 ceph-mon[102314]: pgmap v1577: 109 pgs: 109 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 452 op/s 2024-09-13T15:13:25.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:25 smithi154 ceph-mon[107038]: pgmap v1577: 109 pgs: 109 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 452 op/s 2024-09-13T15:13:25.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:25 smithi172 ceph-mon[100696]: pgmap v1577: 109 pgs: 109 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 452 op/s 2024-09-13T15:13:25.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:25.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:25.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:25.955 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:25.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:26.057 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:26.080 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:26.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:26.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.204 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:26.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:26.218 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:26.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:26.326 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:26.428 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:26.450 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:26.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:26.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:26.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T15:13:26.650 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'unique_pool_5' removed 2024-09-13T15:13:26.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:26.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:26.815 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.840 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:26.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:26.941 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:26.962 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:26.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:26.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2057566892' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:13:26.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2057566892' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:13:26.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2057566892' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-13T15:13:27.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:27.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:27.188 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:27.219 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:27.321 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:27.344 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:27.356 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T15:13:27.356 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-13T15:13:27.356 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-13T15:13:27.357 INFO:tasks.workunit:Pulling workunits from ref reef 2024-09-13T15:13:27.357 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T15:13:27.357 DEBUG:teuthology.orchestra.run.smithi154:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:13:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.395 INFO:teuthology.orchestra.run.smithi154.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T15:13:27.395 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:13:27.400 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:13:27.401 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:13:27.401 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T15:13:27.401 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:13:27.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:27.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.470 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:27.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:27.476 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T15:13:27.476 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T15:13:27.532 INFO:tasks.workunit:timeout=3h 2024-09-13T15:13:27.532 INFO:tasks.workunit:cleanup=True 2024-09-13T15:13:27.533 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-13T15:13:27.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:27.593 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:27.604 INFO:tasks.workunit.client.0.smithi154.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T15:13:27.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:27.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:27.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:27.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:27.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:27.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:27.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:27.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2057566892' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:13:27.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:27 smithi154 ceph-mon[102314]: osdmap e485: 8 total, 8 up, 8 in 2024-09-13T15:13:27.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:27 smithi154 ceph-mon[102314]: pgmap v1579: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2057566892' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:13:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:27 smithi154 ceph-mon[107038]: osdmap e485: 8 total, 8 up, 8 in 2024-09-13T15:13:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:27 smithi154 ceph-mon[107038]: pgmap v1579: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2057566892' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-13T15:13:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:27 smithi172 ceph-mon[100696]: osdmap e485: 8 total, 8 up, 8 in 2024-09-13T15:13:27.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:27 smithi172 ceph-mon[100696]: pgmap v1579: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:28.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-13T15:13:28.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:28.117 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:28.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:28.242 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:28.299 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:28.318 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:28.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:28.365 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:28.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:28.491 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:28.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.615 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:28.677 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is set 2024-09-13T15:13:28.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:28.738 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:28.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.863 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:28.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2235523649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:13:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:13:28.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:28.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2235523649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:13:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:13:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:28.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:28.986 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:28.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:28 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2235523649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:13:28.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:28 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-13T15:13:28.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:28.992 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:29.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:29.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:29.212 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.234 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:29.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:29.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:29.356 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:29.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:29.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:29.583 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:29.673 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:13:29.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:29.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:29.731 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:29 smithi172 ceph-mon[100696]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:29.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:29 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:13:29.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:29 smithi172 ceph-mon[100696]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T15:13:29.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:29 smithi172 ceph-mon[100696]: pgmap v1581: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:29.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:29 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3541671265' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:13:29.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:29.854 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:29.944 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:13:29.945 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:13:29.945 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:13:29.945 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:13:29.945 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:13:29.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:29.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[102314]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[102314]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[102314]: pgmap v1581: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3541671265' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[107038]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[107038]: osdmap e486: 8 total, 8 up, 8 in 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[107038]: pgmap v1581: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:29.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:29 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3541671265' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:13:29.978 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:29.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:30.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:30.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:30.203 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.225 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:30.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:30.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.349 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:30.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:30.450 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:30.471 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:30.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:30.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-13T15:13:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.595 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:30.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:30.697 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:30.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:13:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:30.821 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:30.842 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:30.913 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:30 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3541671265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:13:30.913 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:30 smithi172 ceph-mon[100696]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T15:13:30.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:30.968 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:30.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:30 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3541671265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:13:30.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:30 smithi154 ceph-mon[102314]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T15:13:30.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:30 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3541671265' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:13:30.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:30 smithi154 ceph-mon[107038]: osdmap e487: 8 total, 8 up, 8 in 2024-09-13T15:13:31.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:31.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:31.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:31.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:31.317 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:31.339 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:31.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:31.441 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:31.463 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:31.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:31.586 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:31.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:31.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:31.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:31 smithi154 ceph-mon[102314]: pgmap v1583: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:31.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:31 smithi154 ceph-mon[107038]: pgmap v1583: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:31 smithi172 ceph-mon[100696]: pgmap v1583: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:31.813 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.836 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:31.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:31.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-13T15:13:31.937 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:31.959 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:31.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:32.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:32.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:32.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:32.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:32.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:32.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:32.220 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:13:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:13:31] "GET /metrics HTTP/1.1" 200 34225 "" "Prometheus/2.51.0" 2024-09-13T15:13:32.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:32.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:32.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:32.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:32.453 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:32.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213596 8.5 8:a9c8065b:::benchmark_data_smithi154_261841_object97557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.764454+0000", 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 567.75977719800005, 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.646727278, 2024-09-13T15:13:32.528 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213596 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764454+0000", 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764454+0000", 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764452+0000", 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764457+0000", 2024-09-13T15:13:32.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8219999999999997e-06 2024-09-13T15:13:32.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764465+0000", 2024-09-13T15:13:32.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3389999999999993e-06 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.764471+0000", 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4570000000000002e-06 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.767499+0000", 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003027596 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.767511+0000", 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1590000000000001e-05 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002167+0000", 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.23465654899999999 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002178+0000", 2024-09-13T15:13:32.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1062e-05 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.399719+0000", 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.397541014 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.399855+0000", 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558299999999999 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.399959+0000", 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103823 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404097+0000", 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00413862 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404107+0000", 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8490000000000001e-06 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.410946+0000", 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068393359999999997 2024-09-13T15:13:32.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411005+0000", 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8853e-05 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411055+0000", 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0142000000000002e-05 2024-09-13T15:13:32.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411181+0000", 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012549999999999999 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213599 8.5 8:a4956294:::benchmark_data_smithi154_261841_object97560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.780614+0000", 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 567.74361682899996, 2024-09-13T15:13:32.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.638991925, 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213599 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780614+0000", 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780614+0000", 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780613+0000", 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780618+0000", 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6189999999999999e-06 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780627+0000", 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4549999999999995e-06 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780632+0000", 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3449999999999996e-06 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780668+0000", 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5544999999999999e-05 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.780685+0000", 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7547999999999999e-05 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002203+0000", 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.22151791100000001 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002211+0000", 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8690000000000004e-06 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404144+0000", 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.401932687 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404291+0000", 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014680699999999999 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.404431+0000", 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140438 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.410961+0000", 2024-09-13T15:13:32.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065295869999999999 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411228+0000", 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000267304 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419169+0000", 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079415240000000002 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419445+0000", 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027508099999999999 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419496+0000", 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1307999999999997e-05 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419606+0000", 2024-09-13T15:13:32.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011013 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213601 8.5 8:a4f419b3:::benchmark_data_smithi154_261841_object97562:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.781971+0000", 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 567.742260332, 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.637816929, 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213601 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781971+0000", 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781971+0000", 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781970+0000", 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781975+0000", 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3909999999999997e-06 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781988+0000", 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2799000000000001e-05 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.781994+0000", 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4119999999999998e-06 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.782096+0000", 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010180500000000001 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.782109+0000", 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2653000000000001e-05 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002234+0000", 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.220124976 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002241+0000", 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7190000000000001e-06 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.407884+0000", 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4056430579999999 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.407977+0000", 2024-09-13T15:13:32.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2819999999999996e-05 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.408068+0000", 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0521999999999998e-05 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411452+0000", 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003384569 2024-09-13T15:13:32.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411465+0000", 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2619000000000001e-05 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419368+0000", 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079028039999999994 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419637+0000", 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000269435 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419681+0000", 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3640000000000002e-05 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419787+0000", 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106662 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213607 8.5 8:aaa2bad7:::benchmark_data_smithi154_261841_object97568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.791517+0000", 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 567.73271372800002, 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.636411989, 2024-09-13T15:13:32.556 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213607 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791517+0000", 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791517+0000", 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791516+0000", 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791522+0000", 2024-09-13T15:13:32.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2829999999999998e-06 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791537+0000", 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.517e-05 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791546+0000", 2024-09-13T15:13:32.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9649999999999997e-06 2024-09-13T15:13:32.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:32.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791609+0000", 2024-09-13T15:13:32.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-09-13T15:13:32.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.791620+0000", 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0609e-05 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002264+0000", 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.21064449800000001 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002272+0000", 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1950000000000003e-06 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411518+0000", 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4092451719999999 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411648+0000", 2024-09-13T15:13:32.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000130488 2024-09-13T15:13:32.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.411786+0000", 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013809200000000001 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419180+0000", 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0073940580000000002 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419403+0000", 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222943 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419854+0000", 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000450965 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427743+0000", 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078889719999999993 2024-09-13T15:13:32.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427797+0000", 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4205e-05 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427929+0000", 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013182299999999999 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.571 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.572 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.572 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213617 8.5 8:ad66873f:::benchmark_data_smithi154_261841_object97578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:13:32.572 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.802801+0000", 2024-09-13T15:13:32.573 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 567.72143002799999, 2024-09-13T15:13:32.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.634649596, 2024-09-13T15:13:32.573 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.573 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213617 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802801+0000", 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802801+0000", 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802800+0000", 2024-09-13T15:13:32.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802805+0000", 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4989999999999996e-06 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802815+0000", 2024-09-13T15:13:32.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0200000000000001e-05 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802820+0000", 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5330000000000002e-06 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802870+0000", 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9747999999999999e-05 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.802880+0000", 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0244000000000001e-05 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002295+0000", 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.19941504400000001 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002303+0000", 2024-09-13T15:13:32.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.79e-06 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.415020+0000", 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4127169369999999 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.415150+0000", 2024-09-13T15:13:32.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129488 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.415294+0000", 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014416200000000001 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419617+0000", 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043235770000000003 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419898+0000", 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028118300000000001 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419938+0000", 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9638000000000002e-05 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437248+0000", 2024-09-13T15:13:32.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017309486999999998 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437330+0000", 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2735e-05 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437450+0000", 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120112 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.582 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.35787.0:213622 8.5 8:a3a31e6c:::benchmark_data_smithi154_261841_object97583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:04:04.812504+0000", 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 567.71172718900004, 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.625395114, 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.35787", 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2619320750", 2024-09-13T15:13:32.583 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 213622 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812504+0000", 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812504+0000", 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812503+0000", 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812508+0000", 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2959999999999998e-06 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812519+0000", 2024-09-13T15:13:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1374e-05 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812525+0000", 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4580000000000001e-06 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812578+0000", 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3149000000000002e-05 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:04.812590+0000", 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1857e-05 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002327+0000", 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.189736714 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:05.002353+0000", 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6916000000000001e-05 2024-09-13T15:13:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.419970+0000", 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.417616773 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.420096+0000", 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012563900000000001 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,6,7", 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.420199+0000", 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010286 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427696+0000", 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074976080000000002 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.427705+0000", 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3820000000000004e-06 2024-09-13T15:13:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437185+0000", 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094806119999999994 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437504+0000", 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000319062 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437575+0000", 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0785999999999996e-05 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:04:06.437899+0000", 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032361199999999998 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18364 9.e 9:7d038a3a:::benchmark_data_smithi154_291684_object18363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:13:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.155489+0000", 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 496.368741521, 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.113528574, 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18364 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155489+0000", 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155489+0000", 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155488+0000", 2024-09-13T15:13:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155655+0000", 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167205 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155666+0000", 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599e-05 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155680+0000", 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.38e-05 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155740+0000", 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9725000000000002e-05 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155752+0000", 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1722999999999999e-05 2024-09-13T15:13:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157766+0000", 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.002014593 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157780+0000", 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3730999999999999e-05 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.258720+0000", 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.100940357 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.259080+0000", 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035937099999999999 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.259194+0000", 2024-09-13T15:13:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000114016 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263014+0000", 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038206659999999999 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263071+0000", 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7049999999999998e-05 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.268384+0000", 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005312092 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.268422+0000", 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8442999999999998e-05 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269018+0000", 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00059586600000000002 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18366 9.e 9:7f15721d:::benchmark_data_smithi154_291684_object18365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 496.36853169599999, 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1138882319999999, 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18366 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155699+0000", 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155781+0000", 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2699000000000002e-05 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155791+0000", 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7049999999999994e-06 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155797+0000", 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3740000000000003e-06 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155900+0000", 2024-09-13T15:13:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010247300000000001 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.155909+0000", 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8650000000000007e-06 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157806+0000", 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0018974 2024-09-13T15:13:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.157814+0000", 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9969999999999998e-06 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.260479+0000", 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1026644379999999 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.260631+0000", 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015289599999999999 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.260821+0000", 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018936100000000001 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263902+0000", 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003081056 2024-09-13T15:13:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.265074+0000", 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001172183 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269229+0000", 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041554729999999998 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269263+0000", 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3890999999999998e-05 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269587+0000", 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032401100000000003 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:18371 9.e 9:7b44ede2:::benchmark_data_smithi154_291684_object18370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:05:16.158169+0000", 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 496.36606203899998, 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1119104590000002, 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 18371 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158169+0000", 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158169+0000", 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158168+0000", 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158243+0000", 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5414999999999995e-05 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158259+0000", 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5985e-05 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158267+0000", 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3189999999999995e-06 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158318+0000", 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.079e-05 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:16.158328+0000", 2024-09-13T15:13:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5289999999999997e-06 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.158078+0000", 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99975051999999998 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:17.158090+0000", 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1610999999999999e-05 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.261496+0000", 2024-09-13T15:13:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.103406554 2024-09-13T15:13:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.261596+0000", 2024-09-13T15:13:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9580000000000005e-05 2024-09-13T15:13:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.261722+0000", 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012574400000000001 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.263906+0000", 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002184044 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.268326+0000", 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044207480000000004 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269633+0000", 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013068559999999999 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.269681+0000", 2024-09-13T15:13:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7639999999999998e-05 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:05:18.270079+0000", 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00039841099999999999 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:66495 9.e 9:764958b5:::benchmark_data_smithi154_291684_object66494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:13:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:02.044075+0000", 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 450.480156335, 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.005924974, 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 66495 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044075+0000", 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044075+0000", 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044073+0000", 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044136+0000", 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3113000000000004e-05 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044147+0000", 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1216999999999999e-05 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044153+0000", 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6910000000000002e-06 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044373+0000", 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022015399999999999 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.044386+0000", 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2850999999999999e-05 2024-09-13T15:13:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.043047+0000", 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99866041400000005 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.043168+0000", 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012136799999999999 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.043278+0000", 2024-09-13T15:13:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110513 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.048577+0000", 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052982929999999999 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049856+0000", 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012793310000000001 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049921+0000", 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5167999999999997e-05 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049961+0000", 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9283999999999999e-05 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.049999+0000", 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8924000000000002e-05 2024-09-13T15:13:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:66505 9.e 9:7e252385:::benchmark_data_smithi154_291684_object66504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:02.053055+0000", 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 450.47117582800001, 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0004483980000001, 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 66505 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053055+0000", 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053055+0000", 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053054+0000", 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053101+0000", 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7327000000000003e-05 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053113+0000", 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.189e-05 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053118+0000", 2024-09-13T15:13:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6319999999999996e-06 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053160+0000", 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1854999999999998e-05 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.053168+0000", 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5920000000000003e-06 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.045594+0000", 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99242650899999996 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.045699+0000", 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104552 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.045791+0000", 2024-09-13T15:13:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1812999999999997e-05 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.050075+0000", 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042843610000000004 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053358+0000", 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003282807 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053412+0000", 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4205e-05 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053464+0000", 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1737000000000003e-05 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053503+0000", 2024-09-13T15:13:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9548000000000002e-05 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:66538 9.e 9:78a5717b:::benchmark_data_smithi154_291684_object66537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:06:02.086696+0000", 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 450.43753500600002, 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97339183900000004, 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 66538 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086696+0000", 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086696+0000", 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086695+0000", 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086785+0000", 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0914000000000004e-05 2024-09-13T15:13:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086801+0000", 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5778999999999999e-05 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086809+0000", 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0849999999999997e-06 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086870+0000", 2024-09-13T15:13:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0584000000000001e-05 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:02.086883+0000", 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3322000000000001e-05 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.046925+0000", 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96004212200000005 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.047053+0000", 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127439 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,1", 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.047143+0000", 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0160000000000004e-05 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053368+0000", 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062250099999999996 2024-09-13T15:13:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.053568+0000", 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000199736 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.060002+0000", 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064343539999999998 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.060053+0000", 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1057999999999998e-05 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:06:03.060088+0000", 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4586999999999999e-05 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:140809 9.10 9.ae4d0b30 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:07:27.322545+0000", 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 365.20168569999998, 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96802707600000004, 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140809 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322545+0000", 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322545+0000", 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322544+0000", 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322549+0000", 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0790000000000002e-06 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322557+0000", 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3979999999999999e-06 2024-09-13T15:13:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322562+0000", 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322598+0000", 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6134000000000003e-05 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:27.322604+0000", 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7520000000000002e-06 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:28.290560+0000", 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96795643499999995 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:28.290572+0000", 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1846e-05 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:181320 9.16 9:6caf4932:::benchmark_data_smithi154_291684_object80859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:27.178644+0000", 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 305.34558673100003, 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0000961740000001, 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 181320 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178644+0000", 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178644+0000", 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178643+0000", 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178651+0000", 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.106e-06 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178665+0000", 2024-09-13T15:13:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4652e-05 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178681+0000", 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.525e-05 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178756+0000", 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5454e-05 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178766+0000", 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0172e-05 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.167116+0000", 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98834951999999998 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.167194+0000", 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7991000000000006e-05 2024-09-13T15:13:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.167473+0000", 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279017 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.171334+0000", 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038613100000000002 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.174131+0000", 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027966689999999999 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178679+0000", 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045485079999999997 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178715+0000", 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6291000000000002e-05 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178740+0000", 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4800999999999999e-05 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:181321 9.16 9:6bf1129e:::benchmark_data_smithi154_291684_object80860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 305.34553054200001, 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0060002299999999, 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 181321 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178700+0000", 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178703+0000", 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5439999999999999e-06 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178710+0000", 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4150000000000004e-06 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178715+0000", 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5369999999999997e-06 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178795+0000", 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9561000000000006e-05 2024-09-13T15:13:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:27.178805+0000", 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0088999999999999e-05 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.168868+0000", 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.99006304199999995 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.168941+0000", 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3268000000000004e-05 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1,2", 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.169198+0000", 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025707900000000001 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.174072+0000", 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048734690000000001 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.178772+0000", 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046999190000000003 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.184612+0000", 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058403880000000002 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.184656+0000", 2024-09-13T15:13:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4199999999999997e-05 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:28.184701+0000", 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4305999999999999e-05 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73776 10.1 10:8adf5dcd:::benchmark_data_smithi154_317818_object73775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.958770+0000", 2024-09-13T15:13:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 206.56546042599999, 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.55018106300000003, 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73776 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958770+0000", 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958770+0000", 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958769+0000", 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958857+0000", 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7634999999999997e-05 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958867+0000", 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9860000000000006e-06 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958872+0000", 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0390000000000004e-06 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.958915+0000", 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2933999999999999e-05 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959073+0000", 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015779800000000001 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959212+0000", 2024-09-13T15:13:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013958800000000001 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.962946+0000", 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037334719999999998 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.966378+0000", 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034321540000000002 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508836+0000", 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54245829000000001 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508909+0000", 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2791000000000002e-05 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508951+0000", 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2485999999999999e-05 2024-09-13T15:13:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73779 10.1 10:806e7cc1:::benchmark_data_smithi154_317818_object73778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 206.56444862199999, 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54928832000000005, 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73779 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:13:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959782+0000", 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959857+0000", 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4925999999999994e-05 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959865+0000", 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1280000000000008e-06 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959869+0000", 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5310000000000003e-06 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.961136+0000", 2024-09-13T15:13:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012669070000000001 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.961262+0000", 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012542599999999999 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.961404+0000", 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142538 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.963208+0000", 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018036040000000001 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.966437+0000", 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003228982 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.508999+0000", 2024-09-13T15:13:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54256280499999998 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509044+0000", 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4308000000000003e-05 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509071+0000", 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6755e-05 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73803 10.1 10:87b245ab:::benchmark_data_smithi154_317818_object73802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.980377+0000", 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 206.54385391599999, 2024-09-13T15:13:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.52882779800000002, 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73803 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980377+0000", 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980377+0000", 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980375+0000", 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980421+0000", 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5469000000000001e-05 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980432+0000", 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0791000000000001e-05 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980437+0000", 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0180000000000001e-06 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980475+0000", 2024-09-13T15:13:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8210000000000002e-05 2024-09-13T15:13:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980586+0000", 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110992 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.980709+0000", 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012335499999999999 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.982686+0000", 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019766470000000002 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.983185+0000", 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049875699999999995 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509123+0000", 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.52593843100000004 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509177+0000", 2024-09-13T15:13:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4243999999999998e-05 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.509205+0000", 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7412000000000001e-05 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:199749 10.1 10:8a781fa4:::benchmark_data_smithi154_317818_object74073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:12:20.586778+0000", 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.937452765000003, 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.364094205, 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199749 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.586778+0000", 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.586778+0000", 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.586777+0000", 2024-09-13T15:13:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.586782+0000", 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3260000000000003e-06 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.586797+0000", 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4735999999999999e-05 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.586803+0000", 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.72e-06 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.586881+0000", 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7156000000000005e-05 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.587001+0000", 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120423 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.587121+0000", 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000120102 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.594365+0000", 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0072437939999999996 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.595533+0000", 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011677899999999999 2024-09-13T15:13:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.950778+0000", 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.35524487100000002 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.950832+0000", 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4070000000000002e-05 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.950872+0000", 2024-09-13T15:13:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0586000000000001e-05 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:199788 10.1 10:8940c351:::benchmark_data_smithi154_317818_object74112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:12:20.638131+0000", 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.886099556000005, 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.33327617900000001, 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 199788 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638131+0000", 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638131+0000", 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:32.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638129+0000", 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638139+0000", 2024-09-13T15:13:32.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5289999999999997e-06 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638158+0000", 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9480999999999999e-05 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:32.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638167+0000", 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7309999999999999e-06 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638288+0000", 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121552 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638418+0000", 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000129121 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,7", 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.638557+0000", 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013946600000000001 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.642520+0000", 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039625889999999999 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.642732+0000", 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021280800000000001 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.971115+0000", 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.32838302200000002 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.971312+0000", 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019607000000000001 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:12:20.971407+0000", 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5937e-05 2024-09-13T15:13:32.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.697 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:32.697 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:32.697 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:13:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:32.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:32.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:32.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:32.821 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:32.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:32.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:32.896 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T15:13:32.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:32.944 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:32.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:33.005 INFO:tasks.workunit.client.0.smithi154.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T15:13:33.007 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:13:33.007 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T15:13:33.046 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:33.067 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:33.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:33.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:33.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:33.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:33.254 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T15:13:33.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:33.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:33.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:33.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:33.446 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T15:13:33.449 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:33 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/969196959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:33.449 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:33 smithi172 ceph-mon[100696]: pgmap v1584: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:33.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-13T15:13:33.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:33 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/969196959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:33.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:33 smithi154 ceph-mon[102314]: pgmap v1584: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:33.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:33 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/969196959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:33 smithi154 ceph-mon[107038]: pgmap v1584: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:33.502 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:13:33.504 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:13:33.504 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T15:13:33.540 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:13:33.540 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:13:33.540 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:13:33.540 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:13:33.541 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:13:33.541 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:13:33.541 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:13:33.541 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:13:33.541 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:13:33.541 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:13:33.541 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:13:33.542 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:13:33.542 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:13:33.542 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:13:33.542 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:13:33.542 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:13:33.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:33.543 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:13:33.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:33.583 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T15:13:33.583 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T15:13:33.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:33.667 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-13T15:13:33.669 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-13T15:13:33.669 DEBUG:teuthology.orchestra.run.smithi154:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-13T15:13:33.669 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:33.691 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:33.793 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:33.815 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:33.870 INFO:tasks.workunit.client.0.smithi154.stdout:Running main() from gmock_main.cc 2024-09-13T15:13:33.870 INFO:tasks.workunit.client.0.smithi154.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-13T15:13:33.870 INFO:tasks.workunit.client.0.smithi154.stdout:[----------] Global test environment set-up. 2024-09-13T15:13:33.870 INFO:tasks.workunit.client.0.smithi154.stdout:[----------] 70 tests from TestClsRbd 2024-09-13T15:13:33.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:33.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:33.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:34.040 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.063 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:34.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:34.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:34.186 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:34.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:34.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:34.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.434 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:34.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:34.528 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:34.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:34.560 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:34.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:34.562 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:34.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:34.685 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:34.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:34.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:34.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:34.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:34.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:35.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.056 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:35.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:35.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.179 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:35.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:35.200 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:13:35.200 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:13:35.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:35.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:35.338 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-13T15:13:35.363 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_all_features (26 ms) 2024-09-13T15:13:35.364 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.copyup 2024-09-13T15:13:35.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:35.427 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:35.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:35.547 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.copyup (183 ms) 2024-09-13T15:13:35.547 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-13T15:13:35.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:35.563 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.sparse_copyup (16 ms) 2024-09-13T15:13:35.563 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-13T15:13:35.572 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:35 smithi154 ceph-mon[102314]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T15:13:35.572 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3731978782' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-346161-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:13:35.572 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:35 smithi154 ceph-mon[102314]: pgmap v1586: 125 pgs: 2 creating+peering, 30 unknown, 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:35.572 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:35 smithi154 ceph-mon[107038]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T15:13:35.573 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3731978782' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-346161-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:13:35.573 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:35 smithi154 ceph-mon[107038]: pgmap v1586: 125 pgs: 2 creating+peering, 30 unknown, 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:35.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.676 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:35.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:35 smithi172 ceph-mon[100696]: osdmap e488: 8 total, 8 up, 8 in 2024-09-13T15:13:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3731978782' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-346161-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-13T15:13:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:35 smithi172 ceph-mon[100696]: pgmap v1586: 125 pgs: 2 creating+peering, 30 unknown, 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:35.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:35.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:35.808 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_and_set_id (245 ms) 2024-09-13T15:13:35.808 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-13T15:13:35.818 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.add_remove_child (10 ms) 2024-09-13T15:13:35.818 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-13T15:13:35.842 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.directory_methods (24 ms) 2024-09-13T15:13:35.842 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.create 2024-09-13T15:13:35.862 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.create (20 ms) 2024-09-13T15:13:35.862 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_features 2024-09-13T15:13:35.863 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2024-09-13T15:13:35.863 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-13T15:13:35.865 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_object_prefix (2 ms) 2024-09-13T15:13:35.866 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-13T15:13:35.867 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_create_timestamp (2 ms) 2024-09-13T15:13:35.867 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-13T15:13:35.868 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-13T15:13:35.869 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_access_timestamp (1 ms) 2024-09-13T15:13:35.869 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-13T15:13:35.870 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_modify_timestamp (2 ms) 2024-09-13T15:13:35.870 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-13T15:13:35.873 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_data_pool (3 ms) 2024-09-13T15:13:35.873 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.get_size 2024-09-13T15:13:35.877 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.get_size (3 ms) 2024-09-13T15:13:35.877 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.set_size 2024-09-13T15:13:35.881 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.set_size (4 ms) 2024-09-13T15:13:35.881 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-13T15:13:35.897 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.protection_status (17 ms) 2024-09-13T15:13:35.897 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-13T15:13:35.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:35.904 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-09-13T15:13:35.904 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-13T15:13:35.932 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:35.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:35.952 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.parents_v1 (48 ms) 2024-09-13T15:13:35.953 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-13T15:13:35.981 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.parents_v2 (28 ms) 2024-09-13T15:13:35.981 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-13T15:13:36.001 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.snapshots (20 ms) 2024-09-13T15:13:36.001 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-13T15:13:36.005 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.snapid_race (4 ms) 2024-09-13T15:13:36.005 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-13T15:13:36.013 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.stripingv2 (9 ms) 2024-09-13T15:13:36.013 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-13T15:13:36.015 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.object_map_save (2 ms) 2024-09-13T15:13:36.015 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-13T15:13:36.020 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.object_map_resize (5 ms) 2024-09-13T15:13:36.021 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-13T15:13:36.026 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.object_map_update (5 ms) 2024-09-13T15:13:36.026 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-13T15:13:36.026 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.object_map_load_enoent (1 ms) 2024-09-13T15:13:36.026 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-13T15:13:36.032 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.object_map_snap_add (5 ms) 2024-09-13T15:13:36.032 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-13T15:13:36.034 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:36.039 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.object_map_snap_remove (8 ms) 2024-09-13T15:13:36.039 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.flags 2024-09-13T15:13:36.045 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.flags (6 ms) 2024-09-13T15:13:36.045 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.metadata 2024-09-13T15:13:36.061 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:36.072 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.metadata (27 ms) 2024-09-13T15:13:36.072 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.set_features 2024-09-13T15:13:36.080 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.set_features (7 ms) 2024-09-13T15:13:36.080 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.mirror 2024-09-13T15:13:36.083 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:36.114 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.mirror (34 ms) 2024-09-13T15:13:36.114 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-13T15:13:36.129 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.mirror_image (15 ms) 2024-09-13T15:13:36.129 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-13T15:13:36.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:36.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:36.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:36.313 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:36.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:36.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:36.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T15:13:36.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3731978782' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-346161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:13:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:36 smithi172 ceph-mon[100696]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T15:13:36.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:36 smithi172 ceph-mon[100696]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:13:36.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:36.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:36.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:36.691 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3731978782' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-346161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:13:36.692 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:36 smithi154 ceph-mon[102314]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T15:13:36.692 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:36 smithi154 ceph-mon[102314]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:13:36.692 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3731978782' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-346161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-13T15:13:36.692 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:36 smithi154 ceph-mon[107038]: osdmap e489: 8 total, 8 up, 8 in 2024-09-13T15:13:36.692 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:36 smithi154 ceph-mon[107038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:13:36.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:36.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:36.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:36.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:36.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:36.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:37.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:37.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:37.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:37.210 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:37.312 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:37.334 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:37.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:37.458 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:37 smithi172 ceph-mon[100696]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T15:13:37.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:37 smithi172 ceph-mon[100696]: pgmap v1589: 125 pgs: 2 creating+peering, 25 unknown, 98 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 40 KiB/s rd, 11 KiB/s wr, 60 op/s 2024-09-13T15:13:37.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:37 smithi154 ceph-mon[102314]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T15:13:37.542 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:37 smithi154 ceph-mon[102314]: pgmap v1589: 125 pgs: 2 creating+peering, 25 unknown, 98 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 40 KiB/s rd, 11 KiB/s wr, 60 op/s 2024-09-13T15:13:37.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:37 smithi154 ceph-mon[107038]: osdmap e490: 8 total, 8 up, 8 in 2024-09-13T15:13:37.542 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:37 smithi154 ceph-mon[107038]: pgmap v1589: 125 pgs: 2 creating+peering, 25 unknown, 98 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 40 KiB/s rd, 11 KiB/s wr, 60 op/s 2024-09-13T15:13:37.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:37.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:37.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:37.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:37.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:37.805 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:13:37.806 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:13:37.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:37.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:37.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:37.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:37.967 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:37.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:38.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:38.090 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:38.192 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:38.213 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:38.316 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:38.337 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:38.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:38.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:38.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-13T15:13:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:38 smithi172 ceph-mon[100696]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:13:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3225358585' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:13:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:38 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:13:38.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3453269835' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:38.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:38.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:38.670 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:38.688 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is set 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[102314]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3225358585' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3453269835' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[107038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3225358585' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:13:38.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3453269835' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:38.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:38.715 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:38.771 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.mirror_image_status (2642 ms) 2024-09-13T15:13:38.771 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-13T15:13:38.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.839 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:38.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:38.850 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.mirror_image_map (79 ms) 2024-09-13T15:13:38.850 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-13T15:13:38.858 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.mirror_instances (9 ms) 2024-09-13T15:13:38.859 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-13T15:13:38.869 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.mirror_snapshot (10 ms) 2024-09-13T15:13:38.869 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-13T15:13:38.871 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_dir_list (3 ms) 2024-09-13T15:13:38.872 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-13T15:13:38.873 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-09-13T15:13:38.873 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-13T15:13:38.876 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.dir_add_already_existing (3 ms) 2024-09-13T15:13:38.876 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-13T15:13:38.881 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_dir_rename (5 ms) 2024-09-13T15:13:38.881 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-13T15:13:38.884 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_dir_remove (2 ms) 2024-09-13T15:13:38.884 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-13T15:13:38.887 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-09-13T15:13:38.887 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-13T15:13:38.890 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_image_add (3 ms) 2024-09-13T15:13:38.890 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-13T15:13:38.893 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2024-09-13T15:13:38.894 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-13T15:13:38.897 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2024-09-13T15:13:38.897 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-13T15:13:38.900 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-09-13T15:13:38.901 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-13T15:13:38.903 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-13T15:13:38.903 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-13T15:13:38.906 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.image_group_remove (3 ms) 2024-09-13T15:13:38.906 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-13T15:13:38.907 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.image_group_get (2 ms) 2024-09-13T15:13:38.907 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-13T15:13:38.909 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (1 ms) 2024-09-13T15:13:38.909 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-13T15:13:38.911 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-09-13T15:13:38.911 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-13T15:13:38.914 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-09-13T15:13:38.914 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-13T15:13:38.916 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (3 ms) 2024-09-13T15:13:38.916 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-13T15:13:38.918 INFO:tasks.workunit.client.0.smithi154.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4-843-g2c01458a/rpm/el9/BUILD/ceph-18.2.4-843-g2c01458a/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-13T15:13:38.918 INFO:tasks.workunit.client.0.smithi154.stdout:Expected equality of these values: 2024-09-13T15:13:38.918 INFO:tasks.workunit.client.0.smithi154.stdout: 1U 2024-09-13T15:13:38.919 INFO:tasks.workunit.client.0.smithi154.stdout: Which is: 1 2024-09-13T15:13:38.919 INFO:tasks.workunit.client.0.smithi154.stdout: keys.size() 2024-09-13T15:13:38.919 INFO:tasks.workunit.client.0.smithi154.stdout: Which is: 3 2024-09-13T15:13:38.919 INFO:tasks.workunit.client.0.smithi154.stdout:[ FAILED ] TestClsRbd.group_snap_set (2 ms) 2024-09-13T15:13:38.919 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-13T15:13:38.921 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-09-13T15:13:38.921 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-13T15:13:38.938 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (17 ms) 2024-09-13T15:13:38.938 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-13T15:13:38.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:38.963 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:38.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:39.064 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:39.085 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:39.179 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (240 ms) 2024-09-13T15:13:39.179 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-13T15:13:39.182 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_remove (4 ms) 2024-09-13T15:13:39.182 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-13T15:13:39.185 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-13T15:13:39.185 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-13T15:13:39.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:39.192 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.trash_methods (8 ms) 2024-09-13T15:13:39.192 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.op_features 2024-09-13T15:13:39.198 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.op_features (6 ms) 2024-09-13T15:13:39.198 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-13T15:13:39.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:39.217 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.clone_parent (19 ms) 2024-09-13T15:13:39.217 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-13T15:13:39.225 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.clone_parent_ns (8 ms) 2024-09-13T15:13:39.225 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-13T15:13:39.236 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2024-09-13T15:13:39.236 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-13T15:13:39.243 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.namespace_methods (6 ms) 2024-09-13T15:13:39.243 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.migration 2024-09-13T15:13:39.251 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.migration (9 ms) 2024-09-13T15:13:39.252 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-13T15:13:39.260 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.migration_v1 (8 ms) 2024-09-13T15:13:39.260 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-13T15:13:39.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:39.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:39.400 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:39 smithi172 ceph-mon[100696]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:39.400 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:39 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:13:39.400 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:39 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3453269835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:39.400 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:39 smithi172 ceph-mon[100696]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T15:13:39.400 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:39 smithi172 ceph-mon[100696]: pgmap v1591: 125 pgs: 2 creating+peering, 6 unknown, 117 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 1.03k op/s 2024-09-13T15:13:39.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2 2024-09-13T15:13:39.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.2' 2024-09-13T15:13:39.560 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:39.583 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:39.685 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:39.706 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[102314]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:13:39.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3453269835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[102314]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T15:13:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[102314]: pgmap v1591: 125 pgs: 2 creating+peering, 6 unknown, 117 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 1.03k op/s 2024-09-13T15:13:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[107038]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:13:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3453269835' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[107038]: osdmap e491: 8 total, 8 up, 8 in 2024-09-13T15:13:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:39 smithi154 ceph-mon[107038]: pgmap v1591: 125 pgs: 2 creating+peering, 6 unknown, 117 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 1.03k op/s 2024-09-13T15:13:39.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.831 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:39.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:39.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:39.953 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:39.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:39.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:40.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:40.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:40.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:40.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5 2024-09-13T15:13:40.207 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.5' 2024-09-13T15:13:40.246 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:13:40.246 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:13:40.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:40.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:40.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:40.385 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1125 ms) 2024-09-13T15:13:40.385 INFO:tasks.workunit.client.0.smithi154.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-13T15:13:40.422 INFO:tasks.workunit.client.0.smithi154.stdout:[ OK ] TestClsRbd.sparsify (38 ms) 2024-09-13T15:13:40.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:40.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:40.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:40.584 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:40.686 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.708 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:40.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:40.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:40 smithi154 ceph-mon[102314]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T15:13:40.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:40 smithi154 ceph-mon[107038]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T15:13:40.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:40 smithi172 ceph-mon[100696]: osdmap e492: 8 total, 8 up, 8 in 2024-09-13T15:13:40.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:40.832 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:40.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:40.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-13T15:13:40.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4 2024-09-13T15:13:40.955 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:40.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.4' 2024-09-13T15:13:41.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:41.079 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:41.091 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:41.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:41.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:41.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:41.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:41.402 INFO:tasks.workunit.client.0.smithi154.stdout:[----------] 70 tests from TestClsRbd (5085 ms total) 2024-09-13T15:13:41.402 INFO:tasks.workunit.client.0.smithi154.stdout: 2024-09-13T15:13:41.403 INFO:tasks.workunit.client.0.smithi154.stdout:[----------] Global test environment tear-down 2024-09-13T15:13:41.403 INFO:tasks.workunit.client.0.smithi154.stdout:[==========] 70 tests from 1 test suite ran. (7533 ms total) 2024-09-13T15:13:41.403 INFO:tasks.workunit.client.0.smithi154.stdout:[ PASSED ] 69 tests. 2024-09-13T15:13:41.403 INFO:tasks.workunit.client.0.smithi154.stdout:[ FAILED ] 1 test, listed below: 2024-09-13T15:13:41.403 INFO:tasks.workunit.client.0.smithi154.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-13T15:13:41.403 INFO:tasks.workunit.client.0.smithi154.stdout: 2024-09-13T15:13:41.403 INFO:tasks.workunit.client.0.smithi154.stdout: 1 FAILED TEST 2024-09-13T15:13:41.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1 2024-09-13T15:13:41.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.1' 2024-09-13T15:13:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:41 smithi172 ceph-mon[100696]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T15:13:41.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:41 smithi172 ceph-mon[100696]: pgmap v1594: 125 pgs: 125 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 1.09k op/s 2024-09-13T15:13:41.563 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.586 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-13T15:13:41.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T15:13:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.588 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:41.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:41.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3 2024-09-13T15:13:41.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.3' 2024-09-13T15:13:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:41 smithi154 ceph-mon[102314]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T15:13:41.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:41 smithi154 ceph-mon[102314]: pgmap v1594: 125 pgs: 125 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 1.09k op/s 2024-09-13T15:13:41.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:41 smithi154 ceph-mon[107038]: osdmap e493: 8 total, 8 up, 8 in 2024-09-13T15:13:41.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:41 smithi154 ceph-mon[107038]: pgmap v1594: 125 pgs: 125 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 1.09k op/s 2024-09-13T15:13:41.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:41.835 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:41.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0 2024-09-13T15:13:41.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-13T15:13:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:41.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.0' 2024-09-13T15:13:42.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6 2024-09-13T15:13:42.082 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:42.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.6' 2024-09-13T15:13:42.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7 2024-09-13T15:13:42.201 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:13:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:13:41] "GET /metrics HTTP/1.1" 200 34225 "" "Prometheus/2.51.0" 2024-09-13T15:13:42.205 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T15:13:42.206 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: docker: command not found 2024-09-13T15:13:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:13:42.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi172 with status 1: 'sudo docker kill -s 1 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd.7' 2024-09-13T15:13:42.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:42 smithi154 ceph-mon[102314]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T15:13:42.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:42 smithi154 ceph-mon[107038]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T15:13:42.731 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:13:42.731 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:42 smithi172 ceph-mon[100696]: osdmap e494: 8 total, 8 up, 8 in 2024-09-13T15:13:42.942 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:13:42.942 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:13:43.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T15:13:43.386 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:43.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-13T15:13:43.790 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:44.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:44.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:44 smithi154 ceph-mon[102314]: pgmap v1596: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:44.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:44 smithi154 ceph-mon[107038]: pgmap v1596: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:44.226 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:44.226 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:44.226 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:44 smithi172 ceph-mon[100696]: pgmap v1596: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:44.251 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:44.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset noscrub 2024-09-13T15:13:45.357 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-09-13T15:13:45.357 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T15:13:45.544 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:45 smithi154 ceph-mon[102314]: pgmap v1597: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:45.544 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:45 smithi154 ceph-mon[107038]: pgmap v1597: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:45.567 INFO:teuthology.orchestra.run.smithi154.stdout:osd.0: {} 2024-09-13T15:13:45.567 INFO:teuthology.orchestra.run.smithi154.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T15:13:45.584 INFO:teuthology.orchestra.run.smithi154.stdout:osd.1: {} 2024-09-13T15:13:45.584 INFO:teuthology.orchestra.run.smithi154.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T15:13:45.600 INFO:teuthology.orchestra.run.smithi154.stdout:osd.2: {} 2024-09-13T15:13:45.600 INFO:teuthology.orchestra.run.smithi154.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T15:13:45.616 INFO:teuthology.orchestra.run.smithi154.stdout:osd.3: {} 2024-09-13T15:13:45.616 INFO:teuthology.orchestra.run.smithi154.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T15:13:45.633 INFO:teuthology.orchestra.run.smithi154.stdout:osd.4: {} 2024-09-13T15:13:45.633 INFO:teuthology.orchestra.run.smithi154.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T15:13:45.649 INFO:teuthology.orchestra.run.smithi154.stdout:osd.5: {} 2024-09-13T15:13:45.649 INFO:teuthology.orchestra.run.smithi154.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T15:13:45.665 INFO:teuthology.orchestra.run.smithi154.stdout:osd.6: {} 2024-09-13T15:13:45.665 INFO:teuthology.orchestra.run.smithi154.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T15:13:45.681 INFO:teuthology.orchestra.run.smithi154.stdout:osd.7: {} 2024-09-13T15:13:45.681 INFO:teuthology.orchestra.run.smithi154.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T15:13:45.720 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:45 smithi172 ceph-mon[100696]: pgmap v1597: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:46.000 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-13T15:13:46.213 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:46.559 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:46 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2102287499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:46.560 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:46 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2102287499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:46.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:46 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2102287499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-13T15:13:47.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:47 smithi154 ceph-mon[102314]: pgmap v1598: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:47 smithi154 ceph-mon[107038]: pgmap v1598: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:47 smithi172 ceph-mon[100696]: pgmap v1598: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:47.543 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:135001 9.0 e407/374)", 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:07:21.253118+0000", 2024-09-13T15:13:48.059 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 386.80318534200001, 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.092129774999999997, 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253118+0000", 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253118+0000", 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253117+0000", 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253126+0000", 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4409999999999999e-06 2024-09-13T15:13:48.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253142+0000", 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6256000000000002e-05 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.253149+0000", 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5579999999999997e-06 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.345228+0000", 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.092078907000000002 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.345247+0000", 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9806999999999999e-05 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.063 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.45983.0:135004 9.1 e407/390)", 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:07:21.256337+0000", 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 386.79996634100002, 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.113890226, 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256337+0000", 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256337+0000", 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256336+0000", 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256345+0000", 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9169999999999995e-06 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256363+0000", 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7989e-05 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.256371+0000", 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.303e-06 2024-09-13T15:13:48.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.370207+0000", 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.113835671 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:07:21.370227+0000", 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0285999999999999e-05 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:110129 10.e e465/448 10:76c5518a:::benchmark_data_smithi154_317818_object110128:head v 465'6877, mlcod=465'6873)", 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:40.263572+0000", 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 187.792731304, 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042002386000000003, 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263572+0000", 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263572+0000", 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263571+0000", 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263595+0000", 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4114e-05 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263615+0000", 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9522e-05 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263634+0000", 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9315999999999999e-05 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263813+0000", 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017923700000000001 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.263857+0000", 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3520999999999999e-05 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305474+0000", 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041617020999999997 2024-09-13T15:13:48.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305574+0000", 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100549 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:110130 10.e e465/448 10:7f0c4a40:::benchmark_data_smithi154_317818_object110129:head v 465'6878, mlcod=465'6874)", 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:40.269985+0000", 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 187.786317653, 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035673300999999998, 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.269985+0000", 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.269985+0000", 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.269984+0000", 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270007+0000", 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2802999999999998e-05 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270019+0000", 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1476e-05 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270026+0000", 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.939e-06 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270531+0000", 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050518399999999999 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.270552+0000", 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1248000000000001e-05 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305580+0000", 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035027957999999998 2024-09-13T15:13:48.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:40.305659+0000", 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8716999999999999e-05 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.078 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125908 10.7 e466/448 10:e0fbea1e:::benchmark_data_smithi154_317818_object232:head v 466'7768, mlcod=466'7766)", 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 170.86367360099999, 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041463458000000002, 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192629+0000", 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192637+0000", 2024-09-13T15:13:48.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9710000000000004e-06 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192654+0000", 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7268000000000001e-05 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192661+0000", 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3760000000000002e-06 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192720+0000", 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9172000000000002e-05 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.192748+0000", 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7070999999999999e-05 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.233921+0000", 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041173123999999998 2024-09-13T15:13:48.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234093+0000", 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000172299 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125909 10.f e466/448 10:f830a73c:::benchmark_data_smithi154_317818_object233:head v 466'7757, mlcod=466'7755)", 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.193728+0000", 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 170.86257554299999, 2024-09-13T15:13:48.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041072277999999997, 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193728+0000", 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193728+0000", 2024-09-13T15:13:48.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193727+0000", 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193734+0000", 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5560000000000002e-06 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193749+0000", 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.193757+0000", 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5850000000000002e-06 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.196627+0000", 2024-09-13T15:13:48.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028706080000000002 2024-09-13T15:13:48.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.196682+0000", 2024-09-13T15:13:48.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4557e-05 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234098+0000", 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037416505000000003 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234800+0000", 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070148499999999996 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:125912 10.f e466/448 10:f29520b2:::benchmark_data_smithi154_317818_object236:head v 466'7758, mlcod=466'7755)", 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.197918+0000", 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 170.85838496, 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037050997000000002, 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.089 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197918+0000", 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197918+0000", 2024-09-13T15:13:48.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197917+0000", 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197926+0000", 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2459999999999992e-06 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197943+0000", 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6891000000000001e-05 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.197951+0000", 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4290000000000004e-06 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.200598+0000", 2024-09-13T15:13:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026478259999999998 2024-09-13T15:13:48.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.200614+0000", 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5959000000000001e-05 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234804+0000", 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034189123000000002 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.234969+0000", 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016555300000000001 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:126346 10.b e466/448 10:d0729132:::benchmark_data_smithi154_317818_object670:head v 466'8002, mlcod=466'8000)", 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:57.702779+0000", 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 170.35352363600001, 2024-09-13T15:13:48.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042127080999999997, 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.702779+0000", 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.702779+0000", 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.702779+0000", 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.702784+0000", 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6370000000000004e-06 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.702800+0000", 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6062e-05 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.702807+0000", 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.224e-06 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.704503+0000", 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016967460000000001 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.704534+0000", 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0315999999999999e-05 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.744753+0000", 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040219234999999999 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:57.744907+0000", 2024-09-13T15:13:48.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015352000000000001 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:127301 10.0 e466/448 10:0fa32f11:::benchmark_data_smithi154_317818_object1625:head v 466'8041, mlcod=466'8036)", 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:58.748590+0000", 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.30771334100001, 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040205800999999999, 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.100 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.748590+0000", 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.748590+0000", 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.748589+0000", 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.748595+0000", 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8000000000000004e-06 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.748605+0000", 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0542e-05 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.748611+0000", 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9519999999999999e-06 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.754030+0000", 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054189479999999998 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.754083+0000", 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2164999999999999e-05 2024-09-13T15:13:48.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.788721+0000", 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034638356000000002 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.788796+0000", 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4622000000000005e-05 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:127372 10.b e466/448 10:d3c46655:::benchmark_data_smithi154_317818_object1696:head v 466'8071, mlcod=466'8066)", 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:58.852128+0000", 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.20417532299999, 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.046346976999999998, 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.852128+0000", 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.852128+0000", 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.852127+0000", 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.852131+0000", 2024-09-13T15:13:48.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2059999999999999e-06 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.852136+0000", 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4090000000000002e-06 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.852140+0000", 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8099999999999999e-06 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.854289+0000", 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021493010000000002 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.854332+0000", 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2951999999999998e-05 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.898138+0000", 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043805605999999997 2024-09-13T15:13:48.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.898475+0000", 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033696699999999997 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:127383 10.b e466/448 10:dd81c3ff:::benchmark_data_smithi154_317818_object1707:head v 466'8072, mlcod=466'8066)", 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:58.859071+0000", 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.197231977, 2024-09-13T15:13:48.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039428091999999998, 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.859071+0000", 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.859071+0000", 2024-09-13T15:13:48.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.859070+0000", 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.859078+0000", 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9829999999999996e-06 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.859097+0000", 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8867000000000001e-05 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.859106+0000", 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9539999999999993e-06 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.867112+0000", 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008006549 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.867129+0000", 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7108e-05 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.898480+0000", 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031350836999999999 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.898499+0000", 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:127388 10.b e466/448 10:dfdab130:::benchmark_data_smithi154_317818_object1712:head v 466'8073, mlcod=466'8066)", 2024-09-13T15:13:48.114 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:58.863758+0000", 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.19254475100001, 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040574329999999999, 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.863758+0000", 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.863758+0000", 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.863758+0000", 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.863763+0000", 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4589999999999997e-06 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.863772+0000", 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2769999999999994e-06 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.863778+0000", 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7420000000000003e-06 2024-09-13T15:13:48.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.871710+0000", 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079320970000000008 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.871750+0000", 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9796999999999999e-05 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.904299+0000", 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032548803000000001 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.904333+0000", 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3952999999999998e-05 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:127391 10.b e466/448 10:d5aed4ff:::benchmark_data_smithi154_317818_object1715:head v 466'8074, mlcod=466'8067)", 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:58.870490+0000", 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.18581265099999, 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038267111999999999, 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.870490+0000", 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.870490+0000", 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.870490+0000", 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.870497+0000", 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0280000000000003e-06 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.870506+0000", 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2080000000000006e-06 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.870512+0000", 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6679999999999998e-06 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.876700+0000", 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061887849999999996 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.876721+0000", 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0245e-05 2024-09-13T15:13:48.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.908599+0000", 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031878319000000002 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.908758+0000", 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000158555 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.124 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:127393 10.b e466/448 10:d713235d:::benchmark_data_smithi154_317818_object1717:head v 466'8075, mlcod=466'8069)", 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:58.874305+0000", 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 169.181998268, 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039013280999999997, 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.874305+0000", 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.874305+0000", 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.874305+0000", 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.874308+0000", 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5310000000000002e-06 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.874313+0000", 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1590000000000001e-06 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.874317+0000", 2024-09-13T15:13:48.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9620000000000004e-06 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.898523+0000", 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024205495 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.898540+0000", 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7246000000000001e-05 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.913211+0000", 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014670723 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:58.913318+0000", 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000107422 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.129 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:157210 10.7 e469/448 10:e3d1f3b4:::benchmark_data_smithi154_317818_object31534:head v 469'9742, mlcod=469'9740)", 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:32.458352+0000", 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 135.59795098699999, 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049394774000000002, 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458352+0000", 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458352+0000", 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458351+0000", 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458357+0000", 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-09-13T15:13:48.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458367+0000", 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4059999999999994e-06 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458372+0000", 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2499999999999997e-06 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458410+0000", 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7638000000000001e-05 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.458428+0000", 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8828999999999999e-05 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.463209+0000", 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0047807509999999998 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.507747+0000", 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044537685 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:157222 10.0 e469/448 10:0c0d9846:::benchmark_data_smithi154_317818_object31546:head v 469'9923, mlcod=469'9921)", 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:32.460813+0000", 2024-09-13T15:13:48.134 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 135.595490211, 2024-09-13T15:13:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.048476955000000002, 2024-09-13T15:13:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.136 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460813+0000", 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460813+0000", 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460812+0000", 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460820+0000", 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7770000000000002e-06 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460832+0000", 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2123999999999999e-05 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460838+0000", 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3289999999999999e-06 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460880+0000", 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1545999999999998e-05 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.460899+0000", 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9091000000000001e-05 2024-09-13T15:13:48.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.509187+0000", 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.048287911000000003 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.509290+0000", 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010317799999999999 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:157227 10.b e469/448 10:db3daa76:::benchmark_data_smithi154_317818_object31551:head v 469'9970, mlcod=469'9968)", 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:32.465215+0000", 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 135.59108850999999, 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.056422324000000003, 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465215+0000", 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465215+0000", 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465214+0000", 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465220+0000", 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7739999999999999e-06 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.507545+0000", 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042324878000000003 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.507560+0000", 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4898e-05 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.509412+0000", 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018513900000000001 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.509457+0000", 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5207999999999998e-05 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.521610+0000", 2024-09-13T15:13:48.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012152727 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.521637+0000", 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7328999999999999e-05 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:157216 10.f e469/448 10:ff04a8ae:::benchmark_data_smithi154_317818_object31540:head v 469'9692, mlcod=469'9689)", 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:32.465866+0000", 2024-09-13T15:13:48.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 135.59043720400001, 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.052306185999999998, 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465866+0000", 2024-09-13T15:13:48.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465866+0000", 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465866+0000", 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.465871+0000", 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.028e-06 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.507766+0000", 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.041895802000000003 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.507772+0000", 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2680000000000002e-06 2024-09-13T15:13:48.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.507880+0000", 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108453 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.507899+0000", 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8876e-05 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.518131+0000", 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010231743999999999 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.518172+0000", 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1366999999999999e-05 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:157232 10.e e469/448 10:7a95ee40:::benchmark_data_smithi154_317818_object31556:head v 469'9850, mlcod=469'9848)", 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:32.473661+0000", 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 135.58264245500001, 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045154053, 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.473661+0000", 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.473661+0000", 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.473660+0000", 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.473666+0000", 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0789999999999997e-06 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.510707+0000", 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037041039999999997 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.510718+0000", 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0868e-05 2024-09-13T15:13:48.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.511428+0000", 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070998399999999998 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.511446+0000", 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8014000000000001e-05 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.518726+0000", 2024-09-13T15:13:48.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.007280407 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:32.518815+0000", 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8407999999999998e-05 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.36270.0:180900 10.b e471/448 10:de2788e8:::benchmark_data_smithi154_317818_object55224:head v 471'11486, mlcod=471'11484)", 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:11:59.219034+0000", 2024-09-13T15:13:48.155 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 108.837269351, 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036563443000000001, 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219034+0000", 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219034+0000", 2024-09-13T15:13:48.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219033+0000", 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219040+0000", 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5010000000000001e-06 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219052+0000", 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2704000000000001e-05 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219058+0000", 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9989999999999997e-06 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219225+0000", 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016633800000000001 2024-09-13T15:13:48.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.219244+0000", 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9253e-05 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.255494+0000", 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036250177000000001 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:48.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:11:59.255597+0000", 2024-09-13T15:13:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000103182 2024-09-13T15:13:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:48.160 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:48.160 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:13:48.383 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T15:13:48.665 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1887009294' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:13:48.666 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:48 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:13:48.666 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1887009294' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:13:48.666 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:48 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:13:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1887009294' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:13:48.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:48 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:13:48.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-13T15:13:49.003 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:49.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:13:49.615 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:49 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:13:49.615 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:49 smithi154 ceph-mon[102314]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T15:13:49.615 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:49 smithi154 ceph-mon[102314]: pgmap v1600: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:49.615 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:49 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:13:49.615 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:49 smithi154 ceph-mon[107038]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T15:13:49.615 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:49 smithi154 ceph-mon[107038]: pgmap v1600: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:49 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:13:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:49 smithi172 ceph-mon[100696]: osdmap e495: 8 total, 8 up, 8 in 2024-09-13T15:13:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:49 smithi172 ceph-mon[100696]: pgmap v1600: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:49.871 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:50.399 INFO:teuthology.orchestra.run.smithi154.stderr:noscrub is unset 2024-09-13T15:13:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:50 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3760919464' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:50.580 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:13:50.581 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:13:50.581 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-09-13T15:13:50.581 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:13:50.592 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:50 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3760919464' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:50.592 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:50 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3760919464' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-13T15:13:51.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-13T15:13:51.438 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:51 smithi172 ceph-mon[100696]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3760919464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:13:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:51 smithi172 ceph-mon[100696]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T15:13:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:51 smithi172 ceph-mon[100696]: pgmap v1602: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:51.606 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[102314]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:51.607 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3760919464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:13:51.607 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[102314]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T15:13:51.607 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[102314]: pgmap v1602: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:51.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[107038]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-13T15:13:51.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3760919464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-13T15:13:51.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[107038]: osdmap e496: 8 total, 8 up, 8 in 2024-09-13T15:13:51.608 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-mon[107038]: pgmap v1602: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:51.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-13T15:13:52.192 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:13:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:13:51] "GET /metrics HTTP/1.1" 200 34227 "" "Prometheus/2.51.0" 2024-09-13T15:13:52.384 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:52.539 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:13:52.789 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:52 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1876520286' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:13:52.789 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:52 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:13:52.963 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:52 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1876520286' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:13:52.964 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:52 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:13:52.964 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:52 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1876520286' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:13:52.964 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:52 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:13:53.194 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:13:53.195 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-09-13T15:13:53.195 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-09-13T15:13:53.195 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-09-13T15:13:53.195 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:13:53.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T15:13:53.799 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:53 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:13:53.799 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:53 smithi172 ceph-mon[100696]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T15:13:53.799 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:53 smithi172 ceph-mon[100696]: pgmap v1604: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:53.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-13T15:13:53.862 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:53 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:13:53.863 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:53 smithi154 ceph-mon[102314]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T15:13:53.863 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:53 smithi154 ceph-mon[102314]: pgmap v1604: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:53.863 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:53 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:13:53.863 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:53 smithi154 ceph-mon[107038]: osdmap e497: 8 total, 8 up, 8 in 2024-09-13T15:13:53.863 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:53 smithi154 ceph-mon[107038]: pgmap v1604: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:54.018 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.b/config 2024-09-13T15:13:54.721 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:54.935 INFO:teuthology.orchestra.run.smithi154.stderr:nodeep-scrub is unset 2024-09-13T15:13:55.220 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/37929761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:55.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:54 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/37929761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:55.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:54 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/37929761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:55.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:54 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-13T15:13:55.676 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:182891 9.15 9.519c19d5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201395+0000", 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47235060600002, 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98277276700000005, 2024-09-13T15:13:55.677 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 182891 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201395+0000", 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201395+0000", 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201393+0000", 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201400+0000", 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9580000000000001e-06 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201432+0000", 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1748000000000002e-05 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201439+0000", 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8229999999999997e-06 2024-09-13T15:13:55.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201481+0000", 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2898000000000002e-05 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201497+0000", 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5862000000000002e-05 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184156+0000", 2024-09-13T15:13:55.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98265885900000005 2024-09-13T15:13:55.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184168+0000", 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1684e-05 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:182900 9.15 9.77a38d95 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201444+0000", 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47230132599998, 2024-09-13T15:13:55.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98275084099999999, 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 182900 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201444+0000", 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201444+0000", 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201444+0000", 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201446+0000", 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8589999999999999e-06 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201450+0000", 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.031e-06 2024-09-13T15:13:55.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201453+0000", 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.762e-06 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201522+0000", 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9426999999999998e-05 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201526+0000", 2024-09-13T15:13:55.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8e-06 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184187+0000", 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98266105299999995 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184195+0000", 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2300000000000008e-06 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:182944 9.15 9.d8c2b435 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201475+0000", 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47227068900003, 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98274072999999995, 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.689 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 182944 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201475+0000", 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201475+0000", 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201475+0000", 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201477+0000", 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9309999999999998e-06 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201493+0000", 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6169000000000001e-05 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201498+0000", 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201545+0000", 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7859999999999999e-05 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201548+0000", 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9289999999999998e-06 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184212+0000", 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98266316200000003 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184216+0000", 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2379999999999999e-06 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:182948 9.15 9.8f79795 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.694 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201501+0000", 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47224507599998, 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98282723900000002, 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 182948 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201501+0000", 2024-09-13T15:13:55.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201501+0000", 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201500+0000", 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201502+0000", 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6789999999999999e-06 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201505+0000", 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3079999999999999e-06 2024-09-13T15:13:55.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201508+0000", 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9809999999999999e-06 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201567+0000", 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9104999999999997e-05 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201572+0000", 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6569999999999996e-06 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184319+0000", 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98274731500000001 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.184328+0000", 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4419999999999998e-06 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.699 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:182957 9.15 9.7278f055 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201511+0000", 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47223513799997, 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98407714099999999, 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.700 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 182957 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201511+0000", 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201511+0000", 2024-09-13T15:13:55.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201510+0000", 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201512+0000", 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.39e-06 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201515+0000", 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8109999999999999e-06 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201517+0000", 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201596+0000", 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8792e-05 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201601+0000", 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0350000000000001e-06 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185579+0000", 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98397769199999996 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185588+0000", 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0129999999999999e-06 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.705 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183037 9.15 9.2b41f575 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201519+0000", 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47222660300002, 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.984089836, 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.706 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183037 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201519+0000", 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201519+0000", 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201519+0000", 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201520+0000", 2024-09-13T15:13:55.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.291e-06 2024-09-13T15:13:55.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201523+0000", 2024-09-13T15:13:55.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9129999999999999e-06 2024-09-13T15:13:55.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201526+0000", 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.57e-06 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201650+0000", 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000124065 2024-09-13T15:13:55.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201657+0000", 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5599999999999996e-06 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185604+0000", 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98394623299999995 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185609+0000", 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4179999999999996e-06 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183044 9.15 9.a8017fb5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.713 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201527+0000", 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47221821699998, 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98410804100000004, 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183044 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201527+0000", 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201527+0000", 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201527+0000", 2024-09-13T15:13:55.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201528+0000", 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.144e-06 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201531+0000", 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8119999999999999e-06 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201534+0000", 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3879999999999998e-06 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201699+0000", 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016496699999999999 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201704+0000", 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6400000000000002e-06 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185628+0000", 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98392378199999997 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185636+0000", 2024-09-13T15:13:55.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.525e-06 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183073 9.15 9.2f961415 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201536+0000", 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47221014799999, 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98412685499999997, 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183073 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201536+0000", 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201536+0000", 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201535+0000", 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201540+0000", 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4460000000000003e-06 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201543+0000", 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8619999999999999e-06 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201545+0000", 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4210000000000002e-06 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201737+0000", 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019165600000000001 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201742+0000", 2024-09-13T15:13:55.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6219999999999997e-06 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185655+0000", 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98391266099999997 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185662+0000", 2024-09-13T15:13:55.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4039999999999998e-06 2024-09-13T15:13:55.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.725 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.725 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.725 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183078 9.15 9.d926c2b5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201547+0000", 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47219877700002, 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98415375000000005, 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183078 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201547+0000", 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201547+0000", 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201547+0000", 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201548+0000", 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1319999999999999e-06 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201551+0000", 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8229999999999999e-06 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201553+0000", 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5469999999999999e-06 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201772+0000", 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021904199999999999 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201778+0000", 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3730000000000001e-06 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185692+0000", 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98391435299999996 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185701+0000", 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6920000000000008e-06 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183172 9.15 9.2596de75 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201568+0000", 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47217783899998, 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98418606600000003, 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.731 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183172 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201568+0000", 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201568+0000", 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201567+0000", 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201569+0000", 2024-09-13T15:13:55.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9470000000000002e-06 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201575+0000", 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8259999999999998e-06 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201580+0000", 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6399999999999996e-06 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201806+0000", 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022656299999999999 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201812+0000", 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6509999999999998e-06 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185737+0000", 2024-09-13T15:13:55.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98392468300000002 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185754+0000", 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7147999999999999e-05 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183190 9.15 9.8dace635 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.736 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201598+0000", 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47214748599998, 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98423309400000003, 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183190 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.737 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201598+0000", 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201598+0000", 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201598+0000", 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201600+0000", 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7120000000000001e-06 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201612+0000", 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1882000000000001e-05 2024-09-13T15:13:55.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201615+0000", 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.377e-06 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201839+0000", 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000224448 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201845+0000", 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.9549999999999997e-06 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185824+0000", 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98397832799999996 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185831+0000", 2024-09-13T15:13:55.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7069999999999992e-06 2024-09-13T15:13:55.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.742 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.742 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183191 9.15 9.c2cdff55 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.743 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201643+0000", 2024-09-13T15:13:55.743 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47210273000002, 2024-09-13T15:13:55.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98421607499999997, 2024-09-13T15:13:55.743 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.743 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.743 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.744 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183191 2024-09-13T15:13:55.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.744 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201643+0000", 2024-09-13T15:13:55.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201643+0000", 2024-09-13T15:13:55.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201643+0000", 2024-09-13T15:13:55.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:55.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201644+0000", 2024-09-13T15:13:55.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4959999999999999e-06 2024-09-13T15:13:55.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201648+0000", 2024-09-13T15:13:55.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3950000000000001e-06 2024-09-13T15:13:55.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201650+0000", 2024-09-13T15:13:55.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6400000000000001e-06 2024-09-13T15:13:55.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201876+0000", 2024-09-13T15:13:55.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022624599999999999 2024-09-13T15:13:55.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201883+0000", 2024-09-13T15:13:55.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0079999999999999e-06 2024-09-13T15:13:55.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185851+0000", 2024-09-13T15:13:55.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98396878200000004 2024-09-13T15:13:55.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185859+0000", 2024-09-13T15:13:55.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7770000000000002e-06 2024-09-13T15:13:55.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.754 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.755 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183202 9.15 9.277773b5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.755 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201652+0000", 2024-09-13T15:13:55.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47209352900001, 2024-09-13T15:13:55.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.984232412, 2024-09-13T15:13:55.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.756 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.756 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183202 2024-09-13T15:13:55.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.757 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201652+0000", 2024-09-13T15:13:55.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201652+0000", 2024-09-13T15:13:55.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201652+0000", 2024-09-13T15:13:55.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201653+0000", 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0690000000000001e-06 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201656+0000", 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.915e-06 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201658+0000", 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.357e-06 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201913+0000", 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000254565 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201918+0000", 2024-09-13T15:13:55.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4419999999999997e-06 2024-09-13T15:13:55.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185878+0000", 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98395957999999994 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185885+0000", 2024-09-13T15:13:55.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6869999999999997e-06 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183262 9.15 9.deab2915 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201660+0000", 2024-09-13T15:13:55.764 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47208546899998, 2024-09-13T15:13:55.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98424595699999995, 2024-09-13T15:13:55.765 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.765 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183262 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201660+0000", 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201660+0000", 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201660+0000", 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201661+0000", 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0750000000000001e-06 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201664+0000", 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.23e-06 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201668+0000", 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2940000000000001e-06 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201947+0000", 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027895399999999999 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201952+0000", 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0869999999999997e-06 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185899+0000", 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98394744199999995 2024-09-13T15:13:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185906+0000", 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.092e-06 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183270 9.15 9.60d6c9b5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201695+0000", 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47205029200001, 2024-09-13T15:13:55.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98423579500000002, 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183270 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201695+0000", 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201695+0000", 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201695+0000", 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999995 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201697+0000", 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.765e-06 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201701+0000", 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0149999999999997e-06 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201704+0000", 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8420000000000001e-06 2024-09-13T15:13:55.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201982+0000", 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000278325 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201988+0000", 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0260000000000004e-06 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185924+0000", 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98393626199999995 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185931+0000", 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8709999999999999e-06 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.45983.0:183274 9.15 9.a8316395 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e436)", 2024-09-13T15:13:55.775 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:08:30.201706+0000", 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 325.47204001699998, 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.98425050000000003, 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.45983", 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 183274 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201706+0000", 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201706+0000", 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201705+0000", 2024-09-13T15:13:55.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967296 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201707+0000", 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.062e-06 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201710+0000", 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.579e-06 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.201713+0000", 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5189999999999999e-06 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.202007+0000", 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029483200000000003 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-09-13T15:13:55.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:30.202014+0000", 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1809999999999998e-06 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185949+0000", 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.983935424 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:08:31.185956+0000", 2024-09-13T15:13:55.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1049999999999997e-06 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73757 10.c 10:3d6ad79f:::benchmark_data_smithi154_317818_object73756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.940730+0000", 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 229.73301615899999, 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.56075086699999999, 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.781 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73757 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.940730+0000", 2024-09-13T15:13:55.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.940730+0000", 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.940728+0000", 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.940769+0000", 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1486000000000002e-05 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.940785+0000", 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6103000000000002e-05 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.940792+0000", 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9450000000000004e-06 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.940978+0000", 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018521999999999999 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.941099+0000", 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121846 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.941223+0000", 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012390900000000001 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.943711+0000", 2024-09-13T15:13:55.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0024872240000000001 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.944846+0000", 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001135498 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501387+0000", 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.55654110199999995 2024-09-13T15:13:55.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501435+0000", 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7719000000000002e-05 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501480+0000", 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5552e-05 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73770 10.c 10:3ea51f52:::benchmark_data_smithi154_317818_object73769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.952026+0000", 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 229.72171944799999, 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.55550833700000002, 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73770 2024-09-13T15:13:55.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952026+0000", 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952026+0000", 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952024+0000", 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999981 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952077+0000", 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2342000000000001e-05 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952089+0000", 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2019e-05 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952096+0000", 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6450000000000005e-06 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952133+0000", 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6452000000000003e-05 2024-09-13T15:13:55.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.952242+0000", 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0001094 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.496225+0000", 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54398237100000002 2024-09-13T15:13:55.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501526+0000", 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053016460000000001 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501584+0000", 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7321000000000002e-05 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.507392+0000", 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058085009999999998 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.507492+0000", 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9481000000000005e-05 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.507535+0000", 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3010999999999999e-05 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop_reply(client.36270.0:73757 10.c e454/448 ondisk, result = 0)", 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.953263+0000", 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 229.720482726, 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.54822603999999997, 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.796 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.953263+0000", 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.953263+0000", 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.953262+0000", 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.999999 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.953267+0000", 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2799999999999997e-06 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.496354+0000", 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.543087247 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.496363+0000", 2024-09-13T15:13:55.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1819999999999995e-06 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501371+0000", 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050075279999999998 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501380+0000", 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000001e-06 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501489+0000", 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108624 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.36270.0:73778 10.c 10:32c48017:::benchmark_data_smithi154_317818_object73777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-09-13T15:10:05.959738+0000", 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 229.714007724, 2024-09-13T15:13:55.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.547984527, 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.36270", 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 73778 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959738+0000", 2024-09-13T15:13:55.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959738+0000", 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-09-13T15:13:55.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959736+0000", 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4294967295.9999986 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:05.959816+0000", 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9676000000000001e-05 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-09-13T15:13:55.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.496152+0000", 2024-09-13T15:13:55.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.53633572600000001 2024-09-13T15:13:55.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-09-13T15:13:55.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.496178+0000", 2024-09-13T15:13:55.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6440000000000001e-05 2024-09-13T15:13:55.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-09-13T15:13:55.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.499126+0000", 2024-09-13T15:13:55.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029478939999999999 2024-09-13T15:13:55.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-09-13T15:13:55.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.499360+0000", 2024-09-13T15:13:55.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023358499999999999 2024-09-13T15:13:55.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0,4", 2024-09-13T15:13:55.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.499471+0000", 2024-09-13T15:13:55.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111338 2024-09-13T15:13:55.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:55.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.501576+0000", 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002104879 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.507351+0000", 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005775054 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:13:55.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.507605+0000", 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000254317 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.507687+0000", 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2041999999999998e-05 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-09-13T15:10:06.507723+0000", 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5169999999999997e-05 2024-09-13T15:13:55.811 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.812 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-09-13T15:13:55.812 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-09-13T15:13:55.812 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-13T15:13:56.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[102314]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:13:56.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:13:56.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:56.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[102314]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T15:13:56.002 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[102314]: pgmap v1606: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:56.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[107038]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:13:56.002 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:13:56.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:56.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[107038]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T15:13:56.003 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:55 smithi154 ceph-mon[107038]: pgmap v1606: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:55 smithi172 ceph-mon[100696]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-13T15:13:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:55 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:13:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:55 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-13T15:13:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:55 smithi172 ceph-mon[100696]: osdmap e498: 8 total, 8 up, 8 in 2024-09-13T15:13:56.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:55 smithi172 ceph-mon[100696]: pgmap v1606: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:56.525 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-13T15:13:57.140 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:13:57.266 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T15:13:57.697 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1646034568' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:13:57.698 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1646034568' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:13:57.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1646034568' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:13:58.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:13:58.633 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1646034568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:13:58.633 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[102314]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T15:13:58.633 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[102314]: pgmap v1608: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:58.634 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:58.634 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:58.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1646034568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:13:58.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[107038]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T15:13:58.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[107038]: pgmap v1608: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:58.634 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:58.635 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:13:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:58 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1646034568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:13:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:58 smithi172 ceph-mon[100696]: osdmap e499: 8 total, 8 up, 8 in 2024-09-13T15:13:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:58 smithi172 ceph-mon[100696]: pgmap v1608: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:13:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:13:58.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:13:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:13:59.524 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:13:59.524 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:13:59.525 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T15:13:59.525 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:14:00.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-13T15:14:00.205 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:00.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:00 smithi154 ceph-mon[102314]: pgmap v1609: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:00.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:00 smithi154 ceph-mon[107038]: pgmap v1609: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:00 smithi172 ceph-mon[100696]: pgmap v1609: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:01.420 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:14:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:14:01] "GET /metrics HTTP/1.1" 200 34229 "" "Prometheus/2.51.0" 2024-09-13T15:14:02.285 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:14:02.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:02 smithi154 ceph-mon[107038]: pgmap v1610: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:02.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:02 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/211173385' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:14:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:02 smithi154 ceph-mon[102314]: pgmap v1610: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:02.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:02 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/211173385' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:14:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:02 smithi172 ceph-mon[100696]: pgmap v1610: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:02.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:02 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/211173385' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:14:03.560 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/211173385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:14:03.561 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:03 smithi154 ceph-mon[102314]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T15:14:03.579 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/211173385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:14:03.579 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:03 smithi154 ceph-mon[107038]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T15:14:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/211173385' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:14:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:03 smithi172 ceph-mon[100696]: osdmap e500: 8 total, 8 up, 8 in 2024-09-13T15:14:04.244 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:14:04.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:14:04.402 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:14:04.402 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:14:04.403 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:14:04.403 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:14:04.403 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:14:04.505 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:04 smithi154 ceph-mon[102314]: pgmap v1612: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:04.506 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:04 smithi154 ceph-mon[107038]: pgmap v1612: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:04 smithi172 ceph-mon[100696]: pgmap v1612: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:05.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-13T15:14:05.320 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:06.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:06 smithi154 ceph-mon[102314]: pgmap v1613: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:06.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:06 smithi154 ceph-mon[107038]: pgmap v1613: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:06 smithi172 ceph-mon[100696]: pgmap v1613: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:07.116 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:07.287 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:14:07.288 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:14:08.639 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:08 smithi154 ceph-mon[102314]: pgmap v1614: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:08.639 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:08 smithi154 ceph-mon[102314]: from='client.46478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:08.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:08 smithi154 ceph-mon[107038]: pgmap v1614: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:08.639 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:08 smithi154 ceph-mon[107038]: from='client.46478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:08 smithi172 ceph-mon[100696]: pgmap v1614: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:08 smithi172 ceph-mon[100696]: from='client.46478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:09.024 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2024-09-13T15:14:07.275886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":36635,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4162,"ondisk_log_size":4162,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5898620,"kb_used_data":66140,"kb_used_omap":40,"kb_used_meta":5832343,"kb_avail":744062596,"statfs":{"total":767960285184,"available":761920098304,"internally_reserved":0,"allocated":67727360,"data_stored":46110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":5972319348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002444"},"pg_stats":[{"pgid":"1.11","version":"89'149","reported_seq":871,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295992+0000","last_change":"2024-09-13T15:13:01.179220+0000","last_active":"2024-09-13T15:13:57.295992+0000","last_peered":"2024-09-13T15:13:57.295992+0000","last_clean":"2024-09-13T15:13:57.295992+0000","last_became_active":"2024-09-13T15:12:50.169695+0000","last_became_peered":"2024-09-13T15:12:50.169695+0000","last_unstale":"2024-09-13T15:13:57.295992+0000","last_undegraded":"2024-09-13T15:13:57.295992+0000","last_fullsized":"2024-09-13T15:13:57.295992+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.179154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.179154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:04.917248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":224,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195793+0000","last_change":"2024-09-13T15:08:31.190995+0000","last_active":"2024-09-13T15:14:03.195793+0000","last_peered":"2024-09-13T15:14:03.195793+0000","last_clean":"2024-09-13T15:14:03.195793+0000","last_became_active":"2024-09-13T15:08:31.190610+0000","last_became_peered":"2024-09-13T15:08:31.190610+0000","last_unstale":"2024-09-13T15:14:03.195793+0000","last_undegraded":"2024-09-13T15:14:03.195793+0000","last_fullsized":"2024-09-13T15:14:03.195793+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:42.124757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":81,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295970+0000","last_change":"2024-09-13T15:08:31.191585+0000","last_active":"2024-09-13T15:13:57.295970+0000","last_peered":"2024-09-13T15:13:57.295970+0000","last_clean":"2024-09-13T15:13:57.295970+0000","last_became_active":"2024-09-13T15:08:31.184700+0000","last_became_peered":"2024-09-13T15:08:31.184700+0000","last_unstale":"2024-09-13T15:13:57.295970+0000","last_undegraded":"2024-09-13T15:13:57.295970+0000","last_fullsized":"2024-09-13T15:13:57.295970+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:14.376834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"89'149","reported_seq":839,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295636+0000","last_change":"2024-09-13T15:13:09.203525+0000","last_active":"2024-09-13T15:13:57.295636+0000","last_peered":"2024-09-13T15:13:57.295636+0000","last_clean":"2024-09-13T15:13:57.295636+0000","last_became_active":"2024-09-13T15:12:50.159713+0000","last_became_peered":"2024-09-13T15:12:50.159713+0000","last_unstale":"2024-09-13T15:13:57.295636+0000","last_undegraded":"2024-09-13T15:13:57.295636+0000","last_fullsized":"2024-09-13T15:13:57.295636+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:09.203387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:09.203387+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:07.004095+0000","scrub_duration":0.002438127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":358,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295907+0000","last_change":"2024-09-13T15:08:46.080613+0000","last_active":"2024-09-13T15:13:57.295907+0000","last_peered":"2024-09-13T15:13:57.295907+0000","last_clean":"2024-09-13T15:13:57.295907+0000","last_became_active":"2024-09-13T15:08:46.080320+0000","last_became_peered":"2024-09-13T15:08:46.080320+0000","last_unstale":"2024-09-13T15:13:57.295907+0000","last_undegraded":"2024-09-13T15:13:57.295907+0000","last_fullsized":"2024-09-13T15:13:57.295907+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:09.205609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":837,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295898+0000","last_change":"2024-09-13T15:08:31.196449+0000","last_active":"2024-09-13T15:13:57.295898+0000","last_peered":"2024-09-13T15:13:57.295898+0000","last_clean":"2024-09-13T15:13:57.295898+0000","last_became_active":"2024-09-13T15:08:31.196143+0000","last_became_peered":"2024-09-13T15:08:31.196143+0000","last_unstale":"2024-09-13T15:13:57.295898+0000","last_undegraded":"2024-09-13T15:13:57.295898+0000","last_fullsized":"2024-09-13T15:13:57.295898+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:53.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"89'149","reported_seq":925,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795677+0000","last_change":"2024-09-13T15:13:03.099175+0000","last_active":"2024-09-13T15:14:02.795677+0000","last_peered":"2024-09-13T15:14:02.795677+0000","last_clean":"2024-09-13T15:14:02.795677+0000","last_became_active":"2024-09-13T15:12:50.196974+0000","last_became_peered":"2024-09-13T15:12:50.196974+0000","last_unstale":"2024-09-13T15:14:02.795677+0000","last_undegraded":"2024-09-13T15:14:02.795677+0000","last_fullsized":"2024-09-13T15:14:02.795677+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.098941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.098941+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:19.670702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":840,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296743+0000","last_change":"2024-09-13T15:08:31.195499+0000","last_active":"2024-09-13T15:13:57.296743+0000","last_peered":"2024-09-13T15:13:57.296743+0000","last_clean":"2024-09-13T15:13:57.296743+0000","last_became_active":"2024-09-13T15:08:31.194928+0000","last_became_peered":"2024-09-13T15:08:31.194928+0000","last_unstale":"2024-09-13T15:13:57.296743+0000","last_undegraded":"2024-09-13T15:13:57.296743+0000","last_fullsized":"2024-09-13T15:13:57.296743+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.112396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":75,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295668+0000","last_change":"2024-09-13T15:08:31.185658+0000","last_active":"2024-09-13T15:13:57.295668+0000","last_peered":"2024-09-13T15:13:57.295668+0000","last_clean":"2024-09-13T15:13:57.295668+0000","last_became_active":"2024-09-13T15:08:31.185202+0000","last_became_peered":"2024-09-13T15:08:31.185202+0000","last_unstale":"2024-09-13T15:13:57.295668+0000","last_undegraded":"2024-09-13T15:13:57.295668+0000","last_fullsized":"2024-09-13T15:13:57.295668+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:04.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"89'149","reported_seq":839,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296210+0000","last_change":"2024-09-13T15:13:11.154190+0000","last_active":"2024-09-13T15:13:57.296210+0000","last_peered":"2024-09-13T15:13:57.296210+0000","last_clean":"2024-09-13T15:13:57.296210+0000","last_became_active":"2024-09-13T15:12:50.159857+0000","last_became_peered":"2024-09-13T15:12:50.159857+0000","last_unstale":"2024-09-13T15:13:57.296210+0000","last_undegraded":"2024-09-13T15:13:57.296210+0000","last_fullsized":"2024-09-13T15:13:57.296210+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.153852+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.153852+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:54.235242+0000","scrub_duration":0.0043281400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":345,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196231+0000","last_change":"2024-09-13T15:08:31.199682+0000","last_active":"2024-09-13T15:14:03.196231+0000","last_peered":"2024-09-13T15:14:03.196231+0000","last_clean":"2024-09-13T15:14:03.196231+0000","last_became_active":"2024-09-13T15:08:31.199110+0000","last_became_peered":"2024-09-13T15:08:31.199110+0000","last_unstale":"2024-09-13T15:14:03.196231+0000","last_undegraded":"2024-09-13T15:14:03.196231+0000","last_fullsized":"2024-09-13T15:14:03.196231+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:30.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":447,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295252+0000","last_change":"2024-09-13T15:04:05.021292+0000","last_active":"2024-09-13T15:13:57.295252+0000","last_peered":"2024-09-13T15:13:57.295252+0000","last_clean":"2024-09-13T15:13:57.295252+0000","last_became_active":"2024-09-13T15:04:05.011146+0000","last_became_peered":"2024-09-13T15:04:05.011146+0000","last_unstale":"2024-09-13T15:13:57.295252+0000","last_undegraded":"2024-09-13T15:13:57.295252+0000","last_fullsized":"2024-09-13T15:13:57.295252+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:00.137694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"89'149","reported_seq":840,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296487+0000","last_change":"2024-09-13T15:13:15.494950+0000","last_active":"2024-09-13T15:13:57.296487+0000","last_peered":"2024-09-13T15:13:57.296487+0000","last_clean":"2024-09-13T15:13:57.296487+0000","last_became_active":"2024-09-13T15:12:50.163288+0000","last_became_peered":"2024-09-13T15:12:50.163288+0000","last_unstale":"2024-09-13T15:13:57.296487+0000","last_undegraded":"2024-09-13T15:13:57.296487+0000","last_fullsized":"2024-09-13T15:13:57.296487+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:15.494798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:15.494798+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:34.790443+0000","scrub_duration":0.0082230789999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296864+0000","last_change":"2024-09-13T15:08:31.351504+0000","last_active":"2024-09-13T15:13:57.296864+0000","last_peered":"2024-09-13T15:13:57.296864+0000","last_clean":"2024-09-13T15:13:57.296864+0000","last_became_active":"2024-09-13T15:08:31.320950+0000","last_became_peered":"2024-09-13T15:08:31.320950+0000","last_unstale":"2024-09-13T15:13:57.296864+0000","last_undegraded":"2024-09-13T15:13:57.296864+0000","last_fullsized":"2024-09-13T15:13:57.296864+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:38.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":404,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295807+0000","last_change":"2024-09-13T15:08:46.078296+0000","last_active":"2024-09-13T15:13:57.295807+0000","last_peered":"2024-09-13T15:13:57.295807+0000","last_clean":"2024-09-13T15:13:57.295807+0000","last_became_active":"2024-09-13T15:08:46.078037+0000","last_became_peered":"2024-09-13T15:08:46.078037+0000","last_unstale":"2024-09-13T15:13:57.295807+0000","last_undegraded":"2024-09-13T15:13:57.295807+0000","last_fullsized":"2024-09-13T15:13:57.295807+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:49.959361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"89'149","reported_seq":932,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195672+0000","last_change":"2024-09-13T15:13:02.045213+0000","last_active":"2024-09-13T15:14:03.195672+0000","last_peered":"2024-09-13T15:14:03.195672+0000","last_clean":"2024-09-13T15:14:03.195672+0000","last_became_active":"2024-09-13T15:12:50.170281+0000","last_became_peered":"2024-09-13T15:12:50.170281+0000","last_unstale":"2024-09-13T15:14:03.195672+0000","last_undegraded":"2024-09-13T15:14:03.195672+0000","last_fullsized":"2024-09-13T15:14:03.195672+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:02.045141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:02.045141+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:03.286894+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296377+0000","last_change":"2024-09-13T15:08:31.201939+0000","last_active":"2024-09-13T15:13:57.296377+0000","last_peered":"2024-09-13T15:13:57.296377+0000","last_clean":"2024-09-13T15:13:57.296377+0000","last_became_active":"2024-09-13T15:08:31.191089+0000","last_became_peered":"2024-09-13T15:08:31.191089+0000","last_unstale":"2024-09-13T15:13:57.296377+0000","last_undegraded":"2024-09-13T15:13:57.296377+0000","last_fullsized":"2024-09-13T15:13:57.296377+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:06.249945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":568,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271641+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:13:57.271641+0000","last_peered":"2024-09-13T15:13:57.271641+0000","last_clean":"2024-09-13T15:13:57.271641+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:13:57.271641+0000","last_undegraded":"2024-09-13T15:13:57.271641+0000","last_fullsized":"2024-09-13T15:13:57.271641+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"89'149","reported_seq":899,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296522+0000","last_change":"2024-09-13T15:13:06.128357+0000","last_active":"2024-09-13T15:13:57.296522+0000","last_peered":"2024-09-13T15:13:57.296522+0000","last_clean":"2024-09-13T15:13:57.296522+0000","last_became_active":"2024-09-13T15:12:50.160188+0000","last_became_peered":"2024-09-13T15:12:50.160188+0000","last_unstale":"2024-09-13T15:13:57.296522+0000","last_undegraded":"2024-09-13T15:13:57.296522+0000","last_fullsized":"2024-09-13T15:13:57.296522+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:06.128281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:06.128281+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:11.473186+0000","scrub_duration":0.0011969070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296728+0000","last_change":"2024-09-13T15:08:31.199736+0000","last_active":"2024-09-13T15:13:57.296728+0000","last_peered":"2024-09-13T15:13:57.296728+0000","last_clean":"2024-09-13T15:13:57.296728+0000","last_became_active":"2024-09-13T15:08:31.186018+0000","last_became_peered":"2024-09-13T15:08:31.186018+0000","last_unstale":"2024-09-13T15:13:57.296728+0000","last_undegraded":"2024-09-13T15:13:57.296728+0000","last_fullsized":"2024-09-13T15:13:57.296728+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:26.855187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":580,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795646+0000","last_change":"2024-09-13T15:08:31.193478+0000","last_active":"2024-09-13T15:14:02.795646+0000","last_peered":"2024-09-13T15:14:02.795646+0000","last_clean":"2024-09-13T15:14:02.795646+0000","last_became_active":"2024-09-13T15:08:31.193038+0000","last_became_peered":"2024-09-13T15:08:31.193038+0000","last_unstale":"2024-09-13T15:14:02.795646+0000","last_undegraded":"2024-09-13T15:14:02.795646+0000","last_fullsized":"2024-09-13T15:14:02.795646+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:05.600387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"89'149","reported_seq":908,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272101+0000","last_change":"2024-09-13T15:13:01.670578+0000","last_active":"2024-09-13T15:13:57.272101+0000","last_peered":"2024-09-13T15:13:57.272101+0000","last_clean":"2024-09-13T15:13:57.272101+0000","last_became_active":"2024-09-13T15:12:50.166107+0000","last_became_peered":"2024-09-13T15:12:50.166107+0000","last_unstale":"2024-09-13T15:13:57.272101+0000","last_undegraded":"2024-09-13T15:13:57.272101+0000","last_fullsized":"2024-09-13T15:13:57.272101+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.670400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.670400+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:19.151283+0000","scrub_duration":0.0070424579999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":375,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295748+0000","last_change":"2024-09-13T15:08:46.073352+0000","last_active":"2024-09-13T15:13:57.295748+0000","last_peered":"2024-09-13T15:13:57.295748+0000","last_clean":"2024-09-13T15:13:57.295748+0000","last_became_active":"2024-09-13T15:08:46.073048+0000","last_became_peered":"2024-09-13T15:08:46.073048+0000","last_unstale":"2024-09-13T15:13:57.295748+0000","last_undegraded":"2024-09-13T15:13:57.295748+0000","last_fullsized":"2024-09-13T15:13:57.295748+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:57.277685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":749,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296201+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:13:57.296201+0000","last_peered":"2024-09-13T15:13:57.296201+0000","last_clean":"2024-09-13T15:13:57.296201+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:13:57.296201+0000","last_undegraded":"2024-09-13T15:13:57.296201+0000","last_fullsized":"2024-09-13T15:13:57.296201+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"89'149","reported_seq":880,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295542+0000","last_change":"2024-09-13T15:10:26.504570+0000","last_active":"2024-09-13T15:13:57.295542+0000","last_peered":"2024-09-13T15:13:57.295542+0000","last_clean":"2024-09-13T15:13:57.295542+0000","last_became_active":"2024-09-13T15:10:19.782654+0000","last_became_peered":"2024-09-13T15:10:19.782654+0000","last_unstale":"2024-09-13T15:13:57.295542+0000","last_undegraded":"2024-09-13T15:13:57.295542+0000","last_fullsized":"2024-09-13T15:13:57.295542+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:10:26.504369+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:57.248804+0000","scrub_duration":0.013934767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":879,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195713+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:14:03.195713+0000","last_peered":"2024-09-13T15:14:03.195713+0000","last_clean":"2024-09-13T15:14:03.195713+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:14:03.195713+0000","last_undegraded":"2024-09-13T15:14:03.195713+0000","last_fullsized":"2024-09-13T15:14:03.195713+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":510,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271822+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:13:57.271822+0000","last_peered":"2024-09-13T15:13:57.271822+0000","last_clean":"2024-09-13T15:13:57.271822+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:13:57.271822+0000","last_undegraded":"2024-09-13T15:13:57.271822+0000","last_fullsized":"2024-09-13T15:13:57.271822+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"89'149","reported_seq":858,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295154+0000","last_change":"2024-09-13T15:10:20.069094+0000","last_active":"2024-09-13T15:13:57.295154+0000","last_peered":"2024-09-13T15:13:57.295154+0000","last_clean":"2024-09-13T15:13:57.295154+0000","last_became_active":"2024-09-13T15:10:19.782890+0000","last_became_peered":"2024-09-13T15:10:19.782890+0000","last_unstale":"2024-09-13T15:13:57.295154+0000","last_undegraded":"2024-09-13T15:13:57.295154+0000","last_fullsized":"2024-09-13T15:13:57.295154+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:20.068973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:20.068973+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:30.570922+0000","scrub_duration":0.011313818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":308,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271719+0000","last_change":"2024-09-13T15:04:05.013503+0000","last_active":"2024-09-13T15:13:57.271719+0000","last_peered":"2024-09-13T15:13:57.271719+0000","last_clean":"2024-09-13T15:13:57.271719+0000","last_became_active":"2024-09-13T15:04:05.013204+0000","last_became_peered":"2024-09-13T15:04:05.013204+0000","last_unstale":"2024-09-13T15:13:57.271719+0000","last_undegraded":"2024-09-13T15:13:57.271719+0000","last_fullsized":"2024-09-13T15:13:57.271719+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:17.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":504,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272111+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:13:57.272111+0000","last_peered":"2024-09-13T15:13:57.272111+0000","last_clean":"2024-09-13T15:13:57.272111+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:13:57.272111+0000","last_undegraded":"2024-09-13T15:13:57.272111+0000","last_fullsized":"2024-09-13T15:13:57.272111+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"89'149","reported_seq":925,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795868+0000","last_change":"2024-09-13T15:13:02.125234+0000","last_active":"2024-09-13T15:14:02.795868+0000","last_peered":"2024-09-13T15:14:02.795868+0000","last_clean":"2024-09-13T15:14:02.795868+0000","last_became_active":"2024-09-13T15:12:50.176602+0000","last_became_peered":"2024-09-13T15:12:50.176602+0000","last_unstale":"2024-09-13T15:14:02.795868+0000","last_undegraded":"2024-09-13T15:14:02.795868+0000","last_fullsized":"2024-09-13T15:14:02.795868+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:02.125166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:02.125166+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.682602+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":529,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271697+0000","last_change":"2024-09-13T15:08:31.184973+0000","last_active":"2024-09-13T15:13:57.271697+0000","last_peered":"2024-09-13T15:13:57.271697+0000","last_clean":"2024-09-13T15:13:57.271697+0000","last_became_active":"2024-09-13T15:08:31.184600+0000","last_became_peered":"2024-09-13T15:08:31.184600+0000","last_unstale":"2024-09-13T15:13:57.271697+0000","last_undegraded":"2024-09-13T15:13:57.271697+0000","last_fullsized":"2024-09-13T15:13:57.271697+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:09.452895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":537,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272207+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:13:57.272207+0000","last_peered":"2024-09-13T15:13:57.272207+0000","last_clean":"2024-09-13T15:13:57.272207+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:13:57.272207+0000","last_undegraded":"2024-09-13T15:13:57.272207+0000","last_fullsized":"2024-09-13T15:13:57.272207+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"89'149","reported_seq":858,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296131+0000","last_change":"2024-09-13T15:10:22.099331+0000","last_active":"2024-09-13T15:13:57.296131+0000","last_peered":"2024-09-13T15:13:57.296131+0000","last_clean":"2024-09-13T15:13:57.296131+0000","last_became_active":"2024-09-13T15:10:19.785103+0000","last_became_peered":"2024-09-13T15:10:19.785103+0000","last_unstale":"2024-09-13T15:13:57.296131+0000","last_undegraded":"2024-09-13T15:13:57.296131+0000","last_fullsized":"2024-09-13T15:13:57.296131+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:22.099204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:22.099204+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:34.804057+0000","scrub_duration":0.0069277289999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":345,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195427+0000","last_change":"2024-09-13T15:08:31.198602+0000","last_active":"2024-09-13T15:14:03.195427+0000","last_peered":"2024-09-13T15:14:03.195427+0000","last_clean":"2024-09-13T15:14:03.195427+0000","last_became_active":"2024-09-13T15:08:31.198289+0000","last_became_peered":"2024-09-13T15:08:31.198289+0000","last_unstale":"2024-09-13T15:14:03.195427+0000","last_undegraded":"2024-09-13T15:14:03.195427+0000","last_fullsized":"2024-09-13T15:14:03.195427+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:36.803833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":624,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195962+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:14:03.195962+0000","last_peered":"2024-09-13T15:14:03.195962+0000","last_clean":"2024-09-13T15:14:03.195962+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:14:03.195962+0000","last_undegraded":"2024-09-13T15:14:03.195962+0000","last_fullsized":"2024-09-13T15:14:03.195962+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":885,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295516+0000","last_change":"2024-09-13T15:13:11.597792+0000","last_active":"2024-09-13T15:13:57.295516+0000","last_peered":"2024-09-13T15:13:57.295516+0000","last_clean":"2024-09-13T15:13:57.295516+0000","last_became_active":"2024-09-13T15:12:38.938735+0000","last_became_peered":"2024-09-13T15:12:38.938735+0000","last_unstale":"2024-09-13T15:13:57.295516+0000","last_undegraded":"2024-09-13T15:13:57.295516+0000","last_fullsized":"2024-09-13T15:13:57.295516+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.597623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.597623+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:14.909270+0000","scrub_duration":0.012675130999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1171,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296353+0000","last_change":"2024-09-13T15:08:57.728783+0000","last_active":"2024-09-13T15:13:57.296353+0000","last_peered":"2024-09-13T15:13:57.296353+0000","last_clean":"2024-09-13T15:13:57.296353+0000","last_became_active":"2024-09-13T15:08:47.077688+0000","last_became_peered":"2024-09-13T15:08:47.077688+0000","last_unstale":"2024-09-13T15:13:57.296353+0000","last_undegraded":"2024-09-13T15:13:57.296353+0000","last_fullsized":"2024-09-13T15:13:57.296353+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:52.710051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":841,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195587+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:14:03.195587+0000","last_peered":"2024-09-13T15:14:03.195587+0000","last_clean":"2024-09-13T15:14:03.195587+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:14:03.195587+0000","last_undegraded":"2024-09-13T15:14:03.195587+0000","last_fullsized":"2024-09-13T15:14:03.195587+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":356,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271975+0000","last_change":"2024-09-13T15:08:31.187510+0000","last_active":"2024-09-13T15:13:57.271975+0000","last_peered":"2024-09-13T15:13:57.271975+0000","last_clean":"2024-09-13T15:13:57.271975+0000","last_became_active":"2024-09-13T15:08:31.187108+0000","last_became_peered":"2024-09-13T15:08:31.187108+0000","last_unstale":"2024-09-13T15:13:57.271975+0000","last_undegraded":"2024-09-13T15:13:57.271975+0000","last_fullsized":"2024-09-13T15:13:57.271975+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:25.096543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"89'149","reported_seq":922,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795744+0000","last_change":"2024-09-13T15:13:04.107856+0000","last_active":"2024-09-13T15:14:02.795744+0000","last_peered":"2024-09-13T15:14:02.795744+0000","last_clean":"2024-09-13T15:14:02.795744+0000","last_became_active":"2024-09-13T15:12:38.929136+0000","last_became_peered":"2024-09-13T15:12:38.929136+0000","last_unstale":"2024-09-13T15:14:02.795744+0000","last_undegraded":"2024-09-13T15:14:02.795744+0000","last_fullsized":"2024-09-13T15:14:02.795744+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.107315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.107315+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:20.960972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":93,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295305+0000","last_change":"2024-09-13T15:08:31.218115+0000","last_active":"2024-09-13T15:13:57.295305+0000","last_peered":"2024-09-13T15:13:57.295305+0000","last_clean":"2024-09-13T15:13:57.295305+0000","last_became_active":"2024-09-13T15:08:31.196118+0000","last_became_peered":"2024-09-13T15:08:31.196118+0000","last_unstale":"2024-09-13T15:13:57.295305+0000","last_undegraded":"2024-09-13T15:13:57.295305+0000","last_fullsized":"2024-09-13T15:13:57.295305+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:08.721976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":823,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295890+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:13:57.295890+0000","last_peered":"2024-09-13T15:13:57.295890+0000","last_clean":"2024-09-13T15:13:57.295890+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:13:57.295890+0000","last_undegraded":"2024-09-13T15:13:57.295890+0000","last_fullsized":"2024-09-13T15:13:57.295890+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":818,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296665+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:13:57.296665+0000","last_peered":"2024-09-13T15:13:57.296665+0000","last_clean":"2024-09-13T15:13:57.296665+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:13:57.296665+0000","last_undegraded":"2024-09-13T15:13:57.296665+0000","last_fullsized":"2024-09-13T15:13:57.296665+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"89'149","reported_seq":901,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295245+0000","last_change":"2024-09-13T15:13:05.496011+0000","last_active":"2024-09-13T15:13:57.295245+0000","last_peered":"2024-09-13T15:13:57.295245+0000","last_clean":"2024-09-13T15:13:57.295245+0000","last_became_active":"2024-09-13T15:12:38.946687+0000","last_became_peered":"2024-09-13T15:12:38.946687+0000","last_unstale":"2024-09-13T15:13:57.295245+0000","last_undegraded":"2024-09-13T15:13:57.295245+0000","last_fullsized":"2024-09-13T15:13:57.295245+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:05.495826+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:05.495826+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:01.133250+0000","scrub_duration":0.011811428000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1261,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196006+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:14:03.196006+0000","last_peered":"2024-09-13T15:14:03.196006+0000","last_clean":"2024-09-13T15:14:03.196006+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:14:03.196006+0000","last_undegraded":"2024-09-13T15:14:03.196006+0000","last_fullsized":"2024-09-13T15:14:03.196006+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":890,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795719+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:14:02.795719+0000","last_peered":"2024-09-13T15:14:02.795719+0000","last_clean":"2024-09-13T15:14:02.795719+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:14:02.795719+0000","last_undegraded":"2024-09-13T15:14:02.795719+0000","last_fullsized":"2024-09-13T15:14:02.795719+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":628,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.796071+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:14:02.796071+0000","last_peered":"2024-09-13T15:14:02.796071+0000","last_clean":"2024-09-13T15:14:02.796071+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:14:02.796071+0000","last_undegraded":"2024-09-13T15:14:02.796071+0000","last_fullsized":"2024-09-13T15:14:02.796071+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"89'149","reported_seq":879,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295914+0000","last_change":"2024-09-13T15:13:03.104893+0000","last_active":"2024-09-13T15:13:57.295914+0000","last_peered":"2024-09-13T15:13:57.295914+0000","last_clean":"2024-09-13T15:13:57.295914+0000","last_became_active":"2024-09-13T15:12:38.933486+0000","last_became_peered":"2024-09-13T15:12:38.933486+0000","last_unstale":"2024-09-13T15:13:57.295914+0000","last_undegraded":"2024-09-13T15:13:57.295914+0000","last_fullsized":"2024-09-13T15:13:57.295914+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.104648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.104648+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:41.623486+0000","scrub_duration":0.01681821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":425,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296313+0000","last_change":"2024-09-13T15:08:46.074249+0000","last_active":"2024-09-13T15:13:57.296313+0000","last_peered":"2024-09-13T15:13:57.296313+0000","last_clean":"2024-09-13T15:13:57.296313+0000","last_became_active":"2024-09-13T15:08:46.073842+0000","last_became_peered":"2024-09-13T15:08:46.073842+0000","last_unstale":"2024-09-13T15:13:57.296313+0000","last_undegraded":"2024-09-13T15:13:57.296313+0000","last_fullsized":"2024-09-13T15:13:57.296313+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:44.854067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":529,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272094+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:13:57.272094+0000","last_peered":"2024-09-13T15:13:57.272094+0000","last_clean":"2024-09-13T15:13:57.272094+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:13:57.272094+0000","last_undegraded":"2024-09-13T15:13:57.272094+0000","last_fullsized":"2024-09-13T15:13:57.272094+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":342,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295444+0000","last_change":"2024-09-13T15:08:46.075360+0000","last_active":"2024-09-13T15:13:57.295444+0000","last_peered":"2024-09-13T15:13:57.295444+0000","last_clean":"2024-09-13T15:13:57.295444+0000","last_became_active":"2024-09-13T15:08:46.075046+0000","last_became_peered":"2024-09-13T15:08:46.075046+0000","last_unstale":"2024-09-13T15:13:57.295444+0000","last_undegraded":"2024-09-13T15:13:57.295444+0000","last_fullsized":"2024-09-13T15:13:57.295444+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.044150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"89'149","reported_seq":923,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196225+0000","last_change":"2024-09-13T15:13:04.116692+0000","last_active":"2024-09-13T15:14:03.196225+0000","last_peered":"2024-09-13T15:14:03.196225+0000","last_clean":"2024-09-13T15:14:03.196225+0000","last_became_active":"2024-09-13T15:12:38.940054+0000","last_became_peered":"2024-09-13T15:12:38.940054+0000","last_unstale":"2024-09-13T15:14:03.196225+0000","last_undegraded":"2024-09-13T15:14:03.196225+0000","last_fullsized":"2024-09-13T15:14:03.196225+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.116616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.116616+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:25.864759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":375,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295836+0000","last_change":"2024-09-13T15:08:46.079918+0000","last_active":"2024-09-13T15:13:57.295836+0000","last_peered":"2024-09-13T15:13:57.295836+0000","last_clean":"2024-09-13T15:13:57.295836+0000","last_became_active":"2024-09-13T15:08:46.075051+0000","last_became_peered":"2024-09-13T15:08:46.075051+0000","last_unstale":"2024-09-13T15:13:57.295836+0000","last_undegraded":"2024-09-13T15:13:57.295836+0000","last_fullsized":"2024-09-13T15:13:57.295836+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:54.485416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1542,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795944+0000","last_change":"2024-09-13T15:08:59.341688+0000","last_active":"2024-09-13T15:14:02.795944+0000","last_peered":"2024-09-13T15:14:02.795944+0000","last_clean":"2024-09-13T15:14:02.795944+0000","last_became_active":"2024-09-13T15:08:59.340925+0000","last_became_peered":"2024-09-13T15:08:59.340925+0000","last_unstale":"2024-09-13T15:14:02.795944+0000","last_undegraded":"2024-09-13T15:14:02.795944+0000","last_fullsized":"2024-09-13T15:14:02.795944+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:22.966119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":468,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296600+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:13:57.296600+0000","last_peered":"2024-09-13T15:13:57.296600+0000","last_clean":"2024-09-13T15:13:57.296600+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:13:57.296600+0000","last_undegraded":"2024-09-13T15:13:57.296600+0000","last_fullsized":"2024-09-13T15:13:57.296600+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"89'149","reported_seq":925,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296612+0000","last_change":"2024-09-13T15:13:03.228430+0000","last_active":"2024-09-13T15:13:57.296612+0000","last_peered":"2024-09-13T15:13:57.296612+0000","last_clean":"2024-09-13T15:13:57.296612+0000","last_became_active":"2024-09-13T15:12:38.937552+0000","last_became_peered":"2024-09-13T15:12:38.937552+0000","last_unstale":"2024-09-13T15:13:57.296612+0000","last_undegraded":"2024-09-13T15:13:57.296612+0000","last_fullsized":"2024-09-13T15:13:57.296612+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.228290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.228290+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:30.456205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":882,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795753+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:14:02.795753+0000","last_peered":"2024-09-13T15:14:02.795753+0000","last_clean":"2024-09-13T15:14:02.795753+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:14:02.795753+0000","last_undegraded":"2024-09-13T15:14:02.795753+0000","last_fullsized":"2024-09-13T15:14:02.795753+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":93,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296541+0000","last_change":"2024-09-13T15:08:31.209459+0000","last_active":"2024-09-13T15:13:57.296541+0000","last_peered":"2024-09-13T15:13:57.296541+0000","last_clean":"2024-09-13T15:13:57.296541+0000","last_became_active":"2024-09-13T15:08:31.188811+0000","last_became_peered":"2024-09-13T15:08:31.188811+0000","last_unstale":"2024-09-13T15:13:57.296541+0000","last_undegraded":"2024-09-13T15:13:57.296541+0000","last_fullsized":"2024-09-13T15:13:57.296541+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.667223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":827,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296334+0000","last_change":"2024-09-13T15:08:31.197894+0000","last_active":"2024-09-13T15:13:57.296334+0000","last_peered":"2024-09-13T15:13:57.296334+0000","last_clean":"2024-09-13T15:13:57.296334+0000","last_became_active":"2024-09-13T15:08:31.197516+0000","last_became_peered":"2024-09-13T15:08:31.197516+0000","last_unstale":"2024-09-13T15:13:57.296334+0000","last_undegraded":"2024-09-13T15:13:57.296334+0000","last_fullsized":"2024-09-13T15:13:57.296334+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:42.650595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"89'149","reported_seq":879,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272105+0000","last_change":"2024-09-13T15:13:00.711640+0000","last_active":"2024-09-13T15:13:57.272105+0000","last_peered":"2024-09-13T15:13:57.272105+0000","last_clean":"2024-09-13T15:13:57.272105+0000","last_became_active":"2024-09-13T15:12:38.935375+0000","last_became_peered":"2024-09-13T15:12:38.935375+0000","last_unstale":"2024-09-13T15:13:57.272105+0000","last_undegraded":"2024-09-13T15:13:57.272105+0000","last_fullsized":"2024-09-13T15:13:57.272105+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.711407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.711407+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:41.333318+0000","scrub_duration":0.0091286909999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":934,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196369+0000","last_change":"2024-09-13T15:08:31.202795+0000","last_active":"2024-09-13T15:14:03.196369+0000","last_peered":"2024-09-13T15:14:03.196369+0000","last_clean":"2024-09-13T15:14:03.196369+0000","last_became_active":"2024-09-13T15:08:31.199963+0000","last_became_peered":"2024-09-13T15:08:31.199963+0000","last_unstale":"2024-09-13T15:14:03.196369+0000","last_undegraded":"2024-09-13T15:14:03.196369+0000","last_fullsized":"2024-09-13T15:14:03.196369+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:08.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1368,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.796117+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:14:02.796117+0000","last_peered":"2024-09-13T15:14:02.796117+0000","last_clean":"2024-09-13T15:14:02.796117+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:14:02.796117+0000","last_undegraded":"2024-09-13T15:14:02.796117+0000","last_fullsized":"2024-09-13T15:14:02.796117+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":639,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195541+0000","last_change":"2024-09-13T15:08:31.198729+0000","last_active":"2024-09-13T15:14:03.195541+0000","last_peered":"2024-09-13T15:14:03.195541+0000","last_clean":"2024-09-13T15:14:03.195541+0000","last_became_active":"2024-09-13T15:08:31.198423+0000","last_became_peered":"2024-09-13T15:08:31.198423+0000","last_unstale":"2024-09-13T15:14:03.195541+0000","last_undegraded":"2024-09-13T15:14:03.195541+0000","last_fullsized":"2024-09-13T15:14:03.195541+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:58.362919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"89'149","reported_seq":920,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195629+0000","last_change":"2024-09-13T15:13:03.079534+0000","last_active":"2024-09-13T15:14:03.195629+0000","last_peered":"2024-09-13T15:14:03.195629+0000","last_clean":"2024-09-13T15:14:03.195629+0000","last_became_active":"2024-09-13T15:12:38.941094+0000","last_became_peered":"2024-09-13T15:12:38.941094+0000","last_unstale":"2024-09-13T15:14:03.195629+0000","last_undegraded":"2024-09-13T15:14:03.195629+0000","last_fullsized":"2024-09-13T15:14:03.195629+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.079258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.079258+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:10.140799+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":754,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296749+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:13:57.296749+0000","last_peered":"2024-09-13T15:13:57.296749+0000","last_clean":"2024-09-13T15:13:57.296749+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:13:57.296749+0000","last_undegraded":"2024-09-13T15:13:57.296749+0000","last_fullsized":"2024-09-13T15:13:57.296749+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":932,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296866+0000","last_change":"2024-09-13T15:08:59.333844+0000","last_active":"2024-09-13T15:13:57.296866+0000","last_peered":"2024-09-13T15:13:57.296866+0000","last_clean":"2024-09-13T15:13:57.296866+0000","last_became_active":"2024-09-13T15:08:59.332369+0000","last_became_peered":"2024-09-13T15:08:59.332369+0000","last_unstale":"2024-09-13T15:13:57.296866+0000","last_undegraded":"2024-09-13T15:13:57.296866+0000","last_fullsized":"2024-09-13T15:13:57.296866+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:36.490753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":526,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.286163+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:14:02.286163+0000","last_peered":"2024-09-13T15:14:02.286163+0000","last_clean":"2024-09-13T15:14:02.286163+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:14:02.286163+0000","last_undegraded":"2024-09-13T15:14:02.286163+0000","last_fullsized":"2024-09-13T15:14:02.286163+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":509,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272000+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:13:57.272000+0000","last_peered":"2024-09-13T15:13:57.272000+0000","last_clean":"2024-09-13T15:13:57.272000+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:13:57.272000+0000","last_undegraded":"2024-09-13T15:13:57.272000+0000","last_fullsized":"2024-09-13T15:13:57.272000+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":641,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272002+0000","last_change":"2024-09-13T15:08:31.193477+0000","last_active":"2024-09-13T15:13:57.272002+0000","last_peered":"2024-09-13T15:13:57.272002+0000","last_clean":"2024-09-13T15:13:57.272002+0000","last_became_active":"2024-09-13T15:08:31.181861+0000","last_became_peered":"2024-09-13T15:08:31.181861+0000","last_unstale":"2024-09-13T15:13:57.272002+0000","last_undegraded":"2024-09-13T15:13:57.272002+0000","last_fullsized":"2024-09-13T15:13:57.272002+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:02.424164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":33292,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":893,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795855+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:14:02.795855+0000","last_peered":"2024-09-13T15:14:02.795855+0000","last_clean":"2024-09-13T15:14:02.795855+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:14:02.795855+0000","last_undegraded":"2024-09-13T15:14:02.795855+0000","last_fullsized":"2024-09-13T15:14:02.795855+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":81,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296587+0000","last_change":"2024-09-13T15:08:31.204593+0000","last_active":"2024-09-13T15:13:57.296587+0000","last_peered":"2024-09-13T15:13:57.296587+0000","last_clean":"2024-09-13T15:13:57.296587+0000","last_became_active":"2024-09-13T15:08:31.198883+0000","last_became_peered":"2024-09-13T15:08:31.198883+0000","last_unstale":"2024-09-13T15:13:57.296587+0000","last_undegraded":"2024-09-13T15:13:57.296587+0000","last_fullsized":"2024-09-13T15:13:57.296587+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:08.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"89'149","reported_seq":923,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195697+0000","last_change":"2024-09-13T15:13:00.998330+0000","last_active":"2024-09-13T15:14:03.195697+0000","last_peered":"2024-09-13T15:14:03.195697+0000","last_clean":"2024-09-13T15:14:03.195697+0000","last_became_active":"2024-09-13T15:12:50.190273+0000","last_became_peered":"2024-09-13T15:12:50.190273+0000","last_unstale":"2024-09-13T15:14:03.195697+0000","last_undegraded":"2024-09-13T15:14:03.195697+0000","last_fullsized":"2024-09-13T15:14:03.195697+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.998266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.998266+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:06.347067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":398,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296130+0000","last_change":"2024-09-13T15:08:31.188790+0000","last_active":"2024-09-13T15:13:57.296130+0000","last_peered":"2024-09-13T15:13:57.296130+0000","last_clean":"2024-09-13T15:13:57.296130+0000","last_became_active":"2024-09-13T15:08:31.188441+0000","last_became_peered":"2024-09-13T15:08:31.188441+0000","last_unstale":"2024-09-13T15:13:57.296130+0000","last_undegraded":"2024-09-13T15:13:57.296130+0000","last_fullsized":"2024-09-13T15:13:57.296130+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:52.934163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":881,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195773+0000","last_change":"2024-09-13T15:08:31.197112+0000","last_active":"2024-09-13T15:14:03.195773+0000","last_peered":"2024-09-13T15:14:03.195773+0000","last_clean":"2024-09-13T15:14:03.195773+0000","last_became_active":"2024-09-13T15:08:31.196687+0000","last_became_peered":"2024-09-13T15:08:31.196687+0000","last_unstale":"2024-09-13T15:14:03.195773+0000","last_undegraded":"2024-09-13T15:14:03.195773+0000","last_fullsized":"2024-09-13T15:14:03.195773+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:47.532912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":845,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296466+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:13:57.296466+0000","last_peered":"2024-09-13T15:13:57.296466+0000","last_clean":"2024-09-13T15:13:57.296466+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:13:57.296466+0000","last_undegraded":"2024-09-13T15:13:57.296466+0000","last_fullsized":"2024-09-13T15:13:57.296466+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":911,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195576+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:14:03.195576+0000","last_peered":"2024-09-13T15:14:03.195576+0000","last_clean":"2024-09-13T15:14:03.195576+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:14:03.195576+0000","last_undegraded":"2024-09-13T15:14:03.195576+0000","last_fullsized":"2024-09-13T15:14:03.195576+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":304,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296558+0000","last_change":"2024-09-13T15:08:46.075112+0000","last_active":"2024-09-13T15:13:57.296558+0000","last_peered":"2024-09-13T15:13:57.296558+0000","last_clean":"2024-09-13T15:13:57.296558+0000","last_became_active":"2024-09-13T15:08:46.074818+0000","last_became_peered":"2024-09-13T15:08:46.074818+0000","last_unstale":"2024-09-13T15:13:57.296558+0000","last_undegraded":"2024-09-13T15:13:57.296558+0000","last_fullsized":"2024-09-13T15:13:57.296558+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:56.288263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":875,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195781+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:14:03.195781+0000","last_peered":"2024-09-13T15:14:03.195781+0000","last_clean":"2024-09-13T15:14:03.195781+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:14:03.195781+0000","last_undegraded":"2024-09-13T15:14:03.195781+0000","last_fullsized":"2024-09-13T15:14:03.195781+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":843,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296681+0000","last_change":"2024-09-13T15:08:31.195215+0000","last_active":"2024-09-13T15:13:57.296681+0000","last_peered":"2024-09-13T15:13:57.296681+0000","last_clean":"2024-09-13T15:13:57.296681+0000","last_became_active":"2024-09-13T15:08:31.194526+0000","last_became_peered":"2024-09-13T15:08:31.194526+0000","last_unstale":"2024-09-13T15:13:57.296681+0000","last_undegraded":"2024-09-13T15:13:57.296681+0000","last_fullsized":"2024-09-13T15:13:57.296681+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:56.358764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":847,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795606+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:14:02.795606+0000","last_peered":"2024-09-13T15:14:02.795606+0000","last_clean":"2024-09-13T15:14:02.795606+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:14:02.795606+0000","last_undegraded":"2024-09-13T15:14:02.795606+0000","last_fullsized":"2024-09-13T15:14:02.795606+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":712,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195977+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:14:03.195977+0000","last_peered":"2024-09-13T15:14:03.195977+0000","last_clean":"2024-09-13T15:14:03.195977+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:14:03.195977+0000","last_undegraded":"2024-09-13T15:14:03.195977+0000","last_fullsized":"2024-09-13T15:14:03.195977+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":504,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271833+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:13:57.271833+0000","last_peered":"2024-09-13T15:13:57.271833+0000","last_clean":"2024-09-13T15:13:57.271833+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:13:57.271833+0000","last_undegraded":"2024-09-13T15:13:57.271833+0000","last_fullsized":"2024-09-13T15:13:57.271833+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":463,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195981+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:14:03.195981+0000","last_peered":"2024-09-13T15:14:03.195981+0000","last_clean":"2024-09-13T15:14:03.195981+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:14:03.195981+0000","last_undegraded":"2024-09-13T15:14:03.195981+0000","last_fullsized":"2024-09-13T15:14:03.195981+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":830,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296415+0000","last_change":"2024-09-13T15:08:31.194912+0000","last_active":"2024-09-13T15:13:57.296415+0000","last_peered":"2024-09-13T15:13:57.296415+0000","last_clean":"2024-09-13T15:13:57.296415+0000","last_became_active":"2024-09-13T15:08:31.194553+0000","last_became_peered":"2024-09-13T15:08:31.194553+0000","last_unstale":"2024-09-13T15:13:57.296415+0000","last_undegraded":"2024-09-13T15:13:57.296415+0000","last_fullsized":"2024-09-13T15:13:57.296415+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:17.457690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":514,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271953+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:13:57.271953+0000","last_peered":"2024-09-13T15:13:57.271953+0000","last_clean":"2024-09-13T15:13:57.271953+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:13:57.271953+0000","last_undegraded":"2024-09-13T15:13:57.271953+0000","last_fullsized":"2024-09-13T15:13:57.271953+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":342,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295648+0000","last_change":"2024-09-13T15:08:46.073232+0000","last_active":"2024-09-13T15:13:57.295648+0000","last_peered":"2024-09-13T15:13:57.295648+0000","last_clean":"2024-09-13T15:13:57.295648+0000","last_became_active":"2024-09-13T15:08:46.072807+0000","last_became_peered":"2024-09-13T15:08:46.072807+0000","last_unstale":"2024-09-13T15:13:57.295648+0000","last_undegraded":"2024-09-13T15:13:57.295648+0000","last_fullsized":"2024-09-13T15:13:57.295648+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:11.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":502,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271744+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:13:57.271744+0000","last_peered":"2024-09-13T15:13:57.271744+0000","last_clean":"2024-09-13T15:13:57.271744+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:13:57.271744+0000","last_undegraded":"2024-09-13T15:13:57.271744+0000","last_fullsized":"2024-09-13T15:13:57.271744+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":370,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295988+0000","last_change":"2024-09-13T15:08:46.079600+0000","last_active":"2024-09-13T15:13:57.295988+0000","last_peered":"2024-09-13T15:13:57.295988+0000","last_clean":"2024-09-13T15:13:57.295988+0000","last_became_active":"2024-09-13T15:08:46.075458+0000","last_became_peered":"2024-09-13T15:08:46.075458+0000","last_unstale":"2024-09-13T15:13:57.295988+0000","last_undegraded":"2024-09-13T15:13:57.295988+0000","last_fullsized":"2024-09-13T15:13:57.295988+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:41.707452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":885,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196105+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:14:03.196105+0000","last_peered":"2024-09-13T15:14:03.196105+0000","last_clean":"2024-09-13T15:14:03.196105+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:14:03.196105+0000","last_undegraded":"2024-09-13T15:14:03.196105+0000","last_fullsized":"2024-09-13T15:14:03.196105+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":908,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196311+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:14:03.196311+0000","last_peered":"2024-09-13T15:14:03.196311+0000","last_clean":"2024-09-13T15:14:03.196311+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:14:03.196311+0000","last_undegraded":"2024-09-13T15:14:03.196311+0000","last_fullsized":"2024-09-13T15:14:03.196311+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":535,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271498+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:13:57.271498+0000","last_peered":"2024-09-13T15:13:57.271498+0000","last_clean":"2024-09-13T15:13:57.271498+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:13:57.271498+0000","last_undegraded":"2024-09-13T15:13:57.271498+0000","last_fullsized":"2024-09-13T15:13:57.271498+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":870,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296196+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:13:57.296196+0000","last_peered":"2024-09-13T15:13:57.296196+0000","last_clean":"2024-09-13T15:13:57.296196+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:13:57.296196+0000","last_undegraded":"2024-09-13T15:13:57.296196+0000","last_fullsized":"2024-09-13T15:13:57.296196+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":36578,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234560,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529036,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":522494,"kb_avail":93216116,"statfs":{"total":95995035648,"available":95453302784,"internally_reserved":0,"allocated":6684672,"data_stored":3973242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":535034315},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496196,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727872,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":721022,"kb_avail":93017280,"statfs":{"total":95995035648,"available":95249694720,"internally_reserved":0,"allocated":7000064,"data_stored":4300522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":738326983},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757831,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876288,"kb_used_data":6772,"kb_used_omap":1,"kb_used_meta":869502,"kb_avail":92868864,"statfs":{"total":95995035648,"available":95097716736,"internally_reserved":0,"allocated":6934528,"data_stored":4231687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890370503},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019471,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849800,"kb_used_data":6524,"kb_used_omap":1,"kb_used_meta":843262,"kb_avail":92895352,"statfs":{"total":95995035648,"available":95124840448,"internally_reserved":0,"allocated":6680576,"data_stored":3973481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863500742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468356,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989628,"kb_used_data":11184,"kb_used_omap":4,"kb_used_meta":978427,"kb_avail":92755524,"statfs":{"total":95995035648,"available":94981656576,"internally_reserved":0,"allocated":11452416,"data_stored":8753375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":1001909412},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273255,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433412,"kb_used_data":10872,"kb_used_omap":12,"kb_used_meta":422515,"kb_avail":93311740,"statfs":{"total":95995035648,"available":95551221760,"internally_reserved":0,"allocated":11132928,"data_stored":8426243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":432656256},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894776,"kb_used_data":6508,"kb_used_omap":7,"kb_used_meta":888248,"kb_avail":92850376,"statfs":{"total":95995035648,"available":95078785024,"internally_reserved":0,"allocated":6664192,"data_stored":3968632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":909566578},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502189,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597808,"kb_used_data":10916,"kb_used_omap":10,"kb_used_meta":586869,"kb_avail":93147344,"statfs":{"total":95995035648,"available":95382880256,"internally_reserved":0,"allocated":11177984,"data_stored":8483154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":600954559},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:14:09.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:14:09.617 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:09 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:14:09.617 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:09 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:14:09.717 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:09 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:14:09.730 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:14:09.730 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:14:09.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18355 9.5 9:a1a6939b:::benchmark_data_smithi154_291684_object18354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.140974+0000", 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 533.57461484500004, 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1037532460000001, 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.731 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18355 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.140974+0000", 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.140974+0000", 2024-09-13T15:14:09.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.140975+0000", 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.035e-06 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.140995+0000", 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0543999999999998e-05 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141005+0000", 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7389999999999994e-06 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141012+0000", 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6019999999999996e-06 2024-09-13T15:14:09.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141177+0000", 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016538499999999999 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.142166+0000", 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00098878099999999999 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.142309+0000", 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000143473 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.143598+0000", 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001288305 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150117+0000", 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065194459999999999 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.165723+0000", 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01560604 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.165735+0000", 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2194999999999999e-05 2024-09-13T15:14:09.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.172609+0000", 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.006874185 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.172630+0000", 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0630000000000001e-05 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.244633+0000", 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.072002608 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.244727+0000", 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4277999999999994e-05 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.740 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18359 9.9 9:96f37c08:::benchmark_data_smithi154_291684_object18358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-13T15:14:09.740 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.141617+0000", 2024-09-13T15:14:09.740 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 533.57397124800002, 2024-09-13T15:14:09.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1495450420000002, 2024-09-13T15:14:09.740 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.741 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.741 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:09.741 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.741 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.741 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.741 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18359 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141617+0000", 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141617+0000", 2024-09-13T15:14:09.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141618+0000", 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.61e-07 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141637+0000", 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9097999999999999e-05 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141647+0000", 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0256e-05 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.141655+0000", 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5310000000000004e-06 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.142716+0000", 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001060805 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.142874+0000", 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015864399999999999 2024-09-13T15:14:09.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.143005+0000", 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013074199999999999 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.144123+0000", 2024-09-13T15:14:09.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011176879999999999 2024-09-13T15:14:09.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150419+0000", 2024-09-13T15:14:09.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062958210000000001 2024-09-13T15:14:09.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154012+0000", 2024-09-13T15:14:09.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035937230000000001 2024-09-13T15:14:09.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154025+0000", 2024-09-13T15:14:09.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2354999999999999e-05 2024-09-13T15:14:09.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.164853+0000", 2024-09-13T15:14:09.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.010827948 2024-09-13T15:14:09.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.164877+0000", 2024-09-13T15:14:09.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4848999999999999e-05 2024-09-13T15:14:09.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.291063+0000", 2024-09-13T15:14:09.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1261856219999999 2024-09-13T15:14:09.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.291162+0000", 2024-09-13T15:14:09.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9198999999999997e-05 2024-09-13T15:14:09.752 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.752 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.753 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.753 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18367 9.9 9:986451f7:::benchmark_data_smithi154_291684_object18366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-13T15:14:09.753 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.150034+0000", 2024-09-13T15:14:09.753 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 533.56555480300005, 2024-09-13T15:14:09.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1619533579999999, 2024-09-13T15:14:09.754 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.754 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.754 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:09.754 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.754 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.755 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.755 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18367 2024-09-13T15:14:09.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.755 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150034+0000", 2024-09-13T15:14:09.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150034+0000", 2024-09-13T15:14:09.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150035+0000", 2024-09-13T15:14:09.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.296e-06 2024-09-13T15:14:09.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150058+0000", 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2609999999999999e-05 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150071+0000", 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3821e-05 2024-09-13T15:14:09.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150079+0000", 2024-09-13T15:14:09.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8220000000000007e-06 2024-09-13T15:14:09.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150470+0000", 2024-09-13T15:14:09.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039118300000000003 2024-09-13T15:14:09.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150588+0000", 2024-09-13T15:14:09.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011786 2024-09-13T15:14:09.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 5,7", 2024-09-13T15:14:09.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.150740+0000", 2024-09-13T15:14:09.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000151333 2024-09-13T15:14:09.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154074+0000", 2024-09-13T15:14:09.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033346740000000001 2024-09-13T15:14:09.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154084+0000", 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5519999999999993e-06 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.164973+0000", 2024-09-13T15:14:09.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.010889105 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.164984+0000", 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1167e-05 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.292396+0000", 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1274122529999999 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.293095+0000", 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000699112 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.311840+0000", 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018744013 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.311987+0000", 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014755700000000001 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.768 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18369 9.5 9:a6e44bc8:::benchmark_data_smithi154_291684_object18368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.154949+0000", 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 533.56063977199994, 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1008583870000002, 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.769 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.770 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.770 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18369 2024-09-13T15:14:09.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.770 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154949+0000", 2024-09-13T15:14:09.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154949+0000", 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154950+0000", 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3659999999999999e-06 2024-09-13T15:14:09.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.154974+0000", 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3898999999999999e-05 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157970+0000", 2024-09-13T15:14:09.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029962000000000001 2024-09-13T15:14:09.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.157981+0000", 2024-09-13T15:14:09.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0832999999999999e-05 2024-09-13T15:14:09.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.165765+0000", 2024-09-13T15:14:09.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077840990000000001 2024-09-13T15:14:09.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.165773+0000", 2024-09-13T15:14:09.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0399999999999993e-06 2024-09-13T15:14:09.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.173778+0000", 2024-09-13T15:14:09.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0080048020000001 2024-09-13T15:14:09.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.173789+0000", 2024-09-13T15:14:09.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0652999999999999e-05 2024-09-13T15:14:09.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.244831+0000", 2024-09-13T15:14:09.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.071042228 2024-09-13T15:14:09.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.245047+0000", 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000216216 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,5", 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.245235+0000", 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000187515 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.249793+0000", 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045579369999999998 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.253926+0000", 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041336760000000002 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.254729+0000", 2024-09-13T15:14:09.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080315300000000003 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.254844+0000", 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011449 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.255807+0000", 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096327999999999995 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:18387 9.9 9:92d18070:::benchmark_data_smithi154_291684_object18386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:05:16.181631+0000", 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 533.53395707100003, 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1322882999999999, 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 18387 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181631+0000", 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181631+0000", 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181633+0000", 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7e-06 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181655+0000", 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1835e-05 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181676+0000", 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0822000000000002e-05 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181683+0000", 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5429999999999996e-06 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181740+0000", 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6199000000000002e-05 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:16.181747+0000", 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.306e-06 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.165012+0000", 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.98326508099999999 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:17.165020+0000", 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3000000000000002e-06 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.293139+0000", 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1281190320000001 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.293247+0000", 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010774 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.293419+0000", 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017171999999999999 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.299097+0000", 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056779789999999997 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.300330+0000", 2024-09-13T15:14:09.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001233722 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.312030+0000", 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011699146000000001 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.312107+0000", 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7694000000000005e-05 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:05:18.313920+0000", 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018124809999999999 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:182957 9.15 9:aa0f1e4e:::benchmark_data_smithi154_291684_object82496:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.192485+0000", 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.523103789, 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.302578193, 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.791 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 182957 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192485+0000", 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192485+0000", 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192485+0000", 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5900000000000002e-07 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192487+0000", 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0499999999999999e-06 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192493+0000", 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1380000000000004e-06 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192498+0000", 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.882e-06 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192819+0000", 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032121899999999998 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192826+0000", 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.55e-06 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.486798+0000", 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.293972398 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.486940+0000", 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141936 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.487258+0000", 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031716300000000003 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.489689+0000", 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002431688 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.494337+0000", 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046479720000000002 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.494388+0000", 2024-09-13T15:14:09.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1187999999999998e-05 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.494902+0000", 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00051314799999999995 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.494951+0000", 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9036000000000003e-05 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.495063+0000", 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112366 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183037 9.15 9:aeaf82d4:::benchmark_data_smithi154_291684_object82576:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.192505+0000", 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.52308310199999, 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3131593770000001, 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.799 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183037 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192505+0000", 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192505+0000", 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192506+0000", 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1899999999999998e-07 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192508+0000", 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1909999999999999e-06 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192515+0000", 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8990000000000004e-06 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192520+0000", 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0509999999999996e-06 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192854+0000", 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033451500000000002 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192866+0000", 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1945e-05 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.489721+0000", 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2968543450000001 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.489890+0000", 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016906000000000001 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.490235+0000", 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034519299999999999 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.494623+0000", 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043879879999999998 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.497875+0000", 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032520930000000002 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.497964+0000", 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8453000000000002e-05 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505401+0000", 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074373900000000003 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505513+0000", 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011236 2024-09-13T15:14:09.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505665+0000", 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015147500000000001 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183044 9.15 9:adfe8015:::benchmark_data_smithi154_291684_object82583:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.192527+0000", 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.52306175000001, 2024-09-13T15:14:09.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3258470149999999, 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183044 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192527+0000", 2024-09-13T15:14:09.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192527+0000", 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192527+0000", 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9099999999999999e-07 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192529+0000", 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.827e-06 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192535+0000", 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8669999999999999e-06 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192540+0000", 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0119999999999996e-06 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192889+0000", 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034938999999999999 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192895+0000", 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2199999999999997e-06 2024-09-13T15:14:09.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.492020+0000", 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2991249229999999 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.492147+0000", 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012682599999999999 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.492455+0000", 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000307965 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.497890+0000", 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054350869999999999 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.497926+0000", 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5305999999999997e-05 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.498009+0000", 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3458e-05 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518150+0000", 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020141454 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518221+0000", 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0970999999999995e-05 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518374+0000", 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152318 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183073 9.15 9:a82869f4:::benchmark_data_smithi154_291684_object82612:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.816 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.192545+0000", 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.523043325, 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.326222826, 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183073 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192545+0000", 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192545+0000", 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192545+0000", 2024-09-13T15:14:09.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0400000000000002e-07 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192547+0000", 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.846e-06 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192553+0000", 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0689999999999998e-06 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192559+0000", 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9089999999999997e-06 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192921+0000", 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00036203500000000003 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192928+0000", 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4470000000000003e-06 2024-09-13T15:14:09.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.495090+0000", 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3021620330000001 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.495190+0000", 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010039699999999999 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.495510+0000", 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031964999999999999 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505679+0000", 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010168762 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505720+0000", 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1059000000000001e-05 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505784+0000", 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4715999999999995e-05 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518491+0000", 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012706497000000001 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518542+0000", 2024-09-13T15:14:09.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1202999999999997e-05 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518768+0000", 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000225899 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183078 9.15 9:ad43649b:::benchmark_data_smithi154_291684_object82617:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.192565+0000", 2024-09-13T15:14:09.824 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.52302376099999, 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.327331016, 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183078 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192565+0000", 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192565+0000", 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192565+0000", 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1899999999999998e-07 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192567+0000", 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2060000000000001e-06 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192572+0000", 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2140000000000004e-06 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192577+0000", 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5709999999999999e-06 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192950+0000", 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000373168 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192957+0000", 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4989999999999999e-06 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.498052+0000", 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.305095382 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.498188+0000", 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013544100000000001 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.498541+0000", 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000353832 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505736+0000", 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071945150000000003 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505828+0000", 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2107000000000004e-05 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.510606+0000", 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047778619999999999 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.519052+0000", 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084461499999999995 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.519763+0000", 2024-09-13T15:14:09.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00071067000000000005 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.519896+0000", 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013307999999999999 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183172 9.15 9:ae7b69a4:::benchmark_data_smithi154_291684_object82711:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.192583+0000", 2024-09-13T15:14:09.832 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.52300558100001, 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3275205729999999, 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183172 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192583+0000", 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192583+0000", 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192583+0000", 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.15e-07 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192585+0000", 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6139999999999999e-06 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192591+0000", 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8320000000000002e-06 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192597+0000", 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7850000000000003e-06 2024-09-13T15:14:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192980+0000", 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038302300000000002 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192987+0000", 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8569999999999997e-06 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.501099+0000", 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.308111233 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.501229+0000", 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013050799999999999 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.501617+0000", 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038826000000000003 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505742+0000", 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041246540000000002 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505884+0000", 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141982 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.510661+0000", 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047771999999999997 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.519943+0000", 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092818750000000002 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.519990+0000", 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7197999999999998e-05 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520104+0000", 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113237 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183190 9.15 9:ac6735b1:::benchmark_data_smithi154_291684_object82729:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.192604+0000", 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.52298496399999, 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3276992459999999, 2024-09-13T15:14:09.840 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183190 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192604+0000", 2024-09-13T15:14:09.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192604+0000", 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192604+0000", 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0100000000000001e-07 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192606+0000", 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.04e-06 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192611+0000", 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5659999999999997e-06 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.192617+0000", 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3800000000000002e-06 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.193964+0000", 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001347659 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.193972+0000", 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8390000000000007e-06 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.505920+0000", 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.311947888 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.506040+0000", 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119747 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.506418+0000", 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037769700000000003 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.510617+0000", 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041996179999999996 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.510699+0000", 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1339000000000002e-05 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518814+0000", 2024-09-13T15:14:09.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081157299999999998 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520145+0000", 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013302260000000001 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520191+0000", 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6072000000000002e-05 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520303+0000", 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112144 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183191 9.15 9:aaffb343:::benchmark_data_smithi154_291684_object82730:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.194139+0000", 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.52144989300001, 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3352457630000001, 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.848 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183191 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194139+0000", 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194139+0000", 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194140+0000", 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2399999999999996e-07 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194149+0000", 2024-09-13T15:14:09.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2199999999999998e-06 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194162+0000", 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2828e-05 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194168+0000", 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7580000000000003e-06 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194212+0000", 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4053e-05 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194219+0000", 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.477e-06 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.508197+0000", 2024-09-13T15:14:09.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3139780729999999 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.508306+0000", 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010898700000000001 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.508630+0000", 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032379700000000002 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.515551+0000", 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069210720000000003 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518441+0000", 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028898779999999998 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.518856+0000", 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000415516 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529019+0000", 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010162825 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529200+0000", 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000181156 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529384+0000", 2024-09-13T15:14:09.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018419899999999999 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183202 9.15 9:adceeee4:::benchmark_data_smithi154_291684_object82741:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.194231+0000", 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.52135776699998, 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.335487563, 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.856 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183202 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194231+0000", 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194231+0000", 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194231+0000", 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0200000000000002e-07 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194235+0000", 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3139999999999999e-06 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194241+0000", 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5939999999999999e-06 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194247+0000", 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.003e-06 2024-09-13T15:14:09.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194294+0000", 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6830999999999999e-05 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194302+0000", 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9009999999999994e-06 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.510731+0000", 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.316428645 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.510843+0000", 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112896 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.511172+0000", 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032891300000000001 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520340+0000", 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091677860000000007 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.526066+0000", 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005726007 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.526230+0000", 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016365300000000001 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529427+0000", 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003197624 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529599+0000", 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017129800000000001 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529718+0000", 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119596 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183262 9.15 9:a894d57b:::benchmark_data_smithi154_291684_object82801:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.194316+0000", 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.521272684, 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3356197869999999, 2024-09-13T15:14:09.864 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183262 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194316+0000", 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194316+0000", 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194316+0000", 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4000000000000001e-07 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194320+0000", 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0709999999999999e-06 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194327+0000", 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2629999999999997e-06 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194332+0000", 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5310000000000001e-06 2024-09-13T15:14:09.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194377+0000", 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4320999999999998e-05 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194384+0000", 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4880000000000001e-06 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.513159+0000", 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3187746 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.513278+0000", 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119567 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.513574+0000", 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029547500000000001 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520379+0000", 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068047730000000001 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.526153+0000", 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057747780000000004 2024-09-13T15:14:09.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.526236+0000", 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2464000000000002e-05 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529762+0000", 2024-09-13T15:14:09.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003526394 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529814+0000", 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1663000000000003e-05 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529936+0000", 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121759 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.872 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183270 9.15 9:ad936b06:::benchmark_data_smithi154_291684_object82809:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.194406+0000", 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.521182076, 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.335727619, 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.873 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183270 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194406+0000", 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194406+0000", 2024-09-13T15:14:09.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194407+0000", 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7000000000000005e-07 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194410+0000", 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2650000000000001e-06 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194417+0000", 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8700000000000003e-06 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194456+0000", 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9313000000000001e-05 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194581+0000", 2024-09-13T15:14:09.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012495299999999999 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194590+0000", 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5050000000000007e-06 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.515580+0000", 2024-09-13T15:14:09.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.320990468 2024-09-13T15:14:09.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.515693+0000", 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112939 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.515995+0000", 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030204199999999998 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520418+0000", 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044228160000000004 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.526197+0000", 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005778488 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.526241+0000", 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3945e-05 2024-09-13T15:14:09.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.529972+0000", 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037317100000000001 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.530023+0000", 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1147999999999999e-05 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.530134+0000", 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110587 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:183274 9.15 9:a9c68c15:::benchmark_data_smithi154_291684_object82813:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e437)", 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:31.194604+0000", 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.520984274, 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3463566730000001, 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:09.883 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 183274 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194604+0000", 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194604+0000", 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194605+0000", 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1900000000000002e-07 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194612+0000", 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.888e-06 2024-09-13T15:14:09.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194620+0000", 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9300000000000003e-06 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194625+0000", 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6699999999999999e-06 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194672+0000", 2024-09-13T15:14:09.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6801999999999997e-05 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:31.194683+0000", 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0852e-05 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520449+0000", 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.325765995 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520555+0000", 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105716 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3,7", 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.520944+0000", 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038880499999999999 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.526246+0000", 2024-09-13T15:14:09.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005302289 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.530170+0000", 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039245199999999999 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.534758+0000", 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045873889999999999 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.540731+0000", 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059735910000000003 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.540815+0000", 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3708999999999994e-05 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:32.540961+0000", 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014589799999999999 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.890 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:09.891 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:09.891 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:14:10.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:10 smithi172 ceph-mon[100696]: pgmap v1615: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:10.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:10 smithi154 ceph-mon[102314]: pgmap v1615: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:10.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:10 smithi154 ceph-mon[107038]: pgmap v1615: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:10.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-13T15:14:11.800 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:12.213 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:14:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:14:11] "GET /metrics HTTP/1.1" 200 34229 "" "Prometheus/2.51.0" 2024-09-13T15:14:12.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:12 smithi154 ceph-mon[102314]: pgmap v1616: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:12.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:12 smithi154 ceph-mon[107038]: pgmap v1616: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:12.653 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:12 smithi172 ceph-mon[100696]: pgmap v1616: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:13.125 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:13.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:13.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:13.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:14:14.159 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:14:14.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:14 smithi154 ceph-mon[102314]: pgmap v1617: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:14.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:14 smithi154 ceph-mon[107038]: pgmap v1617: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:14.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:14 smithi172 ceph-mon[100696]: pgmap v1617: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:15.043 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1617,"stamp":"2024-09-13T15:14:13.277281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":36635,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4162,"ondisk_log_size":4162,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5898632,"kb_used_data":66152,"kb_used_omap":40,"kb_used_meta":5832343,"kb_avail":744062584,"statfs":{"total":767960285184,"available":761920086016,"internally_reserved":0,"allocated":67739648,"data_stored":46118383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":5972319348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.992639"},"pg_stats":[{"pgid":"1.11","version":"89'149","reported_seq":873,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499782+0000","last_change":"2024-09-13T15:13:01.179220+0000","last_active":"2024-09-13T15:14:02.499782+0000","last_peered":"2024-09-13T15:14:02.499782+0000","last_clean":"2024-09-13T15:14:02.499782+0000","last_became_active":"2024-09-13T15:12:50.169695+0000","last_became_peered":"2024-09-13T15:12:50.169695+0000","last_unstale":"2024-09-13T15:14:02.499782+0000","last_undegraded":"2024-09-13T15:14:02.499782+0000","last_fullsized":"2024-09-13T15:14:02.499782+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.179154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.179154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:04.917248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":224,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195793+0000","last_change":"2024-09-13T15:08:31.190995+0000","last_active":"2024-09-13T15:14:03.195793+0000","last_peered":"2024-09-13T15:14:03.195793+0000","last_clean":"2024-09-13T15:14:03.195793+0000","last_became_active":"2024-09-13T15:08:31.190610+0000","last_became_peered":"2024-09-13T15:08:31.190610+0000","last_unstale":"2024-09-13T15:14:03.195793+0000","last_undegraded":"2024-09-13T15:14:03.195793+0000","last_fullsized":"2024-09-13T15:14:03.195793+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:42.124757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":81,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295970+0000","last_change":"2024-09-13T15:08:31.191585+0000","last_active":"2024-09-13T15:13:57.295970+0000","last_peered":"2024-09-13T15:13:57.295970+0000","last_clean":"2024-09-13T15:13:57.295970+0000","last_became_active":"2024-09-13T15:08:31.184700+0000","last_became_peered":"2024-09-13T15:08:31.184700+0000","last_unstale":"2024-09-13T15:13:57.295970+0000","last_undegraded":"2024-09-13T15:13:57.295970+0000","last_fullsized":"2024-09-13T15:13:57.295970+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:14.376834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"89'149","reported_seq":839,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295636+0000","last_change":"2024-09-13T15:13:09.203525+0000","last_active":"2024-09-13T15:13:57.295636+0000","last_peered":"2024-09-13T15:13:57.295636+0000","last_clean":"2024-09-13T15:13:57.295636+0000","last_became_active":"2024-09-13T15:12:50.159713+0000","last_became_peered":"2024-09-13T15:12:50.159713+0000","last_unstale":"2024-09-13T15:13:57.295636+0000","last_undegraded":"2024-09-13T15:13:57.295636+0000","last_fullsized":"2024-09-13T15:13:57.295636+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:09.203387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:09.203387+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:07.004095+0000","scrub_duration":0.002438127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":358,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295907+0000","last_change":"2024-09-13T15:08:46.080613+0000","last_active":"2024-09-13T15:13:57.295907+0000","last_peered":"2024-09-13T15:13:57.295907+0000","last_clean":"2024-09-13T15:13:57.295907+0000","last_became_active":"2024-09-13T15:08:46.080320+0000","last_became_peered":"2024-09-13T15:08:46.080320+0000","last_unstale":"2024-09-13T15:13:57.295907+0000","last_undegraded":"2024-09-13T15:13:57.295907+0000","last_fullsized":"2024-09-13T15:13:57.295907+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:09.205609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":839,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499913+0000","last_change":"2024-09-13T15:08:31.196449+0000","last_active":"2024-09-13T15:14:02.499913+0000","last_peered":"2024-09-13T15:14:02.499913+0000","last_clean":"2024-09-13T15:14:02.499913+0000","last_became_active":"2024-09-13T15:08:31.196143+0000","last_became_peered":"2024-09-13T15:08:31.196143+0000","last_unstale":"2024-09-13T15:14:02.499913+0000","last_undegraded":"2024-09-13T15:14:02.499913+0000","last_fullsized":"2024-09-13T15:14:02.499913+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:53.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"89'149","reported_seq":925,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795677+0000","last_change":"2024-09-13T15:13:03.099175+0000","last_active":"2024-09-13T15:14:02.795677+0000","last_peered":"2024-09-13T15:14:02.795677+0000","last_clean":"2024-09-13T15:14:02.795677+0000","last_became_active":"2024-09-13T15:12:50.196974+0000","last_became_peered":"2024-09-13T15:12:50.196974+0000","last_unstale":"2024-09-13T15:14:02.795677+0000","last_undegraded":"2024-09-13T15:14:02.795677+0000","last_fullsized":"2024-09-13T15:14:02.795677+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.098941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.098941+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:19.670702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":842,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500194+0000","last_change":"2024-09-13T15:08:31.195499+0000","last_active":"2024-09-13T15:14:02.500194+0000","last_peered":"2024-09-13T15:14:02.500194+0000","last_clean":"2024-09-13T15:14:02.500194+0000","last_became_active":"2024-09-13T15:08:31.194928+0000","last_became_peered":"2024-09-13T15:08:31.194928+0000","last_unstale":"2024-09-13T15:14:02.500194+0000","last_undegraded":"2024-09-13T15:14:02.500194+0000","last_fullsized":"2024-09-13T15:14:02.500194+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.112396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":75,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295668+0000","last_change":"2024-09-13T15:08:31.185658+0000","last_active":"2024-09-13T15:13:57.295668+0000","last_peered":"2024-09-13T15:13:57.295668+0000","last_clean":"2024-09-13T15:13:57.295668+0000","last_became_active":"2024-09-13T15:08:31.185202+0000","last_became_peered":"2024-09-13T15:08:31.185202+0000","last_unstale":"2024-09-13T15:13:57.295668+0000","last_undegraded":"2024-09-13T15:13:57.295668+0000","last_fullsized":"2024-09-13T15:13:57.295668+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:04.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"89'149","reported_seq":839,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296210+0000","last_change":"2024-09-13T15:13:11.154190+0000","last_active":"2024-09-13T15:13:57.296210+0000","last_peered":"2024-09-13T15:13:57.296210+0000","last_clean":"2024-09-13T15:13:57.296210+0000","last_became_active":"2024-09-13T15:12:50.159857+0000","last_became_peered":"2024-09-13T15:12:50.159857+0000","last_unstale":"2024-09-13T15:13:57.296210+0000","last_undegraded":"2024-09-13T15:13:57.296210+0000","last_fullsized":"2024-09-13T15:13:57.296210+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.153852+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.153852+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:54.235242+0000","scrub_duration":0.0043281400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":345,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196231+0000","last_change":"2024-09-13T15:08:31.199682+0000","last_active":"2024-09-13T15:14:03.196231+0000","last_peered":"2024-09-13T15:14:03.196231+0000","last_clean":"2024-09-13T15:14:03.196231+0000","last_became_active":"2024-09-13T15:08:31.199110+0000","last_became_peered":"2024-09-13T15:08:31.199110+0000","last_unstale":"2024-09-13T15:14:03.196231+0000","last_undegraded":"2024-09-13T15:14:03.196231+0000","last_fullsized":"2024-09-13T15:14:03.196231+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:30.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":449,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499720+0000","last_change":"2024-09-13T15:04:05.021292+0000","last_active":"2024-09-13T15:14:02.499720+0000","last_peered":"2024-09-13T15:14:02.499720+0000","last_clean":"2024-09-13T15:14:02.499720+0000","last_became_active":"2024-09-13T15:04:05.011146+0000","last_became_peered":"2024-09-13T15:04:05.011146+0000","last_unstale":"2024-09-13T15:14:02.499720+0000","last_undegraded":"2024-09-13T15:14:02.499720+0000","last_fullsized":"2024-09-13T15:14:02.499720+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:00.137694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"89'149","reported_seq":840,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296487+0000","last_change":"2024-09-13T15:13:15.494950+0000","last_active":"2024-09-13T15:13:57.296487+0000","last_peered":"2024-09-13T15:13:57.296487+0000","last_clean":"2024-09-13T15:13:57.296487+0000","last_became_active":"2024-09-13T15:12:50.163288+0000","last_became_peered":"2024-09-13T15:12:50.163288+0000","last_unstale":"2024-09-13T15:13:57.296487+0000","last_undegraded":"2024-09-13T15:13:57.296487+0000","last_fullsized":"2024-09-13T15:13:57.296487+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:15.494798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:15.494798+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:34.790443+0000","scrub_duration":0.0082230789999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296864+0000","last_change":"2024-09-13T15:08:31.351504+0000","last_active":"2024-09-13T15:13:57.296864+0000","last_peered":"2024-09-13T15:13:57.296864+0000","last_clean":"2024-09-13T15:13:57.296864+0000","last_became_active":"2024-09-13T15:08:31.320950+0000","last_became_peered":"2024-09-13T15:08:31.320950+0000","last_unstale":"2024-09-13T15:13:57.296864+0000","last_undegraded":"2024-09-13T15:13:57.296864+0000","last_fullsized":"2024-09-13T15:13:57.296864+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:38.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":404,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295807+0000","last_change":"2024-09-13T15:08:46.078296+0000","last_active":"2024-09-13T15:13:57.295807+0000","last_peered":"2024-09-13T15:13:57.295807+0000","last_clean":"2024-09-13T15:13:57.295807+0000","last_became_active":"2024-09-13T15:08:46.078037+0000","last_became_peered":"2024-09-13T15:08:46.078037+0000","last_unstale":"2024-09-13T15:13:57.295807+0000","last_undegraded":"2024-09-13T15:13:57.295807+0000","last_fullsized":"2024-09-13T15:13:57.295807+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:49.959361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"89'149","reported_seq":932,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195672+0000","last_change":"2024-09-13T15:13:02.045213+0000","last_active":"2024-09-13T15:14:03.195672+0000","last_peered":"2024-09-13T15:14:03.195672+0000","last_clean":"2024-09-13T15:14:03.195672+0000","last_became_active":"2024-09-13T15:12:50.170281+0000","last_became_peered":"2024-09-13T15:12:50.170281+0000","last_unstale":"2024-09-13T15:14:03.195672+0000","last_undegraded":"2024-09-13T15:14:03.195672+0000","last_fullsized":"2024-09-13T15:14:03.195672+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:02.045141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:02.045141+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:03.286894+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296377+0000","last_change":"2024-09-13T15:08:31.201939+0000","last_active":"2024-09-13T15:13:57.296377+0000","last_peered":"2024-09-13T15:13:57.296377+0000","last_clean":"2024-09-13T15:13:57.296377+0000","last_became_active":"2024-09-13T15:08:31.191089+0000","last_became_peered":"2024-09-13T15:08:31.191089+0000","last_unstale":"2024-09-13T15:13:57.296377+0000","last_undegraded":"2024-09-13T15:13:57.296377+0000","last_fullsized":"2024-09-13T15:13:57.296377+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:06.249945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":568,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271641+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:13:57.271641+0000","last_peered":"2024-09-13T15:13:57.271641+0000","last_clean":"2024-09-13T15:13:57.271641+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:13:57.271641+0000","last_undegraded":"2024-09-13T15:13:57.271641+0000","last_fullsized":"2024-09-13T15:13:57.271641+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"89'149","reported_seq":899,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296522+0000","last_change":"2024-09-13T15:13:06.128357+0000","last_active":"2024-09-13T15:13:57.296522+0000","last_peered":"2024-09-13T15:13:57.296522+0000","last_clean":"2024-09-13T15:13:57.296522+0000","last_became_active":"2024-09-13T15:12:50.160188+0000","last_became_peered":"2024-09-13T15:12:50.160188+0000","last_unstale":"2024-09-13T15:13:57.296522+0000","last_undegraded":"2024-09-13T15:13:57.296522+0000","last_fullsized":"2024-09-13T15:13:57.296522+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:06.128281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:06.128281+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:11.473186+0000","scrub_duration":0.0011969070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296728+0000","last_change":"2024-09-13T15:08:31.199736+0000","last_active":"2024-09-13T15:13:57.296728+0000","last_peered":"2024-09-13T15:13:57.296728+0000","last_clean":"2024-09-13T15:13:57.296728+0000","last_became_active":"2024-09-13T15:08:31.186018+0000","last_became_peered":"2024-09-13T15:08:31.186018+0000","last_unstale":"2024-09-13T15:13:57.296728+0000","last_undegraded":"2024-09-13T15:13:57.296728+0000","last_fullsized":"2024-09-13T15:13:57.296728+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:26.855187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":580,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795646+0000","last_change":"2024-09-13T15:08:31.193478+0000","last_active":"2024-09-13T15:14:02.795646+0000","last_peered":"2024-09-13T15:14:02.795646+0000","last_clean":"2024-09-13T15:14:02.795646+0000","last_became_active":"2024-09-13T15:08:31.193038+0000","last_became_peered":"2024-09-13T15:08:31.193038+0000","last_unstale":"2024-09-13T15:14:02.795646+0000","last_undegraded":"2024-09-13T15:14:02.795646+0000","last_fullsized":"2024-09-13T15:14:02.795646+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:05.600387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"89'149","reported_seq":908,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272101+0000","last_change":"2024-09-13T15:13:01.670578+0000","last_active":"2024-09-13T15:13:57.272101+0000","last_peered":"2024-09-13T15:13:57.272101+0000","last_clean":"2024-09-13T15:13:57.272101+0000","last_became_active":"2024-09-13T15:12:50.166107+0000","last_became_peered":"2024-09-13T15:12:50.166107+0000","last_unstale":"2024-09-13T15:13:57.272101+0000","last_undegraded":"2024-09-13T15:13:57.272101+0000","last_fullsized":"2024-09-13T15:13:57.272101+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.670400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.670400+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:19.151283+0000","scrub_duration":0.0070424579999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":375,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295748+0000","last_change":"2024-09-13T15:08:46.073352+0000","last_active":"2024-09-13T15:13:57.295748+0000","last_peered":"2024-09-13T15:13:57.295748+0000","last_clean":"2024-09-13T15:13:57.295748+0000","last_became_active":"2024-09-13T15:08:46.073048+0000","last_became_peered":"2024-09-13T15:08:46.073048+0000","last_unstale":"2024-09-13T15:13:57.295748+0000","last_undegraded":"2024-09-13T15:13:57.295748+0000","last_fullsized":"2024-09-13T15:13:57.295748+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:57.277685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":751,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499828+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:14:02.499828+0000","last_peered":"2024-09-13T15:14:02.499828+0000","last_clean":"2024-09-13T15:14:02.499828+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:14:02.499828+0000","last_undegraded":"2024-09-13T15:14:02.499828+0000","last_fullsized":"2024-09-13T15:14:02.499828+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"89'149","reported_seq":880,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295542+0000","last_change":"2024-09-13T15:10:26.504570+0000","last_active":"2024-09-13T15:13:57.295542+0000","last_peered":"2024-09-13T15:13:57.295542+0000","last_clean":"2024-09-13T15:13:57.295542+0000","last_became_active":"2024-09-13T15:10:19.782654+0000","last_became_peered":"2024-09-13T15:10:19.782654+0000","last_unstale":"2024-09-13T15:13:57.295542+0000","last_undegraded":"2024-09-13T15:13:57.295542+0000","last_fullsized":"2024-09-13T15:13:57.295542+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:10:26.504369+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:57.248804+0000","scrub_duration":0.013934767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":879,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195713+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:14:03.195713+0000","last_peered":"2024-09-13T15:14:03.195713+0000","last_clean":"2024-09-13T15:14:03.195713+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:14:03.195713+0000","last_undegraded":"2024-09-13T15:14:03.195713+0000","last_fullsized":"2024-09-13T15:14:03.195713+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":510,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271822+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:13:57.271822+0000","last_peered":"2024-09-13T15:13:57.271822+0000","last_clean":"2024-09-13T15:13:57.271822+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:13:57.271822+0000","last_undegraded":"2024-09-13T15:13:57.271822+0000","last_fullsized":"2024-09-13T15:13:57.271822+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"89'149","reported_seq":858,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295154+0000","last_change":"2024-09-13T15:10:20.069094+0000","last_active":"2024-09-13T15:13:57.295154+0000","last_peered":"2024-09-13T15:13:57.295154+0000","last_clean":"2024-09-13T15:13:57.295154+0000","last_became_active":"2024-09-13T15:10:19.782890+0000","last_became_peered":"2024-09-13T15:10:19.782890+0000","last_unstale":"2024-09-13T15:13:57.295154+0000","last_undegraded":"2024-09-13T15:13:57.295154+0000","last_fullsized":"2024-09-13T15:13:57.295154+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:20.068973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:20.068973+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:30.570922+0000","scrub_duration":0.011313818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":308,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271719+0000","last_change":"2024-09-13T15:04:05.013503+0000","last_active":"2024-09-13T15:13:57.271719+0000","last_peered":"2024-09-13T15:13:57.271719+0000","last_clean":"2024-09-13T15:13:57.271719+0000","last_became_active":"2024-09-13T15:04:05.013204+0000","last_became_peered":"2024-09-13T15:04:05.013204+0000","last_unstale":"2024-09-13T15:13:57.271719+0000","last_undegraded":"2024-09-13T15:13:57.271719+0000","last_fullsized":"2024-09-13T15:13:57.271719+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:17.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":504,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272111+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:13:57.272111+0000","last_peered":"2024-09-13T15:13:57.272111+0000","last_clean":"2024-09-13T15:13:57.272111+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:13:57.272111+0000","last_undegraded":"2024-09-13T15:13:57.272111+0000","last_fullsized":"2024-09-13T15:13:57.272111+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"89'149","reported_seq":925,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795868+0000","last_change":"2024-09-13T15:13:02.125234+0000","last_active":"2024-09-13T15:14:02.795868+0000","last_peered":"2024-09-13T15:14:02.795868+0000","last_clean":"2024-09-13T15:14:02.795868+0000","last_became_active":"2024-09-13T15:12:50.176602+0000","last_became_peered":"2024-09-13T15:12:50.176602+0000","last_unstale":"2024-09-13T15:14:02.795868+0000","last_undegraded":"2024-09-13T15:14:02.795868+0000","last_fullsized":"2024-09-13T15:14:02.795868+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:02.125166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:02.125166+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.682602+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":529,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271697+0000","last_change":"2024-09-13T15:08:31.184973+0000","last_active":"2024-09-13T15:13:57.271697+0000","last_peered":"2024-09-13T15:13:57.271697+0000","last_clean":"2024-09-13T15:13:57.271697+0000","last_became_active":"2024-09-13T15:08:31.184600+0000","last_became_peered":"2024-09-13T15:08:31.184600+0000","last_unstale":"2024-09-13T15:13:57.271697+0000","last_undegraded":"2024-09-13T15:13:57.271697+0000","last_fullsized":"2024-09-13T15:13:57.271697+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:09.452895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":537,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272207+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:13:57.272207+0000","last_peered":"2024-09-13T15:13:57.272207+0000","last_clean":"2024-09-13T15:13:57.272207+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:13:57.272207+0000","last_undegraded":"2024-09-13T15:13:57.272207+0000","last_fullsized":"2024-09-13T15:13:57.272207+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"89'149","reported_seq":858,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296131+0000","last_change":"2024-09-13T15:10:22.099331+0000","last_active":"2024-09-13T15:13:57.296131+0000","last_peered":"2024-09-13T15:13:57.296131+0000","last_clean":"2024-09-13T15:13:57.296131+0000","last_became_active":"2024-09-13T15:10:19.785103+0000","last_became_peered":"2024-09-13T15:10:19.785103+0000","last_unstale":"2024-09-13T15:13:57.296131+0000","last_undegraded":"2024-09-13T15:13:57.296131+0000","last_fullsized":"2024-09-13T15:13:57.296131+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:22.099204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:22.099204+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:34.804057+0000","scrub_duration":0.0069277289999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":345,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195427+0000","last_change":"2024-09-13T15:08:31.198602+0000","last_active":"2024-09-13T15:14:03.195427+0000","last_peered":"2024-09-13T15:14:03.195427+0000","last_clean":"2024-09-13T15:14:03.195427+0000","last_became_active":"2024-09-13T15:08:31.198289+0000","last_became_peered":"2024-09-13T15:08:31.198289+0000","last_unstale":"2024-09-13T15:14:03.195427+0000","last_undegraded":"2024-09-13T15:14:03.195427+0000","last_fullsized":"2024-09-13T15:14:03.195427+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:36.803833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":624,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195962+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:14:03.195962+0000","last_peered":"2024-09-13T15:14:03.195962+0000","last_clean":"2024-09-13T15:14:03.195962+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:14:03.195962+0000","last_undegraded":"2024-09-13T15:14:03.195962+0000","last_fullsized":"2024-09-13T15:14:03.195962+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":885,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295516+0000","last_change":"2024-09-13T15:13:11.597792+0000","last_active":"2024-09-13T15:13:57.295516+0000","last_peered":"2024-09-13T15:13:57.295516+0000","last_clean":"2024-09-13T15:13:57.295516+0000","last_became_active":"2024-09-13T15:12:38.938735+0000","last_became_peered":"2024-09-13T15:12:38.938735+0000","last_unstale":"2024-09-13T15:13:57.295516+0000","last_undegraded":"2024-09-13T15:13:57.295516+0000","last_fullsized":"2024-09-13T15:13:57.295516+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.597623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.597623+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:14.909270+0000","scrub_duration":0.012675130999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1171,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296353+0000","last_change":"2024-09-13T15:08:57.728783+0000","last_active":"2024-09-13T15:13:57.296353+0000","last_peered":"2024-09-13T15:13:57.296353+0000","last_clean":"2024-09-13T15:13:57.296353+0000","last_became_active":"2024-09-13T15:08:47.077688+0000","last_became_peered":"2024-09-13T15:08:47.077688+0000","last_unstale":"2024-09-13T15:13:57.296353+0000","last_undegraded":"2024-09-13T15:13:57.296353+0000","last_fullsized":"2024-09-13T15:13:57.296353+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:52.710051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":841,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195587+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:14:03.195587+0000","last_peered":"2024-09-13T15:14:03.195587+0000","last_clean":"2024-09-13T15:14:03.195587+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:14:03.195587+0000","last_undegraded":"2024-09-13T15:14:03.195587+0000","last_fullsized":"2024-09-13T15:14:03.195587+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":356,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271975+0000","last_change":"2024-09-13T15:08:31.187510+0000","last_active":"2024-09-13T15:13:57.271975+0000","last_peered":"2024-09-13T15:13:57.271975+0000","last_clean":"2024-09-13T15:13:57.271975+0000","last_became_active":"2024-09-13T15:08:31.187108+0000","last_became_peered":"2024-09-13T15:08:31.187108+0000","last_unstale":"2024-09-13T15:13:57.271975+0000","last_undegraded":"2024-09-13T15:13:57.271975+0000","last_fullsized":"2024-09-13T15:13:57.271975+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:25.096543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"89'149","reported_seq":922,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795744+0000","last_change":"2024-09-13T15:13:04.107856+0000","last_active":"2024-09-13T15:14:02.795744+0000","last_peered":"2024-09-13T15:14:02.795744+0000","last_clean":"2024-09-13T15:14:02.795744+0000","last_became_active":"2024-09-13T15:12:38.929136+0000","last_became_peered":"2024-09-13T15:12:38.929136+0000","last_unstale":"2024-09-13T15:14:02.795744+0000","last_undegraded":"2024-09-13T15:14:02.795744+0000","last_fullsized":"2024-09-13T15:14:02.795744+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.107315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.107315+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:20.960972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":93,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295305+0000","last_change":"2024-09-13T15:08:31.218115+0000","last_active":"2024-09-13T15:13:57.295305+0000","last_peered":"2024-09-13T15:13:57.295305+0000","last_clean":"2024-09-13T15:13:57.295305+0000","last_became_active":"2024-09-13T15:08:31.196118+0000","last_became_peered":"2024-09-13T15:08:31.196118+0000","last_unstale":"2024-09-13T15:13:57.295305+0000","last_undegraded":"2024-09-13T15:13:57.295305+0000","last_fullsized":"2024-09-13T15:13:57.295305+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:08.721976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":825,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499708+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:14:02.499708+0000","last_peered":"2024-09-13T15:14:02.499708+0000","last_clean":"2024-09-13T15:14:02.499708+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:14:02.499708+0000","last_undegraded":"2024-09-13T15:14:02.499708+0000","last_fullsized":"2024-09-13T15:14:02.499708+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":820,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500111+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:14:02.500111+0000","last_peered":"2024-09-13T15:14:02.500111+0000","last_clean":"2024-09-13T15:14:02.500111+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:14:02.500111+0000","last_undegraded":"2024-09-13T15:14:02.500111+0000","last_fullsized":"2024-09-13T15:14:02.500111+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"89'149","reported_seq":901,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295245+0000","last_change":"2024-09-13T15:13:05.496011+0000","last_active":"2024-09-13T15:13:57.295245+0000","last_peered":"2024-09-13T15:13:57.295245+0000","last_clean":"2024-09-13T15:13:57.295245+0000","last_became_active":"2024-09-13T15:12:38.946687+0000","last_became_peered":"2024-09-13T15:12:38.946687+0000","last_unstale":"2024-09-13T15:13:57.295245+0000","last_undegraded":"2024-09-13T15:13:57.295245+0000","last_fullsized":"2024-09-13T15:13:57.295245+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:05.495826+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:05.495826+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:01.133250+0000","scrub_duration":0.011811428000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1261,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196006+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:14:03.196006+0000","last_peered":"2024-09-13T15:14:03.196006+0000","last_clean":"2024-09-13T15:14:03.196006+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:14:03.196006+0000","last_undegraded":"2024-09-13T15:14:03.196006+0000","last_fullsized":"2024-09-13T15:14:03.196006+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":890,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795719+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:14:02.795719+0000","last_peered":"2024-09-13T15:14:02.795719+0000","last_clean":"2024-09-13T15:14:02.795719+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:14:02.795719+0000","last_undegraded":"2024-09-13T15:14:02.795719+0000","last_fullsized":"2024-09-13T15:14:02.795719+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":628,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.796071+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:14:02.796071+0000","last_peered":"2024-09-13T15:14:02.796071+0000","last_clean":"2024-09-13T15:14:02.796071+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:14:02.796071+0000","last_undegraded":"2024-09-13T15:14:02.796071+0000","last_fullsized":"2024-09-13T15:14:02.796071+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"89'149","reported_seq":879,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295914+0000","last_change":"2024-09-13T15:13:03.104893+0000","last_active":"2024-09-13T15:13:57.295914+0000","last_peered":"2024-09-13T15:13:57.295914+0000","last_clean":"2024-09-13T15:13:57.295914+0000","last_became_active":"2024-09-13T15:12:38.933486+0000","last_became_peered":"2024-09-13T15:12:38.933486+0000","last_unstale":"2024-09-13T15:13:57.295914+0000","last_undegraded":"2024-09-13T15:13:57.295914+0000","last_fullsized":"2024-09-13T15:13:57.295914+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.104648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.104648+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:41.623486+0000","scrub_duration":0.01681821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":425,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296313+0000","last_change":"2024-09-13T15:08:46.074249+0000","last_active":"2024-09-13T15:13:57.296313+0000","last_peered":"2024-09-13T15:13:57.296313+0000","last_clean":"2024-09-13T15:13:57.296313+0000","last_became_active":"2024-09-13T15:08:46.073842+0000","last_became_peered":"2024-09-13T15:08:46.073842+0000","last_unstale":"2024-09-13T15:13:57.296313+0000","last_undegraded":"2024-09-13T15:13:57.296313+0000","last_fullsized":"2024-09-13T15:13:57.296313+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:44.854067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":530,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.286882+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:14:02.286882+0000","last_peered":"2024-09-13T15:14:02.286882+0000","last_clean":"2024-09-13T15:14:02.286882+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:14:02.286882+0000","last_undegraded":"2024-09-13T15:14:02.286882+0000","last_fullsized":"2024-09-13T15:14:02.286882+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":342,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295444+0000","last_change":"2024-09-13T15:08:46.075360+0000","last_active":"2024-09-13T15:13:57.295444+0000","last_peered":"2024-09-13T15:13:57.295444+0000","last_clean":"2024-09-13T15:13:57.295444+0000","last_became_active":"2024-09-13T15:08:46.075046+0000","last_became_peered":"2024-09-13T15:08:46.075046+0000","last_unstale":"2024-09-13T15:13:57.295444+0000","last_undegraded":"2024-09-13T15:13:57.295444+0000","last_fullsized":"2024-09-13T15:13:57.295444+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.044150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"89'149","reported_seq":923,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196225+0000","last_change":"2024-09-13T15:13:04.116692+0000","last_active":"2024-09-13T15:14:03.196225+0000","last_peered":"2024-09-13T15:14:03.196225+0000","last_clean":"2024-09-13T15:14:03.196225+0000","last_became_active":"2024-09-13T15:12:38.940054+0000","last_became_peered":"2024-09-13T15:12:38.940054+0000","last_unstale":"2024-09-13T15:14:03.196225+0000","last_undegraded":"2024-09-13T15:14:03.196225+0000","last_fullsized":"2024-09-13T15:14:03.196225+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.116616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.116616+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:25.864759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":375,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295836+0000","last_change":"2024-09-13T15:08:46.079918+0000","last_active":"2024-09-13T15:13:57.295836+0000","last_peered":"2024-09-13T15:13:57.295836+0000","last_clean":"2024-09-13T15:13:57.295836+0000","last_became_active":"2024-09-13T15:08:46.075051+0000","last_became_peered":"2024-09-13T15:08:46.075051+0000","last_unstale":"2024-09-13T15:13:57.295836+0000","last_undegraded":"2024-09-13T15:13:57.295836+0000","last_fullsized":"2024-09-13T15:13:57.295836+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:54.485416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1542,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795944+0000","last_change":"2024-09-13T15:08:59.341688+0000","last_active":"2024-09-13T15:14:02.795944+0000","last_peered":"2024-09-13T15:14:02.795944+0000","last_clean":"2024-09-13T15:14:02.795944+0000","last_became_active":"2024-09-13T15:08:59.340925+0000","last_became_peered":"2024-09-13T15:08:59.340925+0000","last_unstale":"2024-09-13T15:14:02.795944+0000","last_undegraded":"2024-09-13T15:14:02.795944+0000","last_fullsized":"2024-09-13T15:14:02.795944+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:22.966119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":470,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499817+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:14:02.499817+0000","last_peered":"2024-09-13T15:14:02.499817+0000","last_clean":"2024-09-13T15:14:02.499817+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:14:02.499817+0000","last_undegraded":"2024-09-13T15:14:02.499817+0000","last_fullsized":"2024-09-13T15:14:02.499817+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"89'149","reported_seq":927,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500081+0000","last_change":"2024-09-13T15:13:03.228430+0000","last_active":"2024-09-13T15:14:02.500081+0000","last_peered":"2024-09-13T15:14:02.500081+0000","last_clean":"2024-09-13T15:14:02.500081+0000","last_became_active":"2024-09-13T15:12:38.937552+0000","last_became_peered":"2024-09-13T15:12:38.937552+0000","last_unstale":"2024-09-13T15:14:02.500081+0000","last_undegraded":"2024-09-13T15:14:02.500081+0000","last_fullsized":"2024-09-13T15:14:02.500081+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.228290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.228290+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:30.456205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":882,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795753+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:14:02.795753+0000","last_peered":"2024-09-13T15:14:02.795753+0000","last_clean":"2024-09-13T15:14:02.795753+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:14:02.795753+0000","last_undegraded":"2024-09-13T15:14:02.795753+0000","last_fullsized":"2024-09-13T15:14:02.795753+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":93,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296541+0000","last_change":"2024-09-13T15:08:31.209459+0000","last_active":"2024-09-13T15:13:57.296541+0000","last_peered":"2024-09-13T15:13:57.296541+0000","last_clean":"2024-09-13T15:13:57.296541+0000","last_became_active":"2024-09-13T15:08:31.188811+0000","last_became_peered":"2024-09-13T15:08:31.188811+0000","last_unstale":"2024-09-13T15:13:57.296541+0000","last_undegraded":"2024-09-13T15:13:57.296541+0000","last_fullsized":"2024-09-13T15:13:57.296541+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.667223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":829,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499936+0000","last_change":"2024-09-13T15:08:31.197894+0000","last_active":"2024-09-13T15:14:02.499936+0000","last_peered":"2024-09-13T15:14:02.499936+0000","last_clean":"2024-09-13T15:14:02.499936+0000","last_became_active":"2024-09-13T15:08:31.197516+0000","last_became_peered":"2024-09-13T15:08:31.197516+0000","last_unstale":"2024-09-13T15:14:02.499936+0000","last_undegraded":"2024-09-13T15:14:02.499936+0000","last_fullsized":"2024-09-13T15:14:02.499936+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:42.650595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"89'149","reported_seq":879,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272105+0000","last_change":"2024-09-13T15:13:00.711640+0000","last_active":"2024-09-13T15:13:57.272105+0000","last_peered":"2024-09-13T15:13:57.272105+0000","last_clean":"2024-09-13T15:13:57.272105+0000","last_became_active":"2024-09-13T15:12:38.935375+0000","last_became_peered":"2024-09-13T15:12:38.935375+0000","last_unstale":"2024-09-13T15:13:57.272105+0000","last_undegraded":"2024-09-13T15:13:57.272105+0000","last_fullsized":"2024-09-13T15:13:57.272105+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.711407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.711407+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:41.333318+0000","scrub_duration":0.0091286909999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":935,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196396+0000","last_change":"2024-09-13T15:08:31.202795+0000","last_active":"2024-09-13T15:14:03.196396+0000","last_peered":"2024-09-13T15:14:03.196396+0000","last_clean":"2024-09-13T15:14:03.196396+0000","last_became_active":"2024-09-13T15:08:31.199963+0000","last_became_peered":"2024-09-13T15:08:31.199963+0000","last_unstale":"2024-09-13T15:14:03.196396+0000","last_undegraded":"2024-09-13T15:14:03.196396+0000","last_fullsized":"2024-09-13T15:14:03.196396+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:08.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1368,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.796117+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:14:02.796117+0000","last_peered":"2024-09-13T15:14:02.796117+0000","last_clean":"2024-09-13T15:14:02.796117+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:14:02.796117+0000","last_undegraded":"2024-09-13T15:14:02.796117+0000","last_fullsized":"2024-09-13T15:14:02.796117+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":639,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195541+0000","last_change":"2024-09-13T15:08:31.198729+0000","last_active":"2024-09-13T15:14:03.195541+0000","last_peered":"2024-09-13T15:14:03.195541+0000","last_clean":"2024-09-13T15:14:03.195541+0000","last_became_active":"2024-09-13T15:08:31.198423+0000","last_became_peered":"2024-09-13T15:08:31.198423+0000","last_unstale":"2024-09-13T15:14:03.195541+0000","last_undegraded":"2024-09-13T15:14:03.195541+0000","last_fullsized":"2024-09-13T15:14:03.195541+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:58.362919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"89'149","reported_seq":920,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195629+0000","last_change":"2024-09-13T15:13:03.079534+0000","last_active":"2024-09-13T15:14:03.195629+0000","last_peered":"2024-09-13T15:14:03.195629+0000","last_clean":"2024-09-13T15:14:03.195629+0000","last_became_active":"2024-09-13T15:12:38.941094+0000","last_became_peered":"2024-09-13T15:12:38.941094+0000","last_unstale":"2024-09-13T15:14:03.195629+0000","last_undegraded":"2024-09-13T15:14:03.195629+0000","last_fullsized":"2024-09-13T15:14:03.195629+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.079258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.079258+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:10.140799+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":756,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500211+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:14:02.500211+0000","last_peered":"2024-09-13T15:14:02.500211+0000","last_clean":"2024-09-13T15:14:02.500211+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:14:02.500211+0000","last_undegraded":"2024-09-13T15:14:02.500211+0000","last_fullsized":"2024-09-13T15:14:02.500211+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":932,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296866+0000","last_change":"2024-09-13T15:08:59.333844+0000","last_active":"2024-09-13T15:13:57.296866+0000","last_peered":"2024-09-13T15:13:57.296866+0000","last_clean":"2024-09-13T15:13:57.296866+0000","last_became_active":"2024-09-13T15:08:59.332369+0000","last_became_peered":"2024-09-13T15:08:59.332369+0000","last_unstale":"2024-09-13T15:13:57.296866+0000","last_undegraded":"2024-09-13T15:13:57.296866+0000","last_fullsized":"2024-09-13T15:13:57.296866+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:36.490753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":526,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.286163+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:14:02.286163+0000","last_peered":"2024-09-13T15:14:02.286163+0000","last_clean":"2024-09-13T15:14:02.286163+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:14:02.286163+0000","last_undegraded":"2024-09-13T15:14:02.286163+0000","last_fullsized":"2024-09-13T15:14:02.286163+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":509,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272000+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:13:57.272000+0000","last_peered":"2024-09-13T15:13:57.272000+0000","last_clean":"2024-09-13T15:13:57.272000+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:13:57.272000+0000","last_undegraded":"2024-09-13T15:13:57.272000+0000","last_fullsized":"2024-09-13T15:13:57.272000+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":641,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272002+0000","last_change":"2024-09-13T15:08:31.193477+0000","last_active":"2024-09-13T15:13:57.272002+0000","last_peered":"2024-09-13T15:13:57.272002+0000","last_clean":"2024-09-13T15:13:57.272002+0000","last_became_active":"2024-09-13T15:08:31.181861+0000","last_became_peered":"2024-09-13T15:08:31.181861+0000","last_unstale":"2024-09-13T15:13:57.272002+0000","last_undegraded":"2024-09-13T15:13:57.272002+0000","last_fullsized":"2024-09-13T15:13:57.272002+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:02.424164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":33292,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":893,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795855+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:14:02.795855+0000","last_peered":"2024-09-13T15:14:02.795855+0000","last_clean":"2024-09-13T15:14:02.795855+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:14:02.795855+0000","last_undegraded":"2024-09-13T15:14:02.795855+0000","last_fullsized":"2024-09-13T15:14:02.795855+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":81,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296587+0000","last_change":"2024-09-13T15:08:31.204593+0000","last_active":"2024-09-13T15:13:57.296587+0000","last_peered":"2024-09-13T15:13:57.296587+0000","last_clean":"2024-09-13T15:13:57.296587+0000","last_became_active":"2024-09-13T15:08:31.198883+0000","last_became_peered":"2024-09-13T15:08:31.198883+0000","last_unstale":"2024-09-13T15:13:57.296587+0000","last_undegraded":"2024-09-13T15:13:57.296587+0000","last_fullsized":"2024-09-13T15:13:57.296587+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:08.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"89'149","reported_seq":923,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195697+0000","last_change":"2024-09-13T15:13:00.998330+0000","last_active":"2024-09-13T15:14:03.195697+0000","last_peered":"2024-09-13T15:14:03.195697+0000","last_clean":"2024-09-13T15:14:03.195697+0000","last_became_active":"2024-09-13T15:12:50.190273+0000","last_became_peered":"2024-09-13T15:12:50.190273+0000","last_unstale":"2024-09-13T15:14:03.195697+0000","last_undegraded":"2024-09-13T15:14:03.195697+0000","last_fullsized":"2024-09-13T15:14:03.195697+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.998266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.998266+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:06.347067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":398,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296130+0000","last_change":"2024-09-13T15:08:31.188790+0000","last_active":"2024-09-13T15:13:57.296130+0000","last_peered":"2024-09-13T15:13:57.296130+0000","last_clean":"2024-09-13T15:13:57.296130+0000","last_became_active":"2024-09-13T15:08:31.188441+0000","last_became_peered":"2024-09-13T15:08:31.188441+0000","last_unstale":"2024-09-13T15:13:57.296130+0000","last_undegraded":"2024-09-13T15:13:57.296130+0000","last_fullsized":"2024-09-13T15:13:57.296130+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:52.934163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":881,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195773+0000","last_change":"2024-09-13T15:08:31.197112+0000","last_active":"2024-09-13T15:14:03.195773+0000","last_peered":"2024-09-13T15:14:03.195773+0000","last_clean":"2024-09-13T15:14:03.195773+0000","last_became_active":"2024-09-13T15:08:31.196687+0000","last_became_peered":"2024-09-13T15:08:31.196687+0000","last_unstale":"2024-09-13T15:14:03.195773+0000","last_undegraded":"2024-09-13T15:14:03.195773+0000","last_fullsized":"2024-09-13T15:14:03.195773+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:47.532912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":847,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500265+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:14:02.500265+0000","last_peered":"2024-09-13T15:14:02.500265+0000","last_clean":"2024-09-13T15:14:02.500265+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:14:02.500265+0000","last_undegraded":"2024-09-13T15:14:02.500265+0000","last_fullsized":"2024-09-13T15:14:02.500265+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":911,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195576+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:14:03.195576+0000","last_peered":"2024-09-13T15:14:03.195576+0000","last_clean":"2024-09-13T15:14:03.195576+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:14:03.195576+0000","last_undegraded":"2024-09-13T15:14:03.195576+0000","last_fullsized":"2024-09-13T15:14:03.195576+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":304,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296558+0000","last_change":"2024-09-13T15:08:46.075112+0000","last_active":"2024-09-13T15:13:57.296558+0000","last_peered":"2024-09-13T15:13:57.296558+0000","last_clean":"2024-09-13T15:13:57.296558+0000","last_became_active":"2024-09-13T15:08:46.074818+0000","last_became_peered":"2024-09-13T15:08:46.074818+0000","last_unstale":"2024-09-13T15:13:57.296558+0000","last_undegraded":"2024-09-13T15:13:57.296558+0000","last_fullsized":"2024-09-13T15:13:57.296558+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:56.288263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":875,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195781+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:14:03.195781+0000","last_peered":"2024-09-13T15:14:03.195781+0000","last_clean":"2024-09-13T15:14:03.195781+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:14:03.195781+0000","last_undegraded":"2024-09-13T15:14:03.195781+0000","last_fullsized":"2024-09-13T15:14:03.195781+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":845,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500151+0000","last_change":"2024-09-13T15:08:31.195215+0000","last_active":"2024-09-13T15:14:02.500151+0000","last_peered":"2024-09-13T15:14:02.500151+0000","last_clean":"2024-09-13T15:14:02.500151+0000","last_became_active":"2024-09-13T15:08:31.194526+0000","last_became_peered":"2024-09-13T15:08:31.194526+0000","last_unstale":"2024-09-13T15:14:02.500151+0000","last_undegraded":"2024-09-13T15:14:02.500151+0000","last_fullsized":"2024-09-13T15:14:02.500151+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:56.358764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":847,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795606+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:14:02.795606+0000","last_peered":"2024-09-13T15:14:02.795606+0000","last_clean":"2024-09-13T15:14:02.795606+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:14:02.795606+0000","last_undegraded":"2024-09-13T15:14:02.795606+0000","last_fullsized":"2024-09-13T15:14:02.795606+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":712,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195977+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:14:03.195977+0000","last_peered":"2024-09-13T15:14:03.195977+0000","last_clean":"2024-09-13T15:14:03.195977+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:14:03.195977+0000","last_undegraded":"2024-09-13T15:14:03.195977+0000","last_fullsized":"2024-09-13T15:14:03.195977+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":504,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271833+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:13:57.271833+0000","last_peered":"2024-09-13T15:13:57.271833+0000","last_clean":"2024-09-13T15:13:57.271833+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:13:57.271833+0000","last_undegraded":"2024-09-13T15:13:57.271833+0000","last_fullsized":"2024-09-13T15:13:57.271833+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":463,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195981+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:14:03.195981+0000","last_peered":"2024-09-13T15:14:03.195981+0000","last_clean":"2024-09-13T15:14:03.195981+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:14:03.195981+0000","last_undegraded":"2024-09-13T15:14:03.195981+0000","last_fullsized":"2024-09-13T15:14:03.195981+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":832,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499987+0000","last_change":"2024-09-13T15:08:31.194912+0000","last_active":"2024-09-13T15:14:02.499987+0000","last_peered":"2024-09-13T15:14:02.499987+0000","last_clean":"2024-09-13T15:14:02.499987+0000","last_became_active":"2024-09-13T15:08:31.194553+0000","last_became_peered":"2024-09-13T15:08:31.194553+0000","last_unstale":"2024-09-13T15:14:02.499987+0000","last_undegraded":"2024-09-13T15:14:02.499987+0000","last_fullsized":"2024-09-13T15:14:02.499987+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:17.457690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":514,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271953+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:13:57.271953+0000","last_peered":"2024-09-13T15:13:57.271953+0000","last_clean":"2024-09-13T15:13:57.271953+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:13:57.271953+0000","last_undegraded":"2024-09-13T15:13:57.271953+0000","last_fullsized":"2024-09-13T15:13:57.271953+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":342,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295648+0000","last_change":"2024-09-13T15:08:46.073232+0000","last_active":"2024-09-13T15:13:57.295648+0000","last_peered":"2024-09-13T15:13:57.295648+0000","last_clean":"2024-09-13T15:13:57.295648+0000","last_became_active":"2024-09-13T15:08:46.072807+0000","last_became_peered":"2024-09-13T15:08:46.072807+0000","last_unstale":"2024-09-13T15:13:57.295648+0000","last_undegraded":"2024-09-13T15:13:57.295648+0000","last_fullsized":"2024-09-13T15:13:57.295648+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:11.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":502,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271744+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:13:57.271744+0000","last_peered":"2024-09-13T15:13:57.271744+0000","last_clean":"2024-09-13T15:13:57.271744+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:13:57.271744+0000","last_undegraded":"2024-09-13T15:13:57.271744+0000","last_fullsized":"2024-09-13T15:13:57.271744+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":370,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295988+0000","last_change":"2024-09-13T15:08:46.079600+0000","last_active":"2024-09-13T15:13:57.295988+0000","last_peered":"2024-09-13T15:13:57.295988+0000","last_clean":"2024-09-13T15:13:57.295988+0000","last_became_active":"2024-09-13T15:08:46.075458+0000","last_became_peered":"2024-09-13T15:08:46.075458+0000","last_unstale":"2024-09-13T15:13:57.295988+0000","last_undegraded":"2024-09-13T15:13:57.295988+0000","last_fullsized":"2024-09-13T15:13:57.295988+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:41.707452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":885,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196105+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:14:03.196105+0000","last_peered":"2024-09-13T15:14:03.196105+0000","last_clean":"2024-09-13T15:14:03.196105+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:14:03.196105+0000","last_undegraded":"2024-09-13T15:14:03.196105+0000","last_fullsized":"2024-09-13T15:14:03.196105+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":908,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196311+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:14:03.196311+0000","last_peered":"2024-09-13T15:14:03.196311+0000","last_clean":"2024-09-13T15:14:03.196311+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:14:03.196311+0000","last_undegraded":"2024-09-13T15:14:03.196311+0000","last_fullsized":"2024-09-13T15:14:03.196311+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":535,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271498+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:13:57.271498+0000","last_peered":"2024-09-13T15:13:57.271498+0000","last_clean":"2024-09-13T15:13:57.271498+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:13:57.271498+0000","last_undegraded":"2024-09-13T15:13:57.271498+0000","last_fullsized":"2024-09-13T15:13:57.271498+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":872,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500149+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:14:02.500149+0000","last_peered":"2024-09-13T15:14:02.500149+0000","last_clean":"2024-09-13T15:14:02.500149+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:14:02.500149+0000","last_undegraded":"2024-09-13T15:14:02.500149+0000","last_fullsized":"2024-09-13T15:14:02.500149+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":36578,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234561,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529036,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":522494,"kb_avail":93216116,"statfs":{"total":95995035648,"available":95453302784,"internally_reserved":0,"allocated":6684672,"data_stored":3973242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":535034315},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496197,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727872,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":721022,"kb_avail":93017280,"statfs":{"total":95995035648,"available":95249694720,"internally_reserved":0,"allocated":7000064,"data_stored":4300522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":738326983},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757832,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876288,"kb_used_data":6772,"kb_used_omap":1,"kb_used_meta":869502,"kb_avail":92868864,"statfs":{"total":95995035648,"available":95097716736,"internally_reserved":0,"allocated":6934528,"data_stored":4231687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890370503},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849800,"kb_used_data":6524,"kb_used_omap":1,"kb_used_meta":843262,"kb_avail":92895352,"statfs":{"total":95995035648,"available":95124840448,"internally_reserved":0,"allocated":6680576,"data_stored":3973481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863500742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468357,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989628,"kb_used_data":11184,"kb_used_omap":4,"kb_used_meta":978427,"kb_avail":92755524,"statfs":{"total":95995035648,"available":94981656576,"internally_reserved":0,"allocated":11452416,"data_stored":8753375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":1001909412},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273256,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433412,"kb_used_data":10872,"kb_used_omap":12,"kb_used_meta":422515,"kb_avail":93311740,"statfs":{"total":95995035648,"available":95551221760,"internally_reserved":0,"allocated":11132928,"data_stored":8426243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":432656256},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795409,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894776,"kb_used_data":6508,"kb_used_omap":7,"kb_used_meta":888248,"kb_avail":92850376,"statfs":{"total":95995035648,"available":95078785024,"internally_reserved":0,"allocated":6664192,"data_stored":3968632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":909566578},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502191,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597820,"kb_used_data":10928,"kb_used_omap":10,"kb_used_meta":586869,"kb_avail":93147332,"statfs":{"total":95995035648,"available":95382867968,"internally_reserved":0,"allocated":11190272,"data_stored":8491201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":600954559},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:14:15.047 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:14:15.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T15:14:15.048 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-13T15:14:15.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:14:15.582 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:15 smithi154 ceph-mon[102314]: from='client.46484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:15.582 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:15 smithi154 ceph-mon[107038]: from='client.46484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:15.642 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:15 smithi172 ceph-mon[100696]: from='client.46484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:16.612 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:16 smithi154 ceph-mon[102314]: pgmap v1618: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:16.612 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:16 smithi154 ceph-mon[107038]: pgmap v1618: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:16.654 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:16 smithi172 ceph-mon[100696]: pgmap v1618: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:17.394 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:17.583 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:17 smithi154 ceph-mon[102314]: pgmap v1619: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:17.583 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:17 smithi154 ceph-mon[107038]: pgmap v1619: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:17 smithi172 ceph-mon[100696]: pgmap v1619: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:18.448 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:14:18.448 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:14:18.448 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T15:14:18.448 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:14:19.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-13T15:14:19.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:19 smithi154 ceph-mon[102314]: pgmap v1620: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:19.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:19 smithi154 ceph-mon[107038]: pgmap v1620: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:19 smithi172 ceph-mon[100696]: pgmap v1620: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:20.718 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:20.926 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:14:20.927 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:14:21.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:14:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:14:21.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:14:21.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:14:21.222 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:14:21.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:14:21.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:14:22.097 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:21 smithi154 ceph-mon[102314]: from='client.46490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:22.097 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:21 smithi154 ceph-mon[102314]: pgmap v1621: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:22.097 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:21 smithi154 ceph-mon[107038]: from='client.46490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:22.098 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:21 smithi154 ceph-mon[107038]: pgmap v1621: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:22.098 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:14:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:14:21] "GET /metrics HTTP/1.1" 200 34226 "" "Prometheus/2.51.0" 2024-09-13T15:14:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:21 smithi172 ceph-mon[100696]: from='client.46490 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:14:22.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:21 smithi172 ceph-mon[100696]: pgmap v1621: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:22.823 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2024-09-13T15:14:19.278774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":36635,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4162,"ondisk_log_size":4162,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5898632,"kb_used_data":66152,"kb_used_omap":40,"kb_used_meta":5832343,"kb_avail":744062584,"statfs":{"total":767960285184,"available":761920086016,"internally_reserved":0,"allocated":67739648,"data_stored":46118383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":5972319348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002887"},"pg_stats":[{"pgid":"1.11","version":"89'149","reported_seq":873,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499782+0000","last_change":"2024-09-13T15:13:01.179220+0000","last_active":"2024-09-13T15:14:02.499782+0000","last_peered":"2024-09-13T15:14:02.499782+0000","last_clean":"2024-09-13T15:14:02.499782+0000","last_became_active":"2024-09-13T15:12:50.169695+0000","last_became_peered":"2024-09-13T15:12:50.169695+0000","last_unstale":"2024-09-13T15:14:02.499782+0000","last_undegraded":"2024-09-13T15:14:02.499782+0000","last_fullsized":"2024-09-13T15:14:02.499782+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.179154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.179154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:54:04.917248+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":224,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195793+0000","last_change":"2024-09-13T15:08:31.190995+0000","last_active":"2024-09-13T15:14:03.195793+0000","last_peered":"2024-09-13T15:14:03.195793+0000","last_clean":"2024-09-13T15:14:03.195793+0000","last_became_active":"2024-09-13T15:08:31.190610+0000","last_became_peered":"2024-09-13T15:08:31.190610+0000","last_unstale":"2024-09-13T15:14:03.195793+0000","last_undegraded":"2024-09-13T15:14:03.195793+0000","last_fullsized":"2024-09-13T15:14:03.195793+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:42.124757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":81,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295970+0000","last_change":"2024-09-13T15:08:31.191585+0000","last_active":"2024-09-13T15:13:57.295970+0000","last_peered":"2024-09-13T15:13:57.295970+0000","last_clean":"2024-09-13T15:13:57.295970+0000","last_became_active":"2024-09-13T15:08:31.184700+0000","last_became_peered":"2024-09-13T15:08:31.184700+0000","last_unstale":"2024-09-13T15:13:57.295970+0000","last_undegraded":"2024-09-13T15:13:57.295970+0000","last_fullsized":"2024-09-13T15:13:57.295970+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:14.376834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"89'149","reported_seq":839,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295636+0000","last_change":"2024-09-13T15:13:09.203525+0000","last_active":"2024-09-13T15:13:57.295636+0000","last_peered":"2024-09-13T15:13:57.295636+0000","last_clean":"2024-09-13T15:13:57.295636+0000","last_became_active":"2024-09-13T15:12:50.159713+0000","last_became_peered":"2024-09-13T15:12:50.159713+0000","last_unstale":"2024-09-13T15:13:57.295636+0000","last_undegraded":"2024-09-13T15:13:57.295636+0000","last_fullsized":"2024-09-13T15:13:57.295636+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:09.203387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:09.203387+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:07.004095+0000","scrub_duration":0.002438127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":358,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295907+0000","last_change":"2024-09-13T15:08:46.080613+0000","last_active":"2024-09-13T15:13:57.295907+0000","last_peered":"2024-09-13T15:13:57.295907+0000","last_clean":"2024-09-13T15:13:57.295907+0000","last_became_active":"2024-09-13T15:08:46.080320+0000","last_became_peered":"2024-09-13T15:08:46.080320+0000","last_unstale":"2024-09-13T15:13:57.295907+0000","last_undegraded":"2024-09-13T15:13:57.295907+0000","last_fullsized":"2024-09-13T15:13:57.295907+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:09.205609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":839,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499913+0000","last_change":"2024-09-13T15:08:31.196449+0000","last_active":"2024-09-13T15:14:02.499913+0000","last_peered":"2024-09-13T15:14:02.499913+0000","last_clean":"2024-09-13T15:14:02.499913+0000","last_became_active":"2024-09-13T15:08:31.196143+0000","last_became_peered":"2024-09-13T15:08:31.196143+0000","last_unstale":"2024-09-13T15:14:02.499913+0000","last_undegraded":"2024-09-13T15:14:02.499913+0000","last_fullsized":"2024-09-13T15:14:02.499913+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:53.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"89'149","reported_seq":925,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795677+0000","last_change":"2024-09-13T15:13:03.099175+0000","last_active":"2024-09-13T15:14:02.795677+0000","last_peered":"2024-09-13T15:14:02.795677+0000","last_clean":"2024-09-13T15:14:02.795677+0000","last_became_active":"2024-09-13T15:12:50.196974+0000","last_became_peered":"2024-09-13T15:12:50.196974+0000","last_unstale":"2024-09-13T15:14:02.795677+0000","last_undegraded":"2024-09-13T15:14:02.795677+0000","last_fullsized":"2024-09-13T15:14:02.795677+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.098941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.098941+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:00:19.670702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":843,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500214+0000","last_change":"2024-09-13T15:08:31.195499+0000","last_active":"2024-09-13T15:14:02.500214+0000","last_peered":"2024-09-13T15:14:02.500214+0000","last_clean":"2024-09-13T15:14:02.500214+0000","last_became_active":"2024-09-13T15:08:31.194928+0000","last_became_peered":"2024-09-13T15:08:31.194928+0000","last_unstale":"2024-09-13T15:14:02.500214+0000","last_undegraded":"2024-09-13T15:14:02.500214+0000","last_fullsized":"2024-09-13T15:14:02.500214+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.112396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":75,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295668+0000","last_change":"2024-09-13T15:08:31.185658+0000","last_active":"2024-09-13T15:13:57.295668+0000","last_peered":"2024-09-13T15:13:57.295668+0000","last_clean":"2024-09-13T15:13:57.295668+0000","last_became_active":"2024-09-13T15:08:31.185202+0000","last_became_peered":"2024-09-13T15:08:31.185202+0000","last_unstale":"2024-09-13T15:13:57.295668+0000","last_undegraded":"2024-09-13T15:13:57.295668+0000","last_fullsized":"2024-09-13T15:13:57.295668+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:04.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"89'149","reported_seq":839,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296210+0000","last_change":"2024-09-13T15:13:11.154190+0000","last_active":"2024-09-13T15:13:57.296210+0000","last_peered":"2024-09-13T15:13:57.296210+0000","last_clean":"2024-09-13T15:13:57.296210+0000","last_became_active":"2024-09-13T15:12:50.159857+0000","last_became_peered":"2024-09-13T15:12:50.159857+0000","last_unstale":"2024-09-13T15:13:57.296210+0000","last_undegraded":"2024-09-13T15:13:57.296210+0000","last_fullsized":"2024-09-13T15:13:57.296210+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.153852+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.153852+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:54.235242+0000","scrub_duration":0.0043281400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":345,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196231+0000","last_change":"2024-09-13T15:08:31.199682+0000","last_active":"2024-09-13T15:14:03.196231+0000","last_peered":"2024-09-13T15:14:03.196231+0000","last_clean":"2024-09-13T15:14:03.196231+0000","last_became_active":"2024-09-13T15:08:31.199110+0000","last_became_peered":"2024-09-13T15:08:31.199110+0000","last_unstale":"2024-09-13T15:14:03.196231+0000","last_undegraded":"2024-09-13T15:14:03.196231+0000","last_fullsized":"2024-09-13T15:14:03.196231+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:30.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":449,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499720+0000","last_change":"2024-09-13T15:04:05.021292+0000","last_active":"2024-09-13T15:14:02.499720+0000","last_peered":"2024-09-13T15:14:02.499720+0000","last_clean":"2024-09-13T15:14:02.499720+0000","last_became_active":"2024-09-13T15:04:05.011146+0000","last_became_peered":"2024-09-13T15:04:05.011146+0000","last_unstale":"2024-09-13T15:14:02.499720+0000","last_undegraded":"2024-09-13T15:14:02.499720+0000","last_fullsized":"2024-09-13T15:14:02.499720+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:00.137694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"89'149","reported_seq":840,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296487+0000","last_change":"2024-09-13T15:13:15.494950+0000","last_active":"2024-09-13T15:13:57.296487+0000","last_peered":"2024-09-13T15:13:57.296487+0000","last_clean":"2024-09-13T15:13:57.296487+0000","last_became_active":"2024-09-13T15:12:50.163288+0000","last_became_peered":"2024-09-13T15:12:50.163288+0000","last_unstale":"2024-09-13T15:13:57.296487+0000","last_undegraded":"2024-09-13T15:13:57.296487+0000","last_fullsized":"2024-09-13T15:13:57.296487+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:15.494798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:15.494798+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:34.790443+0000","scrub_duration":0.0082230789999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296864+0000","last_change":"2024-09-13T15:08:31.351504+0000","last_active":"2024-09-13T15:13:57.296864+0000","last_peered":"2024-09-13T15:13:57.296864+0000","last_clean":"2024-09-13T15:13:57.296864+0000","last_became_active":"2024-09-13T15:08:31.320950+0000","last_became_peered":"2024-09-13T15:08:31.320950+0000","last_unstale":"2024-09-13T15:13:57.296864+0000","last_undegraded":"2024-09-13T15:13:57.296864+0000","last_fullsized":"2024-09-13T15:13:57.296864+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:38.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":404,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295807+0000","last_change":"2024-09-13T15:08:46.078296+0000","last_active":"2024-09-13T15:13:57.295807+0000","last_peered":"2024-09-13T15:13:57.295807+0000","last_clean":"2024-09-13T15:13:57.295807+0000","last_became_active":"2024-09-13T15:08:46.078037+0000","last_became_peered":"2024-09-13T15:08:46.078037+0000","last_unstale":"2024-09-13T15:13:57.295807+0000","last_undegraded":"2024-09-13T15:13:57.295807+0000","last_fullsized":"2024-09-13T15:13:57.295807+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:49.959361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"89'149","reported_seq":932,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195672+0000","last_change":"2024-09-13T15:13:02.045213+0000","last_active":"2024-09-13T15:14:03.195672+0000","last_peered":"2024-09-13T15:14:03.195672+0000","last_clean":"2024-09-13T15:14:03.195672+0000","last_became_active":"2024-09-13T15:12:50.170281+0000","last_became_peered":"2024-09-13T15:12:50.170281+0000","last_unstale":"2024-09-13T15:14:03.195672+0000","last_undegraded":"2024-09-13T15:14:03.195672+0000","last_fullsized":"2024-09-13T15:14:03.195672+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:02.045141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:02.045141+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:03.286894+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296377+0000","last_change":"2024-09-13T15:08:31.201939+0000","last_active":"2024-09-13T15:13:57.296377+0000","last_peered":"2024-09-13T15:13:57.296377+0000","last_clean":"2024-09-13T15:13:57.296377+0000","last_became_active":"2024-09-13T15:08:31.191089+0000","last_became_peered":"2024-09-13T15:08:31.191089+0000","last_unstale":"2024-09-13T15:13:57.296377+0000","last_undegraded":"2024-09-13T15:13:57.296377+0000","last_fullsized":"2024-09-13T15:13:57.296377+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:06.249945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":568,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271641+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:13:57.271641+0000","last_peered":"2024-09-13T15:13:57.271641+0000","last_clean":"2024-09-13T15:13:57.271641+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:13:57.271641+0000","last_undegraded":"2024-09-13T15:13:57.271641+0000","last_fullsized":"2024-09-13T15:13:57.271641+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"89'149","reported_seq":899,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296522+0000","last_change":"2024-09-13T15:13:06.128357+0000","last_active":"2024-09-13T15:13:57.296522+0000","last_peered":"2024-09-13T15:13:57.296522+0000","last_clean":"2024-09-13T15:13:57.296522+0000","last_became_active":"2024-09-13T15:12:50.160188+0000","last_became_peered":"2024-09-13T15:12:50.160188+0000","last_unstale":"2024-09-13T15:13:57.296522+0000","last_undegraded":"2024-09-13T15:13:57.296522+0000","last_fullsized":"2024-09-13T15:13:57.296522+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:06.128281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:06.128281+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:11.473186+0000","scrub_duration":0.0011969070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":79,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296728+0000","last_change":"2024-09-13T15:08:31.199736+0000","last_active":"2024-09-13T15:13:57.296728+0000","last_peered":"2024-09-13T15:13:57.296728+0000","last_clean":"2024-09-13T15:13:57.296728+0000","last_became_active":"2024-09-13T15:08:31.186018+0000","last_became_peered":"2024-09-13T15:08:31.186018+0000","last_unstale":"2024-09-13T15:13:57.296728+0000","last_undegraded":"2024-09-13T15:13:57.296728+0000","last_fullsized":"2024-09-13T15:13:57.296728+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:26.855187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":580,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795646+0000","last_change":"2024-09-13T15:08:31.193478+0000","last_active":"2024-09-13T15:14:02.795646+0000","last_peered":"2024-09-13T15:14:02.795646+0000","last_clean":"2024-09-13T15:14:02.795646+0000","last_became_active":"2024-09-13T15:08:31.193038+0000","last_became_peered":"2024-09-13T15:08:31.193038+0000","last_unstale":"2024-09-13T15:14:02.795646+0000","last_undegraded":"2024-09-13T15:14:02.795646+0000","last_fullsized":"2024-09-13T15:14:02.795646+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:05.600387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"89'149","reported_seq":908,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272101+0000","last_change":"2024-09-13T15:13:01.670578+0000","last_active":"2024-09-13T15:13:57.272101+0000","last_peered":"2024-09-13T15:13:57.272101+0000","last_clean":"2024-09-13T15:13:57.272101+0000","last_became_active":"2024-09-13T15:12:50.166107+0000","last_became_peered":"2024-09-13T15:12:50.166107+0000","last_unstale":"2024-09-13T15:13:57.272101+0000","last_undegraded":"2024-09-13T15:13:57.272101+0000","last_fullsized":"2024-09-13T15:13:57.272101+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.670400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.670400+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:33:19.151283+0000","scrub_duration":0.0070424579999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":375,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295748+0000","last_change":"2024-09-13T15:08:46.073352+0000","last_active":"2024-09-13T15:13:57.295748+0000","last_peered":"2024-09-13T15:13:57.295748+0000","last_clean":"2024-09-13T15:13:57.295748+0000","last_became_active":"2024-09-13T15:08:46.073048+0000","last_became_peered":"2024-09-13T15:08:46.073048+0000","last_unstale":"2024-09-13T15:13:57.295748+0000","last_undegraded":"2024-09-13T15:13:57.295748+0000","last_fullsized":"2024-09-13T15:13:57.295748+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:57.277685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":751,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499828+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:14:02.499828+0000","last_peered":"2024-09-13T15:14:02.499828+0000","last_clean":"2024-09-13T15:14:02.499828+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:14:02.499828+0000","last_undegraded":"2024-09-13T15:14:02.499828+0000","last_fullsized":"2024-09-13T15:14:02.499828+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"89'149","reported_seq":880,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295542+0000","last_change":"2024-09-13T15:10:26.504570+0000","last_active":"2024-09-13T15:13:57.295542+0000","last_peered":"2024-09-13T15:13:57.295542+0000","last_clean":"2024-09-13T15:13:57.295542+0000","last_became_active":"2024-09-13T15:10:19.782654+0000","last_became_peered":"2024-09-13T15:10:19.782654+0000","last_unstale":"2024-09-13T15:13:57.295542+0000","last_undegraded":"2024-09-13T15:13:57.295542+0000","last_fullsized":"2024-09-13T15:13:57.295542+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:10:26.504369+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:57.248804+0000","scrub_duration":0.013934767000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":879,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195713+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:14:03.195713+0000","last_peered":"2024-09-13T15:14:03.195713+0000","last_clean":"2024-09-13T15:14:03.195713+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:14:03.195713+0000","last_undegraded":"2024-09-13T15:14:03.195713+0000","last_fullsized":"2024-09-13T15:14:03.195713+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":510,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271822+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:13:57.271822+0000","last_peered":"2024-09-13T15:13:57.271822+0000","last_clean":"2024-09-13T15:13:57.271822+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:13:57.271822+0000","last_undegraded":"2024-09-13T15:13:57.271822+0000","last_fullsized":"2024-09-13T15:13:57.271822+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"89'149","reported_seq":858,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295154+0000","last_change":"2024-09-13T15:10:20.069094+0000","last_active":"2024-09-13T15:13:57.295154+0000","last_peered":"2024-09-13T15:13:57.295154+0000","last_clean":"2024-09-13T15:13:57.295154+0000","last_became_active":"2024-09-13T15:10:19.782890+0000","last_became_peered":"2024-09-13T15:10:19.782890+0000","last_unstale":"2024-09-13T15:13:57.295154+0000","last_undegraded":"2024-09-13T15:13:57.295154+0000","last_fullsized":"2024-09-13T15:13:57.295154+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:20.068973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:20.068973+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:03:30.570922+0000","scrub_duration":0.011313818,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":308,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271719+0000","last_change":"2024-09-13T15:04:05.013503+0000","last_active":"2024-09-13T15:13:57.271719+0000","last_peered":"2024-09-13T15:13:57.271719+0000","last_clean":"2024-09-13T15:13:57.271719+0000","last_became_active":"2024-09-13T15:04:05.013204+0000","last_became_peered":"2024-09-13T15:04:05.013204+0000","last_unstale":"2024-09-13T15:13:57.271719+0000","last_undegraded":"2024-09-13T15:13:57.271719+0000","last_fullsized":"2024-09-13T15:13:57.271719+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:17.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":504,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272111+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:13:57.272111+0000","last_peered":"2024-09-13T15:13:57.272111+0000","last_clean":"2024-09-13T15:13:57.272111+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:13:57.272111+0000","last_undegraded":"2024-09-13T15:13:57.272111+0000","last_fullsized":"2024-09-13T15:13:57.272111+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"89'149","reported_seq":925,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795868+0000","last_change":"2024-09-13T15:13:02.125234+0000","last_active":"2024-09-13T15:14:02.795868+0000","last_peered":"2024-09-13T15:14:02.795868+0000","last_clean":"2024-09-13T15:14:02.795868+0000","last_became_active":"2024-09-13T15:12:50.176602+0000","last_became_peered":"2024-09-13T15:12:50.176602+0000","last_unstale":"2024-09-13T15:14:02.795868+0000","last_undegraded":"2024-09-13T15:14:02.795868+0000","last_fullsized":"2024-09-13T15:14:02.795868+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:02.125166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:02.125166+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:32.682602+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":529,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271697+0000","last_change":"2024-09-13T15:08:31.184973+0000","last_active":"2024-09-13T15:13:57.271697+0000","last_peered":"2024-09-13T15:13:57.271697+0000","last_clean":"2024-09-13T15:13:57.271697+0000","last_became_active":"2024-09-13T15:08:31.184600+0000","last_became_peered":"2024-09-13T15:08:31.184600+0000","last_unstale":"2024-09-13T15:13:57.271697+0000","last_undegraded":"2024-09-13T15:13:57.271697+0000","last_fullsized":"2024-09-13T15:13:57.271697+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:09.452895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":537,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272207+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:13:57.272207+0000","last_peered":"2024-09-13T15:13:57.272207+0000","last_clean":"2024-09-13T15:13:57.272207+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:13:57.272207+0000","last_undegraded":"2024-09-13T15:13:57.272207+0000","last_fullsized":"2024-09-13T15:13:57.272207+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"89'149","reported_seq":858,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296131+0000","last_change":"2024-09-13T15:10:22.099331+0000","last_active":"2024-09-13T15:13:57.296131+0000","last_peered":"2024-09-13T15:13:57.296131+0000","last_clean":"2024-09-13T15:13:57.296131+0000","last_became_active":"2024-09-13T15:10:19.785103+0000","last_became_peered":"2024-09-13T15:10:19.785103+0000","last_unstale":"2024-09-13T15:13:57.296131+0000","last_undegraded":"2024-09-13T15:13:57.296131+0000","last_fullsized":"2024-09-13T15:13:57.296131+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:10:22.099204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:10:22.099204+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:34.804057+0000","scrub_duration":0.0069277289999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":345,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195427+0000","last_change":"2024-09-13T15:08:31.198602+0000","last_active":"2024-09-13T15:14:03.195427+0000","last_peered":"2024-09-13T15:14:03.195427+0000","last_clean":"2024-09-13T15:14:03.195427+0000","last_became_active":"2024-09-13T15:08:31.198289+0000","last_became_peered":"2024-09-13T15:08:31.198289+0000","last_unstale":"2024-09-13T15:14:03.195427+0000","last_undegraded":"2024-09-13T15:14:03.195427+0000","last_fullsized":"2024-09-13T15:14:03.195427+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:36.803833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":624,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195962+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:14:03.195962+0000","last_peered":"2024-09-13T15:14:03.195962+0000","last_clean":"2024-09-13T15:14:03.195962+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:14:03.195962+0000","last_undegraded":"2024-09-13T15:14:03.195962+0000","last_fullsized":"2024-09-13T15:14:03.195962+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":885,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295516+0000","last_change":"2024-09-13T15:13:11.597792+0000","last_active":"2024-09-13T15:13:57.295516+0000","last_peered":"2024-09-13T15:13:57.295516+0000","last_clean":"2024-09-13T15:13:57.295516+0000","last_became_active":"2024-09-13T15:12:38.938735+0000","last_became_peered":"2024-09-13T15:12:38.938735+0000","last_unstale":"2024-09-13T15:13:57.295516+0000","last_undegraded":"2024-09-13T15:13:57.295516+0000","last_fullsized":"2024-09-13T15:13:57.295516+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.597623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.597623+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:14.909270+0000","scrub_duration":0.012675130999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1171,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296353+0000","last_change":"2024-09-13T15:08:57.728783+0000","last_active":"2024-09-13T15:13:57.296353+0000","last_peered":"2024-09-13T15:13:57.296353+0000","last_clean":"2024-09-13T15:13:57.296353+0000","last_became_active":"2024-09-13T15:08:47.077688+0000","last_became_peered":"2024-09-13T15:08:47.077688+0000","last_unstale":"2024-09-13T15:13:57.296353+0000","last_undegraded":"2024-09-13T15:13:57.296353+0000","last_fullsized":"2024-09-13T15:13:57.296353+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:52.710051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":841,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195587+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:14:03.195587+0000","last_peered":"2024-09-13T15:14:03.195587+0000","last_clean":"2024-09-13T15:14:03.195587+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:14:03.195587+0000","last_undegraded":"2024-09-13T15:14:03.195587+0000","last_fullsized":"2024-09-13T15:14:03.195587+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":356,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271975+0000","last_change":"2024-09-13T15:08:31.187510+0000","last_active":"2024-09-13T15:13:57.271975+0000","last_peered":"2024-09-13T15:13:57.271975+0000","last_clean":"2024-09-13T15:13:57.271975+0000","last_became_active":"2024-09-13T15:08:31.187108+0000","last_became_peered":"2024-09-13T15:08:31.187108+0000","last_unstale":"2024-09-13T15:13:57.271975+0000","last_undegraded":"2024-09-13T15:13:57.271975+0000","last_fullsized":"2024-09-13T15:13:57.271975+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:25.096543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"89'149","reported_seq":922,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795744+0000","last_change":"2024-09-13T15:13:04.107856+0000","last_active":"2024-09-13T15:14:02.795744+0000","last_peered":"2024-09-13T15:14:02.795744+0000","last_clean":"2024-09-13T15:14:02.795744+0000","last_became_active":"2024-09-13T15:12:38.929136+0000","last_became_peered":"2024-09-13T15:12:38.929136+0000","last_unstale":"2024-09-13T15:14:02.795744+0000","last_undegraded":"2024-09-13T15:14:02.795744+0000","last_fullsized":"2024-09-13T15:14:02.795744+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.107315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.107315+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:20.960972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":93,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295305+0000","last_change":"2024-09-13T15:08:31.218115+0000","last_active":"2024-09-13T15:13:57.295305+0000","last_peered":"2024-09-13T15:13:57.295305+0000","last_clean":"2024-09-13T15:13:57.295305+0000","last_became_active":"2024-09-13T15:08:31.196118+0000","last_became_peered":"2024-09-13T15:08:31.196118+0000","last_unstale":"2024-09-13T15:13:57.295305+0000","last_undegraded":"2024-09-13T15:13:57.295305+0000","last_fullsized":"2024-09-13T15:13:57.295305+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:08.721976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":825,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499708+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:14:02.499708+0000","last_peered":"2024-09-13T15:14:02.499708+0000","last_clean":"2024-09-13T15:14:02.499708+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:14:02.499708+0000","last_undegraded":"2024-09-13T15:14:02.499708+0000","last_fullsized":"2024-09-13T15:14:02.499708+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":820,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500111+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:14:02.500111+0000","last_peered":"2024-09-13T15:14:02.500111+0000","last_clean":"2024-09-13T15:14:02.500111+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:14:02.500111+0000","last_undegraded":"2024-09-13T15:14:02.500111+0000","last_fullsized":"2024-09-13T15:14:02.500111+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"89'149","reported_seq":901,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295245+0000","last_change":"2024-09-13T15:13:05.496011+0000","last_active":"2024-09-13T15:13:57.295245+0000","last_peered":"2024-09-13T15:13:57.295245+0000","last_clean":"2024-09-13T15:13:57.295245+0000","last_became_active":"2024-09-13T15:12:38.946687+0000","last_became_peered":"2024-09-13T15:12:38.946687+0000","last_unstale":"2024-09-13T15:13:57.295245+0000","last_undegraded":"2024-09-13T15:13:57.295245+0000","last_fullsized":"2024-09-13T15:13:57.295245+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:05.495826+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:05.495826+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:01.133250+0000","scrub_duration":0.011811428000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1261,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196006+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:14:03.196006+0000","last_peered":"2024-09-13T15:14:03.196006+0000","last_clean":"2024-09-13T15:14:03.196006+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:14:03.196006+0000","last_undegraded":"2024-09-13T15:14:03.196006+0000","last_fullsized":"2024-09-13T15:14:03.196006+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":890,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795719+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:14:02.795719+0000","last_peered":"2024-09-13T15:14:02.795719+0000","last_clean":"2024-09-13T15:14:02.795719+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:14:02.795719+0000","last_undegraded":"2024-09-13T15:14:02.795719+0000","last_fullsized":"2024-09-13T15:14:02.795719+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":628,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.796071+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:14:02.796071+0000","last_peered":"2024-09-13T15:14:02.796071+0000","last_clean":"2024-09-13T15:14:02.796071+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:14:02.796071+0000","last_undegraded":"2024-09-13T15:14:02.796071+0000","last_fullsized":"2024-09-13T15:14:02.796071+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"89'149","reported_seq":879,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295914+0000","last_change":"2024-09-13T15:13:03.104893+0000","last_active":"2024-09-13T15:13:57.295914+0000","last_peered":"2024-09-13T15:13:57.295914+0000","last_clean":"2024-09-13T15:13:57.295914+0000","last_became_active":"2024-09-13T15:12:38.933486+0000","last_became_peered":"2024-09-13T15:12:38.933486+0000","last_unstale":"2024-09-13T15:13:57.295914+0000","last_undegraded":"2024-09-13T15:13:57.295914+0000","last_fullsized":"2024-09-13T15:13:57.295914+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.104648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.104648+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:41.623486+0000","scrub_duration":0.01681821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":425,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296313+0000","last_change":"2024-09-13T15:08:46.074249+0000","last_active":"2024-09-13T15:13:57.296313+0000","last_peered":"2024-09-13T15:13:57.296313+0000","last_clean":"2024-09-13T15:13:57.296313+0000","last_became_active":"2024-09-13T15:08:46.073842+0000","last_became_peered":"2024-09-13T15:08:46.073842+0000","last_unstale":"2024-09-13T15:13:57.296313+0000","last_undegraded":"2024-09-13T15:13:57.296313+0000","last_fullsized":"2024-09-13T15:13:57.296313+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:44.854067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":530,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.286882+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:14:02.286882+0000","last_peered":"2024-09-13T15:14:02.286882+0000","last_clean":"2024-09-13T15:14:02.286882+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:14:02.286882+0000","last_undegraded":"2024-09-13T15:14:02.286882+0000","last_fullsized":"2024-09-13T15:14:02.286882+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":342,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295444+0000","last_change":"2024-09-13T15:08:46.075360+0000","last_active":"2024-09-13T15:13:57.295444+0000","last_peered":"2024-09-13T15:13:57.295444+0000","last_clean":"2024-09-13T15:13:57.295444+0000","last_became_active":"2024-09-13T15:08:46.075046+0000","last_became_peered":"2024-09-13T15:08:46.075046+0000","last_unstale":"2024-09-13T15:13:57.295444+0000","last_undegraded":"2024-09-13T15:13:57.295444+0000","last_fullsized":"2024-09-13T15:13:57.295444+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.044150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"89'149","reported_seq":923,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196225+0000","last_change":"2024-09-13T15:13:04.116692+0000","last_active":"2024-09-13T15:14:03.196225+0000","last_peered":"2024-09-13T15:14:03.196225+0000","last_clean":"2024-09-13T15:14:03.196225+0000","last_became_active":"2024-09-13T15:12:38.940054+0000","last_became_peered":"2024-09-13T15:12:38.940054+0000","last_unstale":"2024-09-13T15:14:03.196225+0000","last_undegraded":"2024-09-13T15:14:03.196225+0000","last_fullsized":"2024-09-13T15:14:03.196225+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.116616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.116616+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:25.864759+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":375,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295836+0000","last_change":"2024-09-13T15:08:46.079918+0000","last_active":"2024-09-13T15:13:57.295836+0000","last_peered":"2024-09-13T15:13:57.295836+0000","last_clean":"2024-09-13T15:13:57.295836+0000","last_became_active":"2024-09-13T15:08:46.075051+0000","last_became_peered":"2024-09-13T15:08:46.075051+0000","last_unstale":"2024-09-13T15:13:57.295836+0000","last_undegraded":"2024-09-13T15:13:57.295836+0000","last_fullsized":"2024-09-13T15:13:57.295836+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:54.485416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1542,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795944+0000","last_change":"2024-09-13T15:08:59.341688+0000","last_active":"2024-09-13T15:14:02.795944+0000","last_peered":"2024-09-13T15:14:02.795944+0000","last_clean":"2024-09-13T15:14:02.795944+0000","last_became_active":"2024-09-13T15:08:59.340925+0000","last_became_peered":"2024-09-13T15:08:59.340925+0000","last_unstale":"2024-09-13T15:14:02.795944+0000","last_undegraded":"2024-09-13T15:14:02.795944+0000","last_fullsized":"2024-09-13T15:14:02.795944+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:22.966119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":470,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499817+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:14:02.499817+0000","last_peered":"2024-09-13T15:14:02.499817+0000","last_clean":"2024-09-13T15:14:02.499817+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:14:02.499817+0000","last_undegraded":"2024-09-13T15:14:02.499817+0000","last_fullsized":"2024-09-13T15:14:02.499817+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"89'149","reported_seq":927,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500081+0000","last_change":"2024-09-13T15:13:03.228430+0000","last_active":"2024-09-13T15:14:02.500081+0000","last_peered":"2024-09-13T15:14:02.500081+0000","last_clean":"2024-09-13T15:14:02.500081+0000","last_became_active":"2024-09-13T15:12:38.937552+0000","last_became_peered":"2024-09-13T15:12:38.937552+0000","last_unstale":"2024-09-13T15:14:02.500081+0000","last_undegraded":"2024-09-13T15:14:02.500081+0000","last_fullsized":"2024-09-13T15:14:02.500081+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.228290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.228290+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:27:30.456205+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":882,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795753+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:14:02.795753+0000","last_peered":"2024-09-13T15:14:02.795753+0000","last_clean":"2024-09-13T15:14:02.795753+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:14:02.795753+0000","last_undegraded":"2024-09-13T15:14:02.795753+0000","last_fullsized":"2024-09-13T15:14:02.795753+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":93,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296541+0000","last_change":"2024-09-13T15:08:31.209459+0000","last_active":"2024-09-13T15:13:57.296541+0000","last_peered":"2024-09-13T15:13:57.296541+0000","last_clean":"2024-09-13T15:13:57.296541+0000","last_became_active":"2024-09-13T15:08:31.188811+0000","last_became_peered":"2024-09-13T15:08:31.188811+0000","last_unstale":"2024-09-13T15:13:57.296541+0000","last_undegraded":"2024-09-13T15:13:57.296541+0000","last_fullsized":"2024-09-13T15:13:57.296541+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.667223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":829,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499936+0000","last_change":"2024-09-13T15:08:31.197894+0000","last_active":"2024-09-13T15:14:02.499936+0000","last_peered":"2024-09-13T15:14:02.499936+0000","last_clean":"2024-09-13T15:14:02.499936+0000","last_became_active":"2024-09-13T15:08:31.197516+0000","last_became_peered":"2024-09-13T15:08:31.197516+0000","last_unstale":"2024-09-13T15:14:02.499936+0000","last_undegraded":"2024-09-13T15:14:02.499936+0000","last_fullsized":"2024-09-13T15:14:02.499936+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:42.650595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"89'149","reported_seq":879,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272105+0000","last_change":"2024-09-13T15:13:00.711640+0000","last_active":"2024-09-13T15:13:57.272105+0000","last_peered":"2024-09-13T15:13:57.272105+0000","last_clean":"2024-09-13T15:13:57.272105+0000","last_became_active":"2024-09-13T15:12:38.935375+0000","last_became_peered":"2024-09-13T15:12:38.935375+0000","last_unstale":"2024-09-13T15:13:57.272105+0000","last_undegraded":"2024-09-13T15:13:57.272105+0000","last_fullsized":"2024-09-13T15:13:57.272105+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.711407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.711407+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:41.333318+0000","scrub_duration":0.0091286909999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":935,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196396+0000","last_change":"2024-09-13T15:08:31.202795+0000","last_active":"2024-09-13T15:14:03.196396+0000","last_peered":"2024-09-13T15:14:03.196396+0000","last_clean":"2024-09-13T15:14:03.196396+0000","last_became_active":"2024-09-13T15:08:31.199963+0000","last_became_peered":"2024-09-13T15:08:31.199963+0000","last_unstale":"2024-09-13T15:14:03.196396+0000","last_undegraded":"2024-09-13T15:14:03.196396+0000","last_fullsized":"2024-09-13T15:14:03.196396+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:08.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1368,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.796117+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:14:02.796117+0000","last_peered":"2024-09-13T15:14:02.796117+0000","last_clean":"2024-09-13T15:14:02.796117+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:14:02.796117+0000","last_undegraded":"2024-09-13T15:14:02.796117+0000","last_fullsized":"2024-09-13T15:14:02.796117+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":639,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195541+0000","last_change":"2024-09-13T15:08:31.198729+0000","last_active":"2024-09-13T15:14:03.195541+0000","last_peered":"2024-09-13T15:14:03.195541+0000","last_clean":"2024-09-13T15:14:03.195541+0000","last_became_active":"2024-09-13T15:08:31.198423+0000","last_became_peered":"2024-09-13T15:08:31.198423+0000","last_unstale":"2024-09-13T15:14:03.195541+0000","last_undegraded":"2024-09-13T15:14:03.195541+0000","last_fullsized":"2024-09-13T15:14:03.195541+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:58.362919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"89'149","reported_seq":920,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195629+0000","last_change":"2024-09-13T15:13:03.079534+0000","last_active":"2024-09-13T15:14:03.195629+0000","last_peered":"2024-09-13T15:14:03.195629+0000","last_clean":"2024-09-13T15:14:03.195629+0000","last_became_active":"2024-09-13T15:12:38.941094+0000","last_became_peered":"2024-09-13T15:12:38.941094+0000","last_unstale":"2024-09-13T15:14:03.195629+0000","last_undegraded":"2024-09-13T15:14:03.195629+0000","last_fullsized":"2024-09-13T15:14:03.195629+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.079258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.079258+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:10.140799+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":756,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500211+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:14:02.500211+0000","last_peered":"2024-09-13T15:14:02.500211+0000","last_clean":"2024-09-13T15:14:02.500211+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:14:02.500211+0000","last_undegraded":"2024-09-13T15:14:02.500211+0000","last_fullsized":"2024-09-13T15:14:02.500211+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":932,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296866+0000","last_change":"2024-09-13T15:08:59.333844+0000","last_active":"2024-09-13T15:13:57.296866+0000","last_peered":"2024-09-13T15:13:57.296866+0000","last_clean":"2024-09-13T15:13:57.296866+0000","last_became_active":"2024-09-13T15:08:59.332369+0000","last_became_peered":"2024-09-13T15:08:59.332369+0000","last_unstale":"2024-09-13T15:13:57.296866+0000","last_undegraded":"2024-09-13T15:13:57.296866+0000","last_fullsized":"2024-09-13T15:13:57.296866+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:36.490753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":526,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.286163+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:14:02.286163+0000","last_peered":"2024-09-13T15:14:02.286163+0000","last_clean":"2024-09-13T15:14:02.286163+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:14:02.286163+0000","last_undegraded":"2024-09-13T15:14:02.286163+0000","last_fullsized":"2024-09-13T15:14:02.286163+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":509,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272000+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:13:57.272000+0000","last_peered":"2024-09-13T15:13:57.272000+0000","last_clean":"2024-09-13T15:13:57.272000+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:13:57.272000+0000","last_undegraded":"2024-09-13T15:13:57.272000+0000","last_fullsized":"2024-09-13T15:13:57.272000+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":641,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.272002+0000","last_change":"2024-09-13T15:08:31.193477+0000","last_active":"2024-09-13T15:13:57.272002+0000","last_peered":"2024-09-13T15:13:57.272002+0000","last_clean":"2024-09-13T15:13:57.272002+0000","last_became_active":"2024-09-13T15:08:31.181861+0000","last_became_peered":"2024-09-13T15:08:31.181861+0000","last_unstale":"2024-09-13T15:13:57.272002+0000","last_undegraded":"2024-09-13T15:13:57.272002+0000","last_fullsized":"2024-09-13T15:13:57.272002+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:02.424164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":33292,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":893,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795855+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:14:02.795855+0000","last_peered":"2024-09-13T15:14:02.795855+0000","last_clean":"2024-09-13T15:14:02.795855+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:14:02.795855+0000","last_undegraded":"2024-09-13T15:14:02.795855+0000","last_fullsized":"2024-09-13T15:14:02.795855+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":81,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296587+0000","last_change":"2024-09-13T15:08:31.204593+0000","last_active":"2024-09-13T15:13:57.296587+0000","last_peered":"2024-09-13T15:13:57.296587+0000","last_clean":"2024-09-13T15:13:57.296587+0000","last_became_active":"2024-09-13T15:08:31.198883+0000","last_became_peered":"2024-09-13T15:08:31.198883+0000","last_unstale":"2024-09-13T15:13:57.296587+0000","last_undegraded":"2024-09-13T15:13:57.296587+0000","last_fullsized":"2024-09-13T15:13:57.296587+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:08.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"89'149","reported_seq":923,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195697+0000","last_change":"2024-09-13T15:13:00.998330+0000","last_active":"2024-09-13T15:14:03.195697+0000","last_peered":"2024-09-13T15:14:03.195697+0000","last_clean":"2024-09-13T15:14:03.195697+0000","last_became_active":"2024-09-13T15:12:50.190273+0000","last_became_peered":"2024-09-13T15:12:50.190273+0000","last_unstale":"2024-09-13T15:14:03.195697+0000","last_undegraded":"2024-09-13T15:14:03.195697+0000","last_fullsized":"2024-09-13T15:14:03.195697+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.998266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.998266+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:06.347067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":398,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296130+0000","last_change":"2024-09-13T15:08:31.188790+0000","last_active":"2024-09-13T15:13:57.296130+0000","last_peered":"2024-09-13T15:13:57.296130+0000","last_clean":"2024-09-13T15:13:57.296130+0000","last_became_active":"2024-09-13T15:08:31.188441+0000","last_became_peered":"2024-09-13T15:08:31.188441+0000","last_unstale":"2024-09-13T15:13:57.296130+0000","last_undegraded":"2024-09-13T15:13:57.296130+0000","last_fullsized":"2024-09-13T15:13:57.296130+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:52.934163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":881,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195773+0000","last_change":"2024-09-13T15:08:31.197112+0000","last_active":"2024-09-13T15:14:03.195773+0000","last_peered":"2024-09-13T15:14:03.195773+0000","last_clean":"2024-09-13T15:14:03.195773+0000","last_became_active":"2024-09-13T15:08:31.196687+0000","last_became_peered":"2024-09-13T15:08:31.196687+0000","last_unstale":"2024-09-13T15:14:03.195773+0000","last_undegraded":"2024-09-13T15:14:03.195773+0000","last_fullsized":"2024-09-13T15:14:03.195773+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:47.532912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":848,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500298+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:14:02.500298+0000","last_peered":"2024-09-13T15:14:02.500298+0000","last_clean":"2024-09-13T15:14:02.500298+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:14:02.500298+0000","last_undegraded":"2024-09-13T15:14:02.500298+0000","last_fullsized":"2024-09-13T15:14:02.500298+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":911,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195576+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:14:03.195576+0000","last_peered":"2024-09-13T15:14:03.195576+0000","last_clean":"2024-09-13T15:14:03.195576+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:14:03.195576+0000","last_undegraded":"2024-09-13T15:14:03.195576+0000","last_fullsized":"2024-09-13T15:14:03.195576+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":304,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.296558+0000","last_change":"2024-09-13T15:08:46.075112+0000","last_active":"2024-09-13T15:13:57.296558+0000","last_peered":"2024-09-13T15:13:57.296558+0000","last_clean":"2024-09-13T15:13:57.296558+0000","last_became_active":"2024-09-13T15:08:46.074818+0000","last_became_peered":"2024-09-13T15:08:46.074818+0000","last_unstale":"2024-09-13T15:13:57.296558+0000","last_undegraded":"2024-09-13T15:13:57.296558+0000","last_fullsized":"2024-09-13T15:13:57.296558+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:56.288263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":875,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195781+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:14:03.195781+0000","last_peered":"2024-09-13T15:14:03.195781+0000","last_clean":"2024-09-13T15:14:03.195781+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:14:03.195781+0000","last_undegraded":"2024-09-13T15:14:03.195781+0000","last_fullsized":"2024-09-13T15:14:03.195781+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":845,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500151+0000","last_change":"2024-09-13T15:08:31.195215+0000","last_active":"2024-09-13T15:14:02.500151+0000","last_peered":"2024-09-13T15:14:02.500151+0000","last_clean":"2024-09-13T15:14:02.500151+0000","last_became_active":"2024-09-13T15:08:31.194526+0000","last_became_peered":"2024-09-13T15:08:31.194526+0000","last_unstale":"2024-09-13T15:14:02.500151+0000","last_undegraded":"2024-09-13T15:14:02.500151+0000","last_fullsized":"2024-09-13T15:14:02.500151+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:56.358764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":847,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.795606+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:14:02.795606+0000","last_peered":"2024-09-13T15:14:02.795606+0000","last_clean":"2024-09-13T15:14:02.795606+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:14:02.795606+0000","last_undegraded":"2024-09-13T15:14:02.795606+0000","last_fullsized":"2024-09-13T15:14:02.795606+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":713,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196019+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:14:03.196019+0000","last_peered":"2024-09-13T15:14:03.196019+0000","last_clean":"2024-09-13T15:14:03.196019+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:14:03.196019+0000","last_undegraded":"2024-09-13T15:14:03.196019+0000","last_fullsized":"2024-09-13T15:14:03.196019+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":504,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271833+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:13:57.271833+0000","last_peered":"2024-09-13T15:13:57.271833+0000","last_clean":"2024-09-13T15:13:57.271833+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:13:57.271833+0000","last_undegraded":"2024-09-13T15:13:57.271833+0000","last_fullsized":"2024-09-13T15:13:57.271833+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":463,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.195981+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:14:03.195981+0000","last_peered":"2024-09-13T15:14:03.195981+0000","last_clean":"2024-09-13T15:14:03.195981+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:14:03.195981+0000","last_undegraded":"2024-09-13T15:14:03.195981+0000","last_fullsized":"2024-09-13T15:14:03.195981+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":832,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.499987+0000","last_change":"2024-09-13T15:08:31.194912+0000","last_active":"2024-09-13T15:14:02.499987+0000","last_peered":"2024-09-13T15:14:02.499987+0000","last_clean":"2024-09-13T15:14:02.499987+0000","last_became_active":"2024-09-13T15:08:31.194553+0000","last_became_peered":"2024-09-13T15:08:31.194553+0000","last_unstale":"2024-09-13T15:14:02.499987+0000","last_undegraded":"2024-09-13T15:14:02.499987+0000","last_fullsized":"2024-09-13T15:14:02.499987+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:17.457690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":514,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271953+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:13:57.271953+0000","last_peered":"2024-09-13T15:13:57.271953+0000","last_clean":"2024-09-13T15:13:57.271953+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:13:57.271953+0000","last_undegraded":"2024-09-13T15:13:57.271953+0000","last_fullsized":"2024-09-13T15:13:57.271953+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":342,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295648+0000","last_change":"2024-09-13T15:08:46.073232+0000","last_active":"2024-09-13T15:13:57.295648+0000","last_peered":"2024-09-13T15:13:57.295648+0000","last_clean":"2024-09-13T15:13:57.295648+0000","last_became_active":"2024-09-13T15:08:46.072807+0000","last_became_peered":"2024-09-13T15:08:46.072807+0000","last_unstale":"2024-09-13T15:13:57.295648+0000","last_undegraded":"2024-09-13T15:13:57.295648+0000","last_fullsized":"2024-09-13T15:13:57.295648+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:11.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":502,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.271744+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:13:57.271744+0000","last_peered":"2024-09-13T15:13:57.271744+0000","last_clean":"2024-09-13T15:13:57.271744+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:13:57.271744+0000","last_undegraded":"2024-09-13T15:13:57.271744+0000","last_fullsized":"2024-09-13T15:13:57.271744+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":370,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-13T15:13:57.295988+0000","last_change":"2024-09-13T15:08:46.079600+0000","last_active":"2024-09-13T15:13:57.295988+0000","last_peered":"2024-09-13T15:13:57.295988+0000","last_clean":"2024-09-13T15:13:57.295988+0000","last_became_active":"2024-09-13T15:08:46.075458+0000","last_became_peered":"2024-09-13T15:08:46.075458+0000","last_unstale":"2024-09-13T15:13:57.295988+0000","last_undegraded":"2024-09-13T15:13:57.295988+0000","last_fullsized":"2024-09-13T15:13:57.295988+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:41.707452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":885,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196105+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:14:03.196105+0000","last_peered":"2024-09-13T15:14:03.196105+0000","last_clean":"2024-09-13T15:14:03.196105+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:14:03.196105+0000","last_undegraded":"2024-09-13T15:14:03.196105+0000","last_fullsized":"2024-09-13T15:14:03.196105+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":908,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:03.196311+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:14:03.196311+0000","last_peered":"2024-09-13T15:14:03.196311+0000","last_clean":"2024-09-13T15:14:03.196311+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:14:03.196311+0000","last_undegraded":"2024-09-13T15:14:03.196311+0000","last_fullsized":"2024-09-13T15:14:03.196311+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":536,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.286085+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:14:02.286085+0000","last_peered":"2024-09-13T15:14:02.286085+0000","last_clean":"2024-09-13T15:14:02.286085+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:14:02.286085+0000","last_undegraded":"2024-09-13T15:14:02.286085+0000","last_fullsized":"2024-09-13T15:14:02.286085+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":872,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-13T15:14:02.500149+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:14:02.500149+0000","last_peered":"2024-09-13T15:14:02.500149+0000","last_clean":"2024-09-13T15:14:02.500149+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:14:02.500149+0000","last_undegraded":"2024-09-13T15:14:02.500149+0000","last_fullsized":"2024-09-13T15:14:02.500149+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":36578,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234563,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529036,"kb_used_data":6528,"kb_used_omap":1,"kb_used_meta":522494,"kb_avail":93216116,"statfs":{"total":95995035648,"available":95453302784,"internally_reserved":0,"allocated":6684672,"data_stored":3973242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":535034315},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496199,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727872,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":721022,"kb_avail":93017280,"statfs":{"total":95995035648,"available":95249694720,"internally_reserved":0,"allocated":7000064,"data_stored":4300522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":738326983},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757834,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876288,"kb_used_data":6772,"kb_used_omap":1,"kb_used_meta":869502,"kb_avail":92868864,"statfs":{"total":95995035648,"available":95097716736,"internally_reserved":0,"allocated":6934528,"data_stored":4231687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890370503},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019474,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":849800,"kb_used_data":6524,"kb_used_omap":1,"kb_used_meta":843262,"kb_avail":92895352,"statfs":{"total":95995035648,"available":95124840448,"internally_reserved":0,"allocated":6680576,"data_stored":3973481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863500742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468358,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989628,"kb_used_data":11184,"kb_used_omap":4,"kb_used_meta":978427,"kb_avail":92755524,"statfs":{"total":95995035648,"available":94981656576,"internally_reserved":0,"allocated":11452416,"data_stored":8753375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":1001909412},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273257,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433412,"kb_used_data":10872,"kb_used_omap":12,"kb_used_meta":422515,"kb_avail":93311740,"statfs":{"total":95995035648,"available":95551221760,"internally_reserved":0,"allocated":11132928,"data_stored":8426243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":432656256},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795410,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894776,"kb_used_data":6508,"kb_used_omap":7,"kb_used_meta":888248,"kb_avail":92850376,"statfs":{"total":95995035648,"available":95078785024,"internally_reserved":0,"allocated":6664192,"data_stored":3968632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":909566578},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502192,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597820,"kb_used_data":10928,"kb_used_omap":10,"kb_used_meta":586869,"kb_avail":93147332,"statfs":{"total":95995035648,"available":95382867968,"internally_reserved":0,"allocated":11190272,"data_stored":8491201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":600954559},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:14:22.827 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-13T15:14:22.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-13T15:14:23.045 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:14:23.046 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T15:14:23.046 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T15:14:23.046 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T15:14:23.046 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:14:23.668 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:23 smithi154 ceph-mon[102314]: pgmap v1622: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:23.668 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:23 smithi154 ceph-mon[107038]: pgmap v1622: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:23.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-13T15:14:23.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:23 smithi172 ceph-mon[100696]: pgmap v1622: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:23.793 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:25.613 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:25 smithi154 ceph-mon[102314]: pgmap v1623: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:25.613 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:25 smithi154 ceph-mon[107038]: pgmap v1623: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:25.657 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:25.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:25 smithi172 ceph-mon[100696]: pgmap v1623: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:26.372 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 1 pg_num to 31 2024-09-13T15:14:26.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4258654745' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T15:14:26.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4258654745' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T15:14:26.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4258654745' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-13T15:14:27.688 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/4258654745' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T15:14:27.688 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[102314]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:14:27.688 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[102314]: pgmap v1625: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:27.688 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T15:14:27.688 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/4258654745' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T15:14:27.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[107038]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:14:27.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[107038]: pgmap v1625: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:27.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T15:14:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/4258654745' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-13T15:14:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:27 smithi172 ceph-mon[100696]: osdmap e501: 8 total, 8 up, 8 in 2024-09-13T15:14:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:27 smithi172 ceph-mon[100696]: pgmap v1625: 93 pgs: 93 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:27.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-13T15:14:28.148 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T15:14:28.148 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134979 9.0 e407/374 9:089187ee:::benchmark_data_smithi154_291684_object34518:head v 407'8381, mlcod=407'8379)", 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.229779+0000", 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 426.862977366, 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30610319800000002, 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229779+0000", 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229779+0000", 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229780+0000", 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1310000000000001e-06 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229788+0000", 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7640000000000005e-06 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229803+0000", 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5176e-05 2024-09-13T15:14:28.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229811+0000", 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0209999999999999e-06 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229881+0000", 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0418000000000002e-05 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.229902+0000", 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0298000000000001e-05 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.535882+0000", 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30598038999999999 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134986 9.0 e407/374 9:0b57539f:::benchmark_data_smithi154_291684_object34525:head v 407'8382, mlcod=407'8379)", 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.231178+0000", 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 426.86157849599999, 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30471788999999999, 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231178+0000", 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231178+0000", 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231179+0000", 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6199999999999996e-07 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231184+0000", 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7490000000000004e-06 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231195+0000", 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0915e-05 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.231202+0000", 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7190000000000004e-06 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.233208+0000", 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002005476 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.233228+0000", 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0911e-05 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.535896+0000", 2024-09-13T15:14:28.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30266725799999999 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134990 9.7 e407/383)", 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.237147+0000", 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 426.85560885299998, 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42793508299999999, 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237147+0000", 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237147+0000", 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237149+0000", 2024-09-13T15:14:28.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.249e-06 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237156+0000", 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.827e-06 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237166+0000", 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0390999999999999e-05 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.237173+0000", 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.024e-06 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.665061+0000", 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.42788764600000001 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.665083+0000", 2024-09-13T15:14:28.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1946e-05 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134987 9.2 e407/374)", 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.253004+0000", 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 426.83975257100002, 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.38170005400000001, 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:28.162 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253004+0000", 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253004+0000", 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253005+0000", 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0499999999999999e-06 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.253014+0000", 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8419999999999994e-06 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.267268+0000", 2024-09-13T15:14:28.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014254271000000001 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.267278+0000", 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8279999999999998e-06 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634685+0000", 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.36740701799999997 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634704+0000", 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9045000000000001e-05 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:134997 9.2 e407/374)", 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.272228+0000", 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 426.82052879999998, 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.36251625199999998, 2024-09-13T15:14:28.166 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272228+0000", 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272228+0000", 2024-09-13T15:14:28.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272229+0000", 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2929999999999999e-06 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272236+0000", 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8680000000000001e-06 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272250+0000", 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.451e-05 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272259+0000", 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6570000000000003e-06 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634731+0000", 2024-09-13T15:14:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.36247178800000002 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.634744+0000", 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3135999999999999e-05 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.45983.0:135001 9.0 e407/374)", 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:14:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 426.82044778300002, 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.25325296200000003, 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272309+0000", 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7399999999999998e-07 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272314+0000", 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5159999999999996e-06 2024-09-13T15:14:28.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272321+0000", 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6620000000000005e-06 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.272326+0000", 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9479999999999999e-06 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.525542+0000", 2024-09-13T15:14:28.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.253215992 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:07:21.525562+0000", 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9369999999999999e-05 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:194327 9.7 9:e2e553f5:::benchmark_data_smithi154_291684_object93866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:45.061955+0000", 2024-09-13T15:14:28.174 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 343.03080137299997, 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0121638749999999, 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 194327 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.061955+0000", 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.061955+0000", 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.061957+0000", 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.012e-06 2024-09-13T15:14:28.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.061964+0000", 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0180000000000004e-06 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.061986+0000", 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2376999999999999e-05 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.061996+0000", 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6639999999999993e-06 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.062167+0000", 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000171096 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.062508+0000", 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000340911 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5,7", 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.062922+0000", 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000414146 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.073309+0000", 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010387087 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.075853+0000", 2024-09-13T15:14:28.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025433169999999998 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.075878+0000", 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5811000000000001e-05 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:46.074112+0000", 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99823311100000001 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:46.074119+0000", 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3250000000000002e-06 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:194345 9.7 9.18f73407 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:45.097051+0000", 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 342.995705703, 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.97708614599999999, 2024-09-13T15:14:28.181 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 194345 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097051+0000", 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097051+0000", 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097053+0000", 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1579999999999999e-06 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097058+0000", 2024-09-13T15:14:28.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2020000000000003e-06 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097071+0000", 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2611e-05 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097078+0000", 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2590000000000003e-06 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097486+0000", 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040792999999999998 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.097494+0000", 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6009999999999995e-06 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:46.074133+0000", 2024-09-13T15:14:28.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.976638267 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:46.074137+0000", 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1180000000000002e-06 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.45983.0:194386 9.7 9.72c537c7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:08:45.142628+0000", 2024-09-13T15:14:28.186 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 342.95012863400001, 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.93152493300000005, 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.45983", 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/3463224064", 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 194386 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142628+0000", 2024-09-13T15:14:28.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142628+0000", 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142629+0000", 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.781e-06 2024-09-13T15:14:28.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142635+0000", 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4190000000000001e-06 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142648+0000", 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3366e-05 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142656+0000", 2024-09-13T15:14:28.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8280000000000003e-06 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142717+0000", 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1266999999999995e-05 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:45.142725+0000", 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9659999999999996e-06 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:46.074149+0000", 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.93142319299999998 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:08:46.074153+0000", 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1130000000000003e-06 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:202979 10.5 10:a05f3667:::benchmark_data_smithi154_317818_object77303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.565222+0000", 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.527534136, 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.362196988, 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202979 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.192 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565222+0000", 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565222+0000", 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565224+0000", 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.66e-06 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565239+0000", 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5321000000000001e-05 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565260+0000", 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1101999999999999e-05 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565270+0000", 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2299999999999997e-06 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565415+0000", 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014503199999999999 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565807+0000", 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039273500000000001 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.565956+0000", 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014836499999999999 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.574923+0000", 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089671019999999994 2024-09-13T15:14:28.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.574979+0000", 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5815e-05 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.926412+0000", 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.35143307200000001 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927349+0000", 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00093697799999999999 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927419+0000", 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0575999999999996e-05 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:202983 10.5 10:aca0bb0d:::benchmark_data_smithi154_317818_object77307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.198 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.568830+0000", 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.523926406, 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.35872014200000002, 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202983 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.568830+0000", 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.568830+0000", 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.568831+0000", 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.234e-06 2024-09-13T15:14:28.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.568836+0000", 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2399999999999998e-06 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.568852+0000", 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5807e-05 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.568875+0000", 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3008000000000001e-05 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.570342+0000", 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001467108 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.570457+0000", 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011469800000000001 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.570570+0000", 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112775 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.575028+0000", 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044578689999999997 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.575043+0000", 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4965e-05 2024-09-13T15:14:28.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927470+0000", 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.35242699300000002 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927517+0000", 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7797999999999999e-05 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927550+0000", 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2647000000000001e-05 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:202991 10.5 10:a14d34d0:::benchmark_data_smithi154_317818_object77315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.578649+0000", 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.514107811, 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.34905406500000002, 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.205 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202991 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578649+0000", 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578649+0000", 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578650+0000", 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2860000000000001e-06 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578655+0000", 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0189999999999997e-06 2024-09-13T15:14:28.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578668+0000", 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3032e-05 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578674+0000", 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1369999999999999e-06 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578727+0000", 2024-09-13T15:14:28.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2652999999999999e-05 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.578885+0000", 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015846100000000001 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.579087+0000", 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020182699999999999 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.583733+0000", 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046462309999999998 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.583910+0000", 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017705600000000001 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927584+0000", 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.34367383000000001 2024-09-13T15:14:28.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927662+0000", 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7628999999999998e-05 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927703+0000", 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0904e-05 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:202999 10.5 10:a7e35698:::benchmark_data_smithi154_317818_object77323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.597974+0000", 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.49478191599999, 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.329861338, 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202999 2024-09-13T15:14:28.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.597974+0000", 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.597974+0000", 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.597976+0000", 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.221e-06 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.597980+0000", 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5480000000000003e-06 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.597993+0000", 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-13T15:14:28.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.597999+0000", 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0009999999999998e-06 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.598047+0000", 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7817e-05 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.598129+0000", 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2199000000000004e-05 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.598203+0000", 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4257000000000004e-05 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.601694+0000", 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003491198 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.602109+0000", 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041437000000000001 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927742+0000", 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.32563323199999999 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927800+0000", 2024-09-13T15:14:28.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8369000000000003e-05 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927836+0000", 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5407000000000002e-05 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:203000 10.5 10:a1f79d11:::benchmark_data_smithi154_317818_object77324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.607961+0000", 2024-09-13T15:14:28.218 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.48479511399999, 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31999526299999997, 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203000 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.607961+0000", 2024-09-13T15:14:28.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.607961+0000", 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.607962+0000", 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2500000000000001e-06 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.607968+0000", 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3140000000000003e-06 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.607981+0000", 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.288e-05 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.607988+0000", 2024-09-13T15:14:28.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4780000000000002e-06 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608040+0000", 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2194999999999997e-05 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608160+0000", 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119957 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608282+0000", 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121563 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.612736+0000", 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044538700000000004 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.612814+0000", 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8633000000000004e-05 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927876+0000", 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31506136699999998 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927920+0000", 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4597e-05 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927956+0000", 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6158999999999997e-05 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:203001 10.3 10:cdb4ebe3:::benchmark_data_smithi154_317818_object77325:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.608012+0000", 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.48474425400001, 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.31058312399999999, 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.225 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203001 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608012+0000", 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608012+0000", 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608013+0000", 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.39e-07 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608015+0000", 2024-09-13T15:14:28.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.525e-06 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608022+0000", 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0480000000000001e-06 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608028+0000", 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4360000000000001e-06 2024-09-13T15:14:28.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608082+0000", 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4948999999999999e-05 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608186+0000", 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103996 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.608290+0000", 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103118 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.612950+0000", 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046599659999999998 2024-09-13T15:14:28.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.613379+0000", 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00042969999999999998 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918337+0000", 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30495740799999999 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918551+0000", 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021404099999999999 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918595+0000", 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4498e-05 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:203005 10.3 10:c825c3e2:::benchmark_data_smithi154_317818_object77329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.614453+0000", 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.47830335099999, 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30426596900000002, 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203005 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.233 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614453+0000", 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614453+0000", 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614455+0000", 2024-09-13T15:14:28.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7579999999999999e-06 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614460+0000", 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5049999999999998e-06 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614475+0000", 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4987999999999999e-05 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614482+0000", 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0890000000000002e-06 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614540+0000", 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7655999999999998e-05 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614650+0000", 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109849 2024-09-13T15:14:28.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.614766+0000", 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011660800000000001 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.618947+0000", 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004180909 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.623495+0000", 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045472849999999999 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918635+0000", 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.29513998400000002 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918678+0000", 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3466e-05 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918719+0000", 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0871999999999998e-05 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:203007 10.3 10:cc847233:::benchmark_data_smithi154_317818_object77331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.619053+0000", 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.4737033, 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.29978021300000002, 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.239 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203007 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619053+0000", 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619053+0000", 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619054+0000", 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.31e-06 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619059+0000", 2024-09-13T15:14:28.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1549999999999998e-06 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619075+0000", 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5279999999999999e-05 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619082+0000", 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4610000000000002e-06 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619124+0000", 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1709999999999999e-05 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619215+0000", 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0728000000000003e-05 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.619313+0000", 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8387999999999994e-05 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.623633+0000", 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043196789999999999 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.623677+0000", 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4230999999999997e-05 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918756+0000", 2024-09-13T15:14:28.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.29507892600000002 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918798+0000", 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2388000000000001e-05 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.918833+0000", 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4956999999999998e-05 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.245 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:203012 10.3 10:c1af58ad:::benchmark_data_smithi154_317818_object77336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.625679+0000", 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.467077774, 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.29689285199999998, 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.246 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203012 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625679+0000", 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625679+0000", 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625680+0000", 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.012e-06 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625684+0000", 2024-09-13T15:14:28.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6779999999999999e-06 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625695+0000", 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0521e-05 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625700+0000", 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0370000000000002e-06 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625741+0000", 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0796e-05 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625819+0000", 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8001000000000001e-05 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,5", 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.625898+0000", 2024-09-13T15:14:28.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8969999999999998e-05 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.629297+0000", 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033990270000000002 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.629512+0000", 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021555599999999999 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.922474+0000", 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.29296193599999998 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.922529+0000", 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4664999999999999e-05 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.922571+0000", 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2653e-05 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.252 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:203013 10.5 10:aa976a44:::benchmark_data_smithi154_317818_object77337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e472)", 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:12:24.626420+0000", 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.466336176, 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.30164271399999998, 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.253 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203013 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626420+0000", 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626420+0000", 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626421+0000", 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5899999999999996e-07 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626425+0000", 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7589999999999999e-06 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626434+0000", 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6469999999999993e-06 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626439+0000", 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.074e-06 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626490+0000", 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0583e-05 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626566+0000", 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5696000000000002e-05 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.626636+0000", 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0421999999999998e-05 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.630057+0000", 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034213830000000001 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.630632+0000", 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000574904 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.927990+0000", 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.29735776600000002 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.928032+0000", 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2429999999999999e-05 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:12:24.928063+0000", 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0391000000000001e-05 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.259 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.36270.0:236839 10.5 10:ad16c1e2:::benchmark_data_smithi154_317818_object111163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T15:13:02.490195+0000", 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 85.602560842000003, 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.25171000500000001, 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.36270", 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2188593023", 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 236839 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T15:14:28.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.490195+0000", 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.490195+0000", 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.490196+0000", 2024-09-13T15:14:28.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5600000000000005e-07 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.490200+0000", 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0040000000000001e-06 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.490211+0000", 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0715e-05 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.490218+0000", 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7680000000000003e-06 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.498361+0000", 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081435499999999994 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.498459+0000", 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7569e-05 2024-09-13T15:14:28.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.498559+0000", 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9827000000000005e-05 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.502912+0000", 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043535300000000004 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.741676+0000", 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.23876336200000001 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.741765+0000", 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9901000000000005e-05 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.741845+0000", 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9758000000000004e-05 2024-09-13T15:14:28.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T15:13:02.741905+0000", 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0265e-05 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T15:14:28.266 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T15:14:28.681 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T15:14:28.682 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:28 smithi154 ceph-mon[102314]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:14:28.682 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:28.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T15:14:28.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:28 smithi154 ceph-mon[107038]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:14:28.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-13T15:14:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:28 smithi172 ceph-mon[100696]: osdmap e502: 8 total, 8 up, 8 in 2024-09-13T15:14:28.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:29.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:29 smithi154 ceph-mon[107038]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:14:29.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:29 smithi154 ceph-mon[107038]: pgmap v1628: 103 pgs: 1 peering, 4 unknown, 98 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:29 smithi154 ceph-mon[102314]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:14:29.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:29 smithi154 ceph-mon[102314]: pgmap v1628: 103 pgs: 1 peering, 4 unknown, 98 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:29 smithi172 ceph-mon[100696]: osdmap e503: 8 total, 8 up, 8 in 2024-09-13T15:14:29.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:29 smithi172 ceph-mon[100696]: pgmap v1628: 103 pgs: 1 peering, 4 unknown, 98 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.6 scrub ok 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.7 scrub ok 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.b scrub starts 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.b scrub ok 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.5 deep-scrub starts 2024-09-13T15:14:30.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.5 deep-scrub ok 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.9 scrub starts 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[102314]: 1.9 scrub ok 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.6 scrub ok 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.7 scrub ok 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.b scrub starts 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.b scrub ok 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.5 deep-scrub starts 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.5 deep-scrub ok 2024-09-13T15:14:30.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.9 scrub starts 2024-09-13T15:14:30.723 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:30 smithi154 ceph-mon[107038]: 1.9 scrub ok 2024-09-13T15:14:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:14:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.6 scrub ok 2024-09-13T15:14:30.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:14:30.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.7 scrub ok 2024-09-13T15:14:30.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.b scrub starts 2024-09-13T15:14:30.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.b scrub ok 2024-09-13T15:14:30.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.5 deep-scrub starts 2024-09-13T15:14:30.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.5 deep-scrub ok 2024-09-13T15:14:30.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.9 scrub starts 2024-09-13T15:14:30.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:30 smithi172 ceph-mon[100696]: 1.9 scrub ok 2024-09-13T15:14:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.16 scrub starts 2024-09-13T15:14:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.16 scrub ok 2024-09-13T15:14:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.c scrub starts 2024-09-13T15:14:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.c scrub ok 2024-09-13T15:14:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.1b scrub starts 2024-09-13T15:14:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.1b scrub ok 2024-09-13T15:14:31.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.e scrub starts 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.e scrub ok 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.d deep-scrub starts 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: 1.d deep-scrub ok 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[107038]: pgmap v1629: 103 pgs: 1 peering, 3 unknown, 99 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.16 scrub starts 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.16 scrub ok 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.c scrub starts 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.c scrub ok 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.1b scrub starts 2024-09-13T15:14:31.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.1b scrub ok 2024-09-13T15:14:31.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.e scrub starts 2024-09-13T15:14:31.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.e scrub ok 2024-09-13T15:14:31.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.d deep-scrub starts 2024-09-13T15:14:31.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: 1.d deep-scrub ok 2024-09-13T15:14:31.723 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-mon[102314]: pgmap v1629: 103 pgs: 1 peering, 3 unknown, 99 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.16 scrub starts 2024-09-13T15:14:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.16 scrub ok 2024-09-13T15:14:31.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.c scrub starts 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.c scrub ok 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.1b scrub starts 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.1b scrub ok 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.e scrub starts 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.e scrub ok 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.d deep-scrub starts 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: 1.d deep-scrub ok 2024-09-13T15:14:31.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:31 smithi172 ceph-mon[100696]: pgmap v1629: 103 pgs: 1 peering, 3 unknown, 99 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:32.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:14:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:14:31] "GET /metrics HTTP/1.1" 200 34229 "" "Prometheus/2.51.0" 2024-09-13T15:14:32.490 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:32 smithi154 ceph-mon[102314]: 1.17 scrub starts 2024-09-13T15:14:32.490 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:32 smithi154 ceph-mon[102314]: 1.17 scrub ok 2024-09-13T15:14:32.490 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:32 smithi154 ceph-mon[107038]: 1.17 scrub starts 2024-09-13T15:14:32.490 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:32 smithi154 ceph-mon[107038]: 1.17 scrub ok 2024-09-13T15:14:32.585 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-13T15:14:32.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-09-13T15:14:32.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:32 smithi172 ceph-mon[100696]: 1.17 scrub starts 2024-09-13T15:14:32.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:32 smithi172 ceph-mon[100696]: 1.17 scrub ok 2024-09-13T15:14:32.799 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:33.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[102314]: 1.1c scrub starts 2024-09-13T15:14:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[102314]: 1.1c scrub ok 2024-09-13T15:14:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[102314]: pgmap v1630: 103 pgs: 103 active+clean; 4.7 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:33.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T15:14:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[107038]: 1.1c scrub starts 2024-09-13T15:14:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[107038]: 1.1c scrub ok 2024-09-13T15:14:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[107038]: pgmap v1630: 103 pgs: 103 active+clean; 4.7 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:33.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:33 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T15:14:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:33 smithi172 ceph-mon[100696]: 1.1c scrub starts 2024-09-13T15:14:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:33 smithi172 ceph-mon[100696]: 1.1c scrub ok 2024-09-13T15:14:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:33 smithi172 ceph-mon[100696]: pgmap v1630: 103 pgs: 103 active+clean; 4.7 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:33.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:33 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-13T15:14:34.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:34 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T15:14:34.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:34 smithi154 ceph-mon[102314]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:14:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:34 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T15:14:34.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:34 smithi154 ceph-mon[107038]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:14:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:34 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-13T15:14:34.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:34 smithi172 ceph-mon[100696]: osdmap e504: 8 total, 8 up, 8 in 2024-09-13T15:14:35.420 INFO:teuthology.orchestra.run.smithi154.stderr:set pool 1 pgp_num to 31 2024-09-13T15:14:35.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[102314]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/732344377' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[102314]: 1.8 scrub starts 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[102314]: 1.8 scrub ok 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[102314]: pgmap v1633: 103 pgs: 1 peering, 1 remapped+peering, 5 unknown, 96 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.8 KiB/s rd, 8.0 KiB/s wr, 2 op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[107038]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/732344377' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[107038]: 1.8 scrub starts 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[107038]: 1.8 scrub ok 2024-09-13T15:14:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:35 smithi154 ceph-mon[107038]: pgmap v1633: 103 pgs: 1 peering, 1 remapped+peering, 5 unknown, 96 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.8 KiB/s rd, 8.0 KiB/s wr, 2 op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T15:14:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:35 smithi172 ceph-mon[100696]: osdmap e505: 8 total, 8 up, 8 in 2024-09-13T15:14:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:35 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/732344377' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-13T15:14:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:35 smithi172 ceph-mon[100696]: 1.8 scrub starts 2024-09-13T15:14:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:35 smithi172 ceph-mon[100696]: 1.8 scrub ok 2024-09-13T15:14:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:35 smithi172 ceph-mon[100696]: pgmap v1633: 103 pgs: 1 peering, 1 remapped+peering, 5 unknown, 96 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.8 KiB/s rd, 8.0 KiB/s wr, 2 op/s; 13 KiB/s, 0 objects/s recovering 2024-09-13T15:14:36.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T15:14:36.390 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:36.724 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[102314]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:14:36.724 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/732344377' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[102314]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[102314]: 1.a scrub starts 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[102314]: 1.a scrub ok 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[107038]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/732344377' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[107038]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[107038]: 1.a scrub starts 2024-09-13T15:14:36.725 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:36 smithi154 ceph-mon[107038]: 1.a scrub ok 2024-09-13T15:14:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:36 smithi172 ceph-mon[100696]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-13T15:14:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/732344377' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-13T15:14:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:36 smithi172 ceph-mon[100696]: osdmap e506: 8 total, 8 up, 8 in 2024-09-13T15:14:36.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:36 smithi172 ceph-mon[100696]: 1.a scrub starts 2024-09-13T15:14:36.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:36 smithi172 ceph-mon[100696]: 1.a scrub ok 2024-09-13T15:14:37.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:37 smithi154 ceph-mon[102314]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:14:37.718 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:37 smithi154 ceph-mon[102314]: pgmap v1636: 103 pgs: 3 peering, 1 remapped+peering, 5 unknown, 94 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 13 op/s; 92 KiB/s, 0 objects/s recovering 2024-09-13T15:14:37.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:37 smithi154 ceph-mon[107038]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:14:37.718 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:37 smithi154 ceph-mon[107038]: pgmap v1636: 103 pgs: 3 peering, 1 remapped+peering, 5 unknown, 94 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 13 op/s; 92 KiB/s, 0 objects/s recovering 2024-09-13T15:14:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:37 smithi172 ceph-mon[100696]: osdmap e507: 8 total, 8 up, 8 in 2024-09-13T15:14:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:37 smithi172 ceph-mon[100696]: pgmap v1636: 103 pgs: 3 peering, 1 remapped+peering, 5 unknown, 94 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 19 KiB/s rd, 55 KiB/s wr, 13 op/s; 92 KiB/s, 0 objects/s recovering 2024-09-13T15:14:38.078 INFO:teuthology.orchestra.run.smithi154.stdout:mon.a: {} 2024-09-13T15:14:38.079 INFO:teuthology.orchestra.run.smithi154.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:14:38.079 INFO:teuthology.orchestra.run.smithi154.stdout:mon.b: {} 2024-09-13T15:14:38.079 INFO:teuthology.orchestra.run.smithi154.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:14:38.088 INFO:teuthology.orchestra.run.smithi154.stdout:mon.c: {} 2024-09-13T15:14:38.088 INFO:teuthology.orchestra.run.smithi154.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T15:14:38.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[107038]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T15:14:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[107038]: 1.1d scrub starts 2024-09-13T15:14:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[107038]: 1.1d scrub ok 2024-09-13T15:14:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/999789208' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:14:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[107038]: 1.19 scrub starts 2024-09-13T15:14:38.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[107038]: 1.19 scrub ok 2024-09-13T15:14:38.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[102314]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T15:14:38.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[102314]: 1.1d scrub starts 2024-09-13T15:14:38.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[102314]: 1.1d scrub ok 2024-09-13T15:14:38.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/999789208' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:14:38.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[102314]: 1.19 scrub starts 2024-09-13T15:14:38.722 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:38 smithi154 ceph-mon[102314]: 1.19 scrub ok 2024-09-13T15:14:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:38 smithi172 ceph-mon[100696]: osdmap e508: 8 total, 8 up, 8 in 2024-09-13T15:14:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:38 smithi172 ceph-mon[100696]: 1.1d scrub starts 2024-09-13T15:14:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:38 smithi172 ceph-mon[100696]: 1.1d scrub ok 2024-09-13T15:14:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:38 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/999789208' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-13T15:14:38.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:38 smithi172 ceph-mon[100696]: 1.19 scrub starts 2024-09-13T15:14:38.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:38 smithi172 ceph-mon[100696]: 1.19 scrub ok 2024-09-13T15:14:38.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 1 1 2024-09-13T15:14:38.960 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:39 smithi154 ceph-mon[102314]: 1.1e scrub starts 2024-09-13T15:14:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:39 smithi154 ceph-mon[102314]: 1.1e scrub ok 2024-09-13T15:14:39.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:39 smithi154 ceph-mon[102314]: pgmap v1638: 103 pgs: 2 peering, 1 remapped+peering, 100 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 27 KiB/s rd, 79 KiB/s wr, 19 op/s; 184 KiB/s, 1 objects/s recovering 2024-09-13T15:14:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:39 smithi154 ceph-mon[107038]: 1.1e scrub starts 2024-09-13T15:14:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:39 smithi154 ceph-mon[107038]: 1.1e scrub ok 2024-09-13T15:14:39.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:39 smithi154 ceph-mon[107038]: pgmap v1638: 103 pgs: 2 peering, 1 remapped+peering, 100 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 27 KiB/s rd, 79 KiB/s wr, 19 op/s; 184 KiB/s, 1 objects/s recovering 2024-09-13T15:14:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:39 smithi172 ceph-mon[100696]: 1.1e scrub starts 2024-09-13T15:14:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:39 smithi172 ceph-mon[100696]: 1.1e scrub ok 2024-09-13T15:14:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:39 smithi172 ceph-mon[100696]: pgmap v1638: 103 pgs: 2 peering, 1 remapped+peering, 100 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 27 KiB/s rd, 79 KiB/s wr, 19 op/s; 184 KiB/s, 1 objects/s recovering 2024-09-13T15:14:40.994 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T15:14:41.244 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:40 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2137953420' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:41.245 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:40 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:41.301 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:40 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2137953420' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:41.302 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:40 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:41.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:40 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2137953420' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:41.302 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:40 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:41.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-13T15:14:41.934 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:41.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:14:41] "GET /metrics HTTP/1.1" 200 34229 "" "Prometheus/2.51.0" 2024-09-13T15:14:42.330 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[102314]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[102314]: pgmap v1640: 103 pgs: 2 peering, 101 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 47 KiB/s wr, 11 op/s; 122 KiB/s, 0 objects/s recovering 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[102314]: 1.1a scrub starts 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[102314]: 1.1a scrub ok 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[107038]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[107038]: pgmap v1640: 103 pgs: 2 peering, 101 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 47 KiB/s wr, 11 op/s; 122 KiB/s, 0 objects/s recovering 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[107038]: 1.1a scrub starts 2024-09-13T15:14:42.331 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:41 smithi154 ceph-mon[107038]: 1.1a scrub ok 2024-09-13T15:14:42.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:41 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:14:42.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:41 smithi172 ceph-mon[100696]: osdmap e509: 8 total, 8 up, 8 in 2024-09-13T15:14:42.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:41 smithi172 ceph-mon[100696]: pgmap v1640: 103 pgs: 2 peering, 101 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 16 KiB/s rd, 47 KiB/s wr, 11 op/s; 122 KiB/s, 0 objects/s recovering 2024-09-13T15:14:42.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:41 smithi172 ceph-mon[100696]: 1.1a scrub starts 2024-09-13T15:14:42.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:41 smithi172 ceph-mon[100696]: 1.1a scrub ok 2024-09-13T15:14:43.258 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[102314]: 1.1c scrub starts 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[102314]: 1.1c scrub ok 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[102314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[102314]: Cluster is now healthy 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[102314]: 1.18 scrub starts 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[102314]: 1.18 scrub ok 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[107038]: 1.1c scrub starts 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[107038]: 1.1c scrub ok 2024-09-13T15:14:43.259 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[107038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T15:14:43.260 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[107038]: Cluster is now healthy 2024-09-13T15:14:43.260 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[107038]: 1.18 scrub starts 2024-09-13T15:14:43.260 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:42 smithi154 ceph-mon[107038]: 1.18 scrub ok 2024-09-13T15:14:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:42 smithi172 ceph-mon[100696]: 1.1c scrub starts 2024-09-13T15:14:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:42 smithi172 ceph-mon[100696]: 1.1c scrub ok 2024-09-13T15:14:43.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:42 smithi172 ceph-mon[100696]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-13T15:14:43.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:42 smithi172 ceph-mon[100696]: Cluster is now healthy 2024-09-13T15:14:43.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:42 smithi172 ceph-mon[100696]: 1.18 scrub starts 2024-09-13T15:14:43.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:42 smithi172 ceph-mon[100696]: 1.18 scrub ok 2024-09-13T15:14:44.001 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T15:14:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[102314]: 1.15 scrub starts 2024-09-13T15:14:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[102314]: 1.15 scrub ok 2024-09-13T15:14:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[102314]: pgmap v1641: 103 pgs: 103 active+clean; 4.5 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 5.4 KiB/s rd, 16 KiB/s wr, 3 op/s; 64 KiB/s, 0 objects/s recovering 2024-09-13T15:14:44.376 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1682837318' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:44.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[107038]: 1.15 scrub starts 2024-09-13T15:14:44.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[107038]: 1.15 scrub ok 2024-09-13T15:14:44.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:44.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[107038]: pgmap v1641: 103 pgs: 103 active+clean; 4.5 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 5.4 KiB/s rd, 16 KiB/s wr, 3 op/s; 64 KiB/s, 0 objects/s recovering 2024-09-13T15:14:44.377 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:44 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1682837318' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:44 smithi172 ceph-mon[100696]: 1.15 scrub starts 2024-09-13T15:14:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:44 smithi172 ceph-mon[100696]: 1.15 scrub ok 2024-09-13T15:14:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:44 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:44 smithi172 ceph-mon[100696]: pgmap v1641: 103 pgs: 103 active+clean; 4.5 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 5.4 KiB/s rd, 16 KiB/s wr, 3 op/s; 64 KiB/s, 0 objects/s recovering 2024-09-13T15:14:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:44 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1682837318' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-13T15:14:45.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 3 1 2024-09-13T15:14:45.277 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:45.393 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:45 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1682837318' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:14:45.393 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:45 smithi172 ceph-mon[100696]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T15:14:45.398 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:45 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1682837318' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:14:45.398 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:45 smithi154 ceph-mon[102314]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T15:14:45.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:45 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1682837318' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-13T15:14:45.398 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:45 smithi154 ceph-mon[107038]: osdmap e510: 8 total, 8 up, 8 in 2024-09-13T15:14:46.409 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:46 smithi172 ceph-mon[100696]: pgmap v1643: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:46.431 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:46 smithi154 ceph-mon[102314]: pgmap v1643: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:46.431 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:46 smithi154 ceph-mon[107038]: pgmap v1643: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:47 smithi154 ceph-mon[102314]: 1.1b deep-scrub starts 2024-09-13T15:14:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:47 smithi154 ceph-mon[102314]: 1.1b deep-scrub ok 2024-09-13T15:14:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:47 smithi154 ceph-mon[107038]: 1.1b deep-scrub starts 2024-09-13T15:14:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:47 smithi154 ceph-mon[107038]: 1.1b deep-scrub ok 2024-09-13T15:14:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:47 smithi172 ceph-mon[100696]: 1.1b deep-scrub starts 2024-09-13T15:14:47.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:47 smithi172 ceph-mon[100696]: 1.1b deep-scrub ok 2024-09-13T15:14:48.015 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T15:14:48.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:48 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2292309097' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:48.320 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:48 smithi154 ceph-mon[102314]: pgmap v1644: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:48.320 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:48 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2292309097' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:48.321 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:48 smithi154 ceph-mon[107038]: pgmap v1644: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:48 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2292309097' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:48.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:48 smithi172 ceph-mon[100696]: pgmap v1644: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:48.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-13T15:14:48.940 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:49.325 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:49 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2292309097' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:14:49.325 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:49 smithi154 ceph-mon[102314]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T15:14:49.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:49 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2292309097' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:14:49.326 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:49 smithi154 ceph-mon[107038]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T15:14:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:49 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2292309097' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:14:49.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:49 smithi172 ceph-mon[100696]: osdmap e511: 8 total, 8 up, 8 in 2024-09-13T15:14:50.288 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:50 smithi154 ceph-mon[102314]: pgmap v1646: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:50.289 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:50 smithi154 ceph-mon[107038]: pgmap v1646: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:50.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:50 smithi172 ceph-mon[100696]: pgmap v1646: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:51.027 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T15:14:51.404 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:51 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3870617722' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:51.405 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:51 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:51.405 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:51 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3870617722' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:51.405 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:51 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:51 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3870617722' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:51.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:51 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-13T15:14:51.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 4 1 2024-09-13T15:14:52.019 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:52.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:52 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:14:52.141 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:52 smithi154 ceph-mon[102314]: osdmap e512: 8 total, 8 up, 8 in 2024-09-13T15:14:52.142 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:52 smithi154 ceph-mon[102314]: pgmap v1648: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:52.142 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:14:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:14:51] "GET /metrics HTTP/1.1" 200 34250 "" "Prometheus/2.51.0" 2024-09-13T15:14:52.142 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:52 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:14:52.142 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:52 smithi154 ceph-mon[107038]: osdmap e512: 8 total, 8 up, 8 in 2024-09-13T15:14:52.142 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:52 smithi154 ceph-mon[107038]: pgmap v1648: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:52 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-13T15:14:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:52 smithi172 ceph-mon[100696]: osdmap e512: 8 total, 8 up, 8 in 2024-09-13T15:14:52.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:52 smithi172 ceph-mon[100696]: pgmap v1648: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:53.641 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:53 smithi154 ceph-mon[102314]: pgmap v1649: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:53.642 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:53 smithi154 ceph-mon[107038]: pgmap v1649: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:53 smithi172 ceph-mon[100696]: pgmap v1649: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:54.373 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T15:14:54.698 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:54 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3816505461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:54.698 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:54 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3816505461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:54.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:54 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3816505461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:55.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-13T15:14:55.315 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:55.700 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:55 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3816505461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:14:55.700 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:55 smithi154 ceph-mon[102314]: osdmap e513: 8 total, 8 up, 8 in 2024-09-13T15:14:55.700 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:55 smithi154 ceph-mon[102314]: pgmap v1651: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:55.700 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:55 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3816505461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:14:55.701 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:55 smithi154 ceph-mon[107038]: osdmap e513: 8 total, 8 up, 8 in 2024-09-13T15:14:55.701 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:55 smithi154 ceph-mon[107038]: pgmap v1651: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:55 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3816505461' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:14:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:55 smithi172 ceph-mon[100696]: osdmap e513: 8 total, 8 up, 8 in 2024-09-13T15:14:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:55 smithi172 ceph-mon[100696]: pgmap v1651: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:57.141 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T15:14:57.445 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:57 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1366091452' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:57.445 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:57 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:57.445 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:57 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1366091452' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:57.445 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:57 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:57 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1366091452' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:57.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:57 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-13T15:14:57.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 5 1 2024-09-13T15:14:58.050 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:14:58.449 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:14:58.449 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[102314]: osdmap e514: 8 total, 8 up, 8 in 2024-09-13T15:14:58.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[102314]: pgmap v1653: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:58.450 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:58.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:14:58.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[107038]: osdmap e514: 8 total, 8 up, 8 in 2024-09-13T15:14:58.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[107038]: pgmap v1653: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:58.450 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:58 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-13T15:14:58.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:58 smithi172 ceph-mon[100696]: osdmap e514: 8 total, 8 up, 8 in 2024-09-13T15:14:58.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:58 smithi172 ceph-mon[100696]: pgmap v1653: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:58.492 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:14:59.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:14:59 smithi154 ceph-mon[102314]: pgmap v1654: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:59.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:14:59 smithi154 ceph-mon[107038]: pgmap v1654: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:14:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:14:59 smithi172 ceph-mon[100696]: pgmap v1654: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:00.379 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T15:15:00.702 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:00 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3366630410' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:00.703 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:00 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3366630410' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:00.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:00 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3366630410' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:01.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-13T15:15:01.246 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:01.665 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:01 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3366630410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:15:01.665 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:01 smithi154 ceph-mon[102314]: osdmap e515: 8 total, 8 up, 8 in 2024-09-13T15:15:01.665 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:01 smithi154 ceph-mon[102314]: pgmap v1656: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:01.665 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:01 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3366630410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:15:01.666 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:01 smithi154 ceph-mon[107038]: osdmap e515: 8 total, 8 up, 8 in 2024-09-13T15:15:01.666 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:01 smithi154 ceph-mon[107038]: pgmap v1656: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:01 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3366630410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:15:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:01 smithi172 ceph-mon[100696]: osdmap e515: 8 total, 8 up, 8 in 2024-09-13T15:15:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:01 smithi172 ceph-mon[100696]: pgmap v1656: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:01.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:15:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:15:01] "GET /metrics HTTP/1.1" 200 34228 "" "Prometheus/2.51.0" 2024-09-13T15:15:03.254 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T15:15:03.534 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:03 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1901730924' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:03.535 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:03 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:03.535 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:03 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1901730924' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:03.535 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:03 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:03 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1901730924' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:03 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-13T15:15:03.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 6 1 2024-09-13T15:15:04.158 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:04.541 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:04 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:15:04.541 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:04 smithi154 ceph-mon[102314]: osdmap e516: 8 total, 8 up, 8 in 2024-09-13T15:15:04.541 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:04 smithi154 ceph-mon[102314]: pgmap v1658: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:04.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:04 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:15:04.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:04 smithi154 ceph-mon[107038]: osdmap e516: 8 total, 8 up, 8 in 2024-09-13T15:15:04.541 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:04 smithi154 ceph-mon[107038]: pgmap v1658: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:04 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-13T15:15:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:04 smithi172 ceph-mon[100696]: osdmap e516: 8 total, 8 up, 8 in 2024-09-13T15:15:04.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:04 smithi172 ceph-mon[100696]: pgmap v1658: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:05.583 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:05 smithi154 ceph-mon[107038]: pgmap v1659: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:05.583 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:05 smithi154 ceph-mon[102314]: pgmap v1659: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:05.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:05 smithi172 ceph-mon[100696]: pgmap v1659: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:06.392 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T15:15:06.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1787405783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:06.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:06 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:06.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1787405783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:06.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:06 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1787405783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:06.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:06 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:07.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-13T15:15:07.341 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:07.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:07 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:15:07.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:07 smithi154 ceph-mon[102314]: osdmap e517: 8 total, 8 up, 8 in 2024-09-13T15:15:07.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:07 smithi154 ceph-mon[102314]: pgmap v1661: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:07.661 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:07 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:15:07.661 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:07 smithi154 ceph-mon[107038]: osdmap e517: 8 total, 8 up, 8 in 2024-09-13T15:15:07.661 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:07 smithi154 ceph-mon[107038]: pgmap v1661: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:07 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:15:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:07 smithi172 ceph-mon[100696]: osdmap e517: 8 total, 8 up, 8 in 2024-09-13T15:15:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:07 smithi172 ceph-mon[100696]: pgmap v1661: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:09.350 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T15:15:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:09 smithi154 ceph-mon[102314]: pgmap v1662: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:09 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3069531606' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:09 smithi154 ceph-mon[107038]: pgmap v1662: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:09 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3069531606' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:09 smithi172 ceph-mon[100696]: pgmap v1662: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:09 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3069531606' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-13T15:15:10.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 7 1 2024-09-13T15:15:10.372 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:10.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:10 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3069531606' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:15:10.471 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:10 smithi154 ceph-mon[102314]: osdmap e518: 8 total, 8 up, 8 in 2024-09-13T15:15:10.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:10 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3069531606' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:15:10.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:10 smithi154 ceph-mon[107038]: osdmap e518: 8 total, 8 up, 8 in 2024-09-13T15:15:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:10 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3069531606' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-13T15:15:10.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:10 smithi172 ceph-mon[100696]: osdmap e518: 8 total, 8 up, 8 in 2024-09-13T15:15:11.589 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:11 smithi154 ceph-mon[102314]: osdmap e519: 8 total, 8 up, 8 in 2024-09-13T15:15:11.589 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:11 smithi154 ceph-mon[102314]: pgmap v1665: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:11.590 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:11 smithi154 ceph-mon[107038]: osdmap e519: 8 total, 8 up, 8 in 2024-09-13T15:15:11.590 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:11 smithi154 ceph-mon[107038]: pgmap v1665: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:11 smithi172 ceph-mon[100696]: osdmap e519: 8 total, 8 up, 8 in 2024-09-13T15:15:11.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:11 smithi172 ceph-mon[100696]: pgmap v1665: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:12.221 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:15:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:15:11] "GET /metrics HTTP/1.1" 200 34228 "" "Prometheus/2.51.0" 2024-09-13T15:15:12.380 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T15:15:12.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:12 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3528541967' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:15:12.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:12 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3528541967' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:15:12.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:12 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3528541967' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:15:13.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-13T15:15:13.358 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:13.386 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[107038]: 1.6 scrub starts 2024-09-13T15:15:13.386 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[107038]: 1.6 scrub ok 2024-09-13T15:15:13.386 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3528541967' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:15:13.386 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[107038]: osdmap e520: 8 total, 8 up, 8 in 2024-09-13T15:15:13.387 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:13.709 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[107038]: pgmap v1667: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:13.709 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[102314]: 1.6 scrub starts 2024-09-13T15:15:13.709 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[102314]: 1.6 scrub ok 2024-09-13T15:15:13.710 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3528541967' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:15:13.710 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[102314]: osdmap e520: 8 total, 8 up, 8 in 2024-09-13T15:15:13.710 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:13.710 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:13 smithi154 ceph-mon[102314]: pgmap v1667: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:13.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:13 smithi172 ceph-mon[100696]: 1.6 scrub starts 2024-09-13T15:15:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:13 smithi172 ceph-mon[100696]: 1.6 scrub ok 2024-09-13T15:15:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3528541967' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:15:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:13 smithi172 ceph-mon[100696]: osdmap e520: 8 total, 8 up, 8 in 2024-09-13T15:15:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:13 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:13.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:13 smithi172 ceph-mon[100696]: pgmap v1667: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:14.838 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:14 smithi154 ceph-mon[102314]: 1.f deep-scrub starts 2024-09-13T15:15:14.838 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:14 smithi154 ceph-mon[102314]: 1.f deep-scrub ok 2024-09-13T15:15:14.839 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:14 smithi154 ceph-mon[107038]: 1.f deep-scrub starts 2024-09-13T15:15:14.839 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:14 smithi154 ceph-mon[107038]: 1.f deep-scrub ok 2024-09-13T15:15:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:14 smithi172 ceph-mon[100696]: 1.f deep-scrub starts 2024-09-13T15:15:14.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:14 smithi172 ceph-mon[100696]: 1.f deep-scrub ok 2024-09-13T15:15:15.610 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T15:15:15.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:15 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3546760443' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:15:15.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:15 smithi172 ceph-mon[100696]: pgmap v1668: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 9.3 KiB/s rd, 27 KiB/s wr, 6 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-13T15:15:15.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:15 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3546760443' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:15:15.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:15 smithi154 ceph-mon[102314]: pgmap v1668: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 9.3 KiB/s rd, 27 KiB/s wr, 6 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-13T15:15:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:15 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3546760443' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-13T15:15:15.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:15 smithi154 ceph-mon[107038]: pgmap v1668: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 9.3 KiB/s rd, 27 KiB/s wr, 6 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-13T15:15:16.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 0 1 2024-09-13T15:15:16.599 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:16.934 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:16 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3546760443' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:15:16.959 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:16 smithi154 ceph-mon[102314]: osdmap e521: 8 total, 8 up, 8 in 2024-09-13T15:15:16.959 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:16 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3546760443' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:15:16.959 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:16 smithi154 ceph-mon[107038]: osdmap e521: 8 total, 8 up, 8 in 2024-09-13T15:15:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:16 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3546760443' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-13T15:15:16.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:16 smithi172 ceph-mon[100696]: osdmap e521: 8 total, 8 up, 8 in 2024-09-13T15:15:17.689 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:17 smithi154 ceph-mon[102314]: pgmap v1670: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.1 KiB/s rd, 24 KiB/s wr, 5 op/s; 34 KiB/s, 0 objects/s recovering 2024-09-13T15:15:17.689 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:17 smithi154 ceph-mon[107038]: pgmap v1670: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.1 KiB/s rd, 24 KiB/s wr, 5 op/s; 34 KiB/s, 0 objects/s recovering 2024-09-13T15:15:17.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:17 smithi172 ceph-mon[100696]: pgmap v1670: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.1 KiB/s rd, 24 KiB/s wr, 5 op/s; 34 KiB/s, 0 objects/s recovering 2024-09-13T15:15:18.640 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T15:15:18.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:18 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/944759185' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:18.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:18 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:18 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/944759185' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:18.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:18 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:18 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/944759185' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:18.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:18 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:19.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-13T15:15:19.669 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:19.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:19 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:15:19.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:19 smithi154 ceph-mon[102314]: osdmap e522: 8 total, 8 up, 8 in 2024-09-13T15:15:19.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:19 smithi154 ceph-mon[102314]: pgmap v1672: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.6 KiB/s rd, 8.0 KiB/s wr, 1 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T15:15:19.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:19 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:15:19.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:19 smithi154 ceph-mon[107038]: osdmap e522: 8 total, 8 up, 8 in 2024-09-13T15:15:19.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:19 smithi154 ceph-mon[107038]: pgmap v1672: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.6 KiB/s rd, 8.0 KiB/s wr, 1 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T15:15:19.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:19 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:15:19.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:19 smithi172 ceph-mon[100696]: osdmap e522: 8 total, 8 up, 8 in 2024-09-13T15:15:19.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:19 smithi172 ceph-mon[100696]: pgmap v1672: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.6 KiB/s rd, 8.0 KiB/s wr, 1 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-13T15:15:20.765 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:20 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:15:20.766 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:20 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:15:20.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:20 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-13T15:15:21.863 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T15:15:22.073 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.073 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.073 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.073 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:21 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.073 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:21 smithi172 ceph-mon[100696]: pgmap v1673: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.9 KiB/s wr, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-09-13T15:15:22.074 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:21 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/772874568' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:22.205 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:15:21] "GET /metrics HTTP/1.1" 200 34228 "" "Prometheus/2.51.0" 2024-09-13T15:15:22.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[107038]: pgmap v1673: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.9 KiB/s wr, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-09-13T15:15:22.206 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/772874568' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:22.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:22.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[102314]: pgmap v1673: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.9 KiB/s wr, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-09-13T15:15:22.207 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:21 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/772874568' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-13T15:15:22.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd reweight 2 1 2024-09-13T15:15:23.127 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:22 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/772874568' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:15:23.128 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:22 smithi172 ceph-mon[100696]: osdmap e523: 8 total, 8 up, 8 in 2024-09-13T15:15:23.213 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:23.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:22 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/772874568' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:15:23.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:22 smithi154 ceph-mon[102314]: osdmap e523: 8 total, 8 up, 8 in 2024-09-13T15:15:23.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:22 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/772874568' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-13T15:15:23.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:22 smithi154 ceph-mon[107038]: osdmap e523: 8 total, 8 up, 8 in 2024-09-13T15:15:23.871 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:23 smithi154 ceph-mon[102314]: pgmap v1675: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:23.871 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:23 smithi154 ceph-mon[102314]: 1.7 scrub starts 2024-09-13T15:15:23.871 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:23 smithi154 ceph-mon[102314]: 1.7 scrub ok 2024-09-13T15:15:24.201 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:23 smithi154 ceph-mon[107038]: pgmap v1675: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:24.201 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:23 smithi154 ceph-mon[107038]: 1.7 scrub starts 2024-09-13T15:15:24.201 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:23 smithi154 ceph-mon[107038]: 1.7 scrub ok 2024-09-13T15:15:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:23 smithi172 ceph-mon[100696]: pgmap v1675: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:23 smithi172 ceph-mon[100696]: 1.7 scrub starts 2024-09-13T15:15:24.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:23 smithi172 ceph-mon[100696]: 1.7 scrub ok 2024-09-13T15:15:26.192 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:26.193 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:25 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:26.193 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:25 smithi154 ceph-mon[102314]: pgmap v1676: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:26.193 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:26.193 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:25 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:26.193 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:25 smithi154 ceph-mon[107038]: pgmap v1676: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:25 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:26.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:25 smithi172 ceph-mon[100696]: pgmap v1676: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:26.826 INFO:teuthology.orchestra.run.smithi154.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T15:15:27.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:26 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1213431224' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:27.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:26 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1213431224' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:27.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:26 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1213431224' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:27.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-13T15:15:27.913 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:27.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/1213431224' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[102314]: osdmap e524: 8 total, 8 up, 8 in 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[102314]: pgmap v1678: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/1213431224' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[107038]: osdmap e524: 8 total, 8 up, 8 in 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:27.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:27 smithi154 ceph-mon[107038]: pgmap v1678: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:27 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/1213431224' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:15:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:27 smithi172 ceph-mon[100696]: osdmap e524: 8 total, 8 up, 8 in 2024-09-13T15:15:28.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:28.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:27 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:28.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:27 smithi172 ceph-mon[100696]: pgmap v1678: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:29.178 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:15:29.178 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:15:29.178 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:29.178 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:29.179 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:15:29.179 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:15:29.179 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:29.179 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:28 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-13T15:15:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-13T15:15:29.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' 2024-09-13T15:15:29.242 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:28 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:29.836 INFO:teuthology.orchestra.run.smithi154.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T15:15:30.179 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:29 smithi154 ceph-mon[102314]: pgmap v1679: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:30.179 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:29 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/938132085' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:30.179 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:29 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:30.180 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:29 smithi154 ceph-mon[107038]: pgmap v1679: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:30.180 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:29 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/938132085' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:30.180 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:29 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:29 smithi172 ceph-mon[100696]: pgmap v1679: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:29 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/938132085' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:30.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:29 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-13T15:15:30.631 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T15:15:30.631 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T15:15:30.631 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T15:15:30.631 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T15:15:30.631 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T15:15:30.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:15:30.836 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:30.958 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:30 smithi154 ceph-mon[102314]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:15:30.958 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:30 smithi154 ceph-mon[102314]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T15:15:30.958 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:30 smithi154 ceph-mon[107038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:15:30.959 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:30 smithi154 ceph-mon[107038]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T15:15:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:30 smithi172 ceph-mon[100696]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-13T15:15:31.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:30 smithi172 ceph-mon[100696]: osdmap e525: 8 total, 8 up, 8 in 2024-09-13T15:15:32.123 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:31 smithi154 ceph-mon[102314]: pgmap v1681: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:32.123 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:15:31 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:15:31] "GET /metrics HTTP/1.1" 200 34228 "" "Prometheus/2.51.0" 2024-09-13T15:15:32.124 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:31 smithi154 ceph-mon[107038]: pgmap v1681: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:32.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:31 smithi172 ceph-mon[100696]: pgmap v1681: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:32.860 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:15:32.861 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":525,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T15:15:29.827516+0000","last_up_change":"2024-09-13T15:00:14.027470+0000","last_in_change":"2024-09-13T15:08:30.165081+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"506","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"502","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:28:12.645576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:28:13.527124+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":93,"last_clean_end":301,"up_from":321,"up_thru":506,"down_at":301,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6803","nonce":3127717424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6805","nonce":3127717424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6809","nonce":3127717424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6807","nonce":3127717424}]},"public_addr":"172.21.15.154:6803/3127717424","cluster_addr":"172.21.15.154:6805/3127717424","heartbeat_back_addr":"172.21.15.154:6809/3127717424","heartbeat_front_addr":"172.21.15.154:6807/3127717424","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":94,"up_from":98,"up_thru":507,"down_at":95,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6811","nonce":148745834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6813","nonce":148745834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6817","nonce":148745834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6815","nonce":148745834}]},"public_addr":"172.21.15.154:6811/148745834","cluster_addr":"172.21.15.154:6813/148745834","heartbeat_back_addr":"172.21.15.154:6817/148745834","heartbeat_front_addr":"172.21.15.154:6815/148745834","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":103,"last_clean_end":280,"up_from":328,"up_thru":506,"down_at":281,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6819","nonce":2560992094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6821","nonce":2560992094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6825","nonce":2560992094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6823","nonce":2560992094}]},"public_addr":"172.21.15.154:6819/2560992094","cluster_addr":"172.21.15.154:6821/2560992094","heartbeat_back_addr":"172.21.15.154:6825/2560992094","heartbeat_front_addr":"172.21.15.154:6823/2560992094","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":104,"up_from":108,"up_thru":507,"down_at":105,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6827","nonce":381867861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6829","nonce":381867861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6833","nonce":381867861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6831","nonce":381867861}]},"public_addr":"172.21.15.154:6827/381867861","cluster_addr":"172.21.15.154:6829/381867861","heartbeat_back_addr":"172.21.15.154:6833/381867861","heartbeat_front_addr":"172.21.15.154:6831/381867861","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":300},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":94},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":280},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":104},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":0}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":3,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/2090383084":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/1584830159":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/2189531829":"2024-09-14T14:34:12.641827+0000","172.21.15.154:6801/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:6800/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/3255496958":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/18164567":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/4103428848":"2024-09-14T14:31:33.289382+0000","172.21.15.172:0/3191030600":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6801/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/213038927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/3680637464":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:0/2607163146":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1379294727":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/2289953927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/3924873105":"2024-09-14T14:31:04.278426+0000","172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/3573827628":"2024-09-14T14:31:33.289382+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.172:6834/3129749965":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:6835/3129749965":"2024-09-14T14:31:33.289382+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:15:32.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:32 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2423049135' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:15:32.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:32 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2423049135' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:15:33.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:32 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2423049135' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:15:33.708 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T15:15:33.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:15:33.922 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:33.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:33 smithi154 ceph-mon[102314]: pgmap v1682: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:33.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:33 smithi154 ceph-mon[107038]: pgmap v1682: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:34.241 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:33 smithi172 ceph-mon[100696]: pgmap v1682: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:35.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:35 smithi154 ceph-mon[102314]: pgmap v1683: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:35.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:35 smithi154 ceph-mon[107038]: pgmap v1683: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:35.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:35 smithi172 ceph-mon[100696]: pgmap v1683: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:35.769 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:15:35.769 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":525,"fsid":"447b0cd2-71db-11ef-bceb-c7b262605968","created":"2024-09-13T14:21:30.909284+0000","modified":"2024-09-13T15:15:29.827516+0000","last_up_change":"2024-09-13T15:00:14.027470+0000","last_in_change":"2024-09-13T15:08:30.165081+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T14:24:31.633615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"506","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"502","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T14:26:59.738789+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T14:28:12.645576+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T14:28:13.527124+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fa9860db-d569-4402-8a9e-c41b083f2042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":93,"last_clean_end":301,"up_from":321,"up_thru":506,"down_at":301,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6803","nonce":3127717424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6805","nonce":3127717424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6809","nonce":3127717424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":3127717424},{"type":"v1","addr":"172.21.15.154:6807","nonce":3127717424}]},"public_addr":"172.21.15.154:6803/3127717424","cluster_addr":"172.21.15.154:6805/3127717424","heartbeat_back_addr":"172.21.15.154:6809/3127717424","heartbeat_front_addr":"172.21.15.154:6807/3127717424","state":["exists","up"]},{"osd":1,"uuid":"539dd807-6e38-4237-8d23-28b57f0f1b4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":94,"up_from":98,"up_thru":507,"down_at":95,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6811","nonce":148745834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6813","nonce":148745834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6817","nonce":148745834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":148745834},{"type":"v1","addr":"172.21.15.154:6815","nonce":148745834}]},"public_addr":"172.21.15.154:6811/148745834","cluster_addr":"172.21.15.154:6813/148745834","heartbeat_back_addr":"172.21.15.154:6817/148745834","heartbeat_front_addr":"172.21.15.154:6815/148745834","state":["exists","up"]},{"osd":2,"uuid":"ba93fa41-fa08-4d94-b499-149ded9151f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":103,"last_clean_end":280,"up_from":328,"up_thru":506,"down_at":281,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6819","nonce":2560992094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6821","nonce":2560992094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6825","nonce":2560992094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2560992094},{"type":"v1","addr":"172.21.15.154:6823","nonce":2560992094}]},"public_addr":"172.21.15.154:6819/2560992094","cluster_addr":"172.21.15.154:6821/2560992094","heartbeat_back_addr":"172.21.15.154:6825/2560992094","heartbeat_front_addr":"172.21.15.154:6823/2560992094","state":["exists","up"]},{"osd":3,"uuid":"4422727c-268e-4356-83fe-118d25e6cb4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":104,"up_from":108,"up_thru":507,"down_at":105,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6827","nonce":381867861}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6829","nonce":381867861}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6833","nonce":381867861}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":381867861},{"type":"v1","addr":"172.21.15.154:6831","nonce":381867861}]},"public_addr":"172.21.15.154:6827/381867861","cluster_addr":"172.21.15.154:6829/381867861","heartbeat_back_addr":"172.21.15.154:6833/381867861","heartbeat_front_addr":"172.21.15.154:6831/381867861","state":["exists","up"]},{"osd":4,"uuid":"9d3372f5-f7f2-4798-a221-5adc2acfc3c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6801","nonce":822621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6803","nonce":822621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6807","nonce":822621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":822621689},{"type":"v1","addr":"172.21.15.172:6805","nonce":822621689}]},"public_addr":"172.21.15.172:6801/822621689","cluster_addr":"172.21.15.172:6803/822621689","heartbeat_back_addr":"172.21.15.172:6807/822621689","heartbeat_front_addr":"172.21.15.172:6805/822621689","state":["exists","up"]},{"osd":5,"uuid":"6458ef05-8c2f-4ef0-a0a1-080908418a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6809","nonce":1992976433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6811","nonce":1992976433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6815","nonce":1992976433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1992976433},{"type":"v1","addr":"172.21.15.172:6813","nonce":1992976433}]},"public_addr":"172.21.15.172:6809/1992976433","cluster_addr":"172.21.15.172:6811/1992976433","heartbeat_back_addr":"172.21.15.172:6815/1992976433","heartbeat_front_addr":"172.21.15.172:6813/1992976433","state":["exists","up"]},{"osd":6,"uuid":"8e768676-0ffa-46e3-8a03-db3fdde2951c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6817","nonce":1801776288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6819","nonce":1801776288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6823","nonce":1801776288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":1801776288},{"type":"v1","addr":"172.21.15.172:6821","nonce":1801776288}]},"public_addr":"172.21.15.172:6817/1801776288","cluster_addr":"172.21.15.172:6819/1801776288","heartbeat_back_addr":"172.21.15.172:6823/1801776288","heartbeat_front_addr":"172.21.15.172:6821/1801776288","state":["exists","up"]},{"osd":7,"uuid":"ef36cdfb-77d4-4974-aa46-d64d79a2b0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":507,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6825","nonce":173465567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6827","nonce":173465567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6831","nonce":173465567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":173465567},{"type":"v1","addr":"172.21.15.172:6829","nonce":173465567}]},"public_addr":"172.21.15.172:6825/173465567","cluster_addr":"172.21.15.172:6827/173465567","heartbeat_back_addr":"172.21.15.172:6831/173465567","heartbeat_front_addr":"172.21.15.172:6829/173465567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:23:41.598523+0000","dead_epoch":300},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:05.085893+0000","dead_epoch":94},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:28.734197+0000","dead_epoch":280},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:24:52.968191+0000","dead_epoch":104},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:14.326426+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:25:37.551557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:00.968344+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T14:26:24.546567+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":0}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":3,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:0/3255496958":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:0/4103428848":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/2090383084":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/1584830159":"2024-09-14T14:34:12.641827+0000","172.21.15.154:6801/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/18164567":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6800/544101962":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/3191030600":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1349885828":"2024-09-14T14:26:57.118038+0000","172.21.15.154:6800/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/4127949890":"2024-09-14T14:22:00.624052+0000","172.21.15.154:6800/557082616":"2024-09-14T14:22:00.624052+0000","172.21.15.172:6835/3129749965":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/1231123258":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/548682894":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/3579365666":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/966915265":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6800/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.154:6801/302860313":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/213038927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/2810096914":"2024-09-14T14:22:23.331946+0000","172.21.15.154:0/3680637464":"2024-09-14T14:34:12.641827+0000","172.21.15.154:0/3924873105":"2024-09-14T14:31:04.278426+0000","172.21.15.154:0/3142124210":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/2189531829":"2024-09-14T14:34:12.641827+0000","172.21.15.172:0/2607163146":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/490371939":"2024-09-14T14:22:00.624052+0000","172.21.15.154:0/1379294727":"2024-09-14T14:31:04.278426+0000","172.21.15.172:0/2289953927":"2024-09-14T14:31:33.289382+0000","172.21.15.154:0/3482226595":"2024-09-14T14:26:57.118038+0000","172.21.15.154:0/106598025":"2024-09-14T14:26:57.118038+0000","172.21.15.172:0/3573827628":"2024-09-14T14:31:33.289382+0000","172.21.15.154:6800/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.154:6801/3662635163":"2024-09-14T14:22:23.331946+0000","172.21.15.172:6834/3129749965":"2024-09-14T14:31:33.289382+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:15:36.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T15:15:36.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T15:15:36.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T15:15:36.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T15:15:36.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T15:15:36.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T15:15:36.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T15:15:36.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T15:15:36.482 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:36 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/970786373' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:15:36.483 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:36 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/970786373' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:15:36.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:36 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/970786373' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-13T15:15:36.876 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:36.876 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:36.877 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:36.882 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:36.882 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:36.882 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:36.887 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:36.900 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:37.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:37 smithi154 ceph-mon[102314]: pgmap v1684: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:37.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:37 smithi154 ceph-mon[107038]: pgmap v1684: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:37.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:37 smithi172 ceph-mon[100696]: pgmap v1684: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:39.706 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:39 smithi154 ceph-mon[102314]: pgmap v1685: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:39.707 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:39 smithi154 ceph-mon[107038]: pgmap v1685: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:39.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:39 smithi172 ceph-mon[100696]: pgmap v1685: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:41.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:41 smithi154 ceph-mon[102314]: pgmap v1686: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:41.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:41 smithi154 ceph-mon[107038]: pgmap v1686: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:41.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:41 smithi172 ceph-mon[100696]: pgmap v1686: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:42.213 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:15:41 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:15:41] "GET /metrics HTTP/1.1" 200 34228 "" "Prometheus/2.51.0" 2024-09-13T15:15:43.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:43 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:43.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:43 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:43.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:43 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:44.406 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:44 smithi154 ceph-mon[102314]: pgmap v1687: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:44.407 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:44 smithi154 ceph-mon[107038]: pgmap v1687: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:44.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:44 smithi172 ceph-mon[100696]: pgmap v1687: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:45.714 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:45 smithi154 ceph-mon[102314]: pgmap v1688: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:45.715 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:45 smithi154 ceph-mon[107038]: pgmap v1688: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:45.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:45 smithi172 ceph-mon[100696]: pgmap v1688: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:47.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:47 smithi154 ceph-mon[102314]: pgmap v1689: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:47.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:47 smithi154 ceph-mon[107038]: pgmap v1689: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:47.635 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:47 smithi172 ceph-mon[100696]: pgmap v1689: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:48.750 INFO:teuthology.orchestra.run.smithi154.stdout:1378684502210 2024-09-13T15:15:48.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T15:15:49.117 INFO:teuthology.orchestra.run.smithi154.stdout:420906795428 2024-09-13T15:15:49.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T15:15:49.721 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:49 smithi154 ceph-mon[102314]: pgmap v1690: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:49.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:49 smithi154 ceph-mon[107038]: pgmap v1690: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:49.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:49 smithi172 ceph-mon[100696]: pgmap v1690: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:51.660 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:51 smithi154 ceph-mon[102314]: pgmap v1691: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:51.661 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:51 smithi154 ceph-mon[107038]: pgmap v1691: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:51.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:51 smithi172 ceph-mon[100696]: pgmap v1691: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:51.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:15:51 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:15:51] "GET /metrics HTTP/1.1" 200 34225 "" "Prometheus/2.51.0" 2024-09-13T15:15:52.043 INFO:teuthology.orchestra.run.smithi154.stdout:163208757853 2024-09-13T15:15:52.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T15:15:53.599 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:53 smithi154 ceph-mon[102314]: pgmap v1692: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:53.599 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:53 smithi154 ceph-mon[107038]: pgmap v1692: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:53.620 INFO:teuthology.orchestra.run.smithi154.stdout:463856468377 2024-09-13T15:15:53.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T15:15:53.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:53 smithi172 ceph-mon[100696]: pgmap v1692: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:54.871 INFO:teuthology.orchestra.run.smithi154.stdout:197568496219 2024-09-13T15:15:54.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T15:15:55.257 INFO:teuthology.orchestra.run.smithi154.stdout:1408749273277 2024-09-13T15:15:55.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T15:15:55.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:55 smithi154 ceph-mon[102314]: pgmap v1693: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:55.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:55 smithi154 ceph-mon[107038]: pgmap v1693: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:55.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:55 smithi172 ceph-mon[100696]: pgmap v1693: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:55.749 INFO:teuthology.orchestra.run.smithi154.stdout:128849019493 2024-09-13T15:15:55.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T15:15:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:56.036 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:56.036 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:56.457 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:56.458 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:56.458 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:56.984 INFO:teuthology.orchestra.run.smithi154.stdout:231928234582 2024-09-13T15:15:56.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T15:15:57.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:57 smithi172 ceph-mon[100696]: pgmap v1694: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:57.627 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:57 smithi154 ceph-mon[102314]: pgmap v1694: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:57.627 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:57 smithi154 ceph-mon[107038]: pgmap v1694: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:58.131 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:58.554 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:58 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:58.554 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:58 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:58.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:58 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:15:58.783 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:15:58.944 INFO:teuthology.orchestra.run.smithi154.stdout:197568496219 2024-09-13T15:15:59.661 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:59 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/273799760' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:15:59.661 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:15:59 smithi154 ceph-mon[107038]: pgmap v1695: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:59.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:59 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/273799760' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:15:59.661 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:15:59 smithi154 ceph-mon[102314]: pgmap v1695: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:15:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:59 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/273799760' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-13T15:15:59.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:15:59 smithi172 ceph-mon[100696]: pgmap v1695: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:01.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:01 smithi154 ceph-mon[102314]: pgmap v1696: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:01.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:01 smithi154 ceph-mon[107038]: pgmap v1696: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:01.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:01 smithi172 ceph-mon[100696]: pgmap v1696: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:02.114 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:01 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:16:01] "GET /metrics HTTP/1.1" 200 34228 "" "Prometheus/2.51.0" 2024-09-13T15:16:02.571 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568496219 got 197568496219 for osd.6 2024-09-13T15:16:02.572 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:03.646 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:03 smithi154 ceph-mon[102314]: pgmap v1697: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:03.646 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:03 smithi154 ceph-mon[107038]: pgmap v1697: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:03.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:03 smithi172 ceph-mon[100696]: pgmap v1697: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:05.119 INFO:teuthology.orchestra.run.smithi154.stdout:231928234584 2024-09-13T15:16:05.119 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:05 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/864997556' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:16:05.120 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:05 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/864997556' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:16:05.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:05 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/864997556' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-13T15:16:05.562 INFO:teuthology.orchestra.run.smithi154.stdout:1408749273279 2024-09-13T15:16:06.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/872225609' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:16:06.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:06 smithi154 ceph-mon[102314]: pgmap v1698: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:06.365 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:06 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3843254629' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:16:06.366 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/872225609' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:16:06.366 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:06 smithi154 ceph-mon[107038]: pgmap v1698: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:06.366 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:06 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3843254629' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:16:06.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234582 got 231928234584 for osd.7 2024-09-13T15:16:06.388 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/872225609' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-13T15:16:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:06 smithi172 ceph-mon[100696]: pgmap v1698: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:06.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:06 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3843254629' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-13T15:16:07.414 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:07 smithi154 ceph-mon[102314]: pgmap v1699: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:07.414 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:07 smithi154 ceph-mon[107038]: pgmap v1699: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:07.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:07 smithi172 ceph-mon[100696]: pgmap v1699: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:08.538 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1408749273277 got 1408749273279 for osd.2 2024-09-13T15:16:08.538 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:08.631 INFO:teuthology.orchestra.run.smithi154.stdout:163208757856 2024-09-13T15:16:08.720 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:08 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3344206269' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:16:08.721 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:08 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3344206269' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:16:08.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:08 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3344206269' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-13T15:16:08.909 INFO:teuthology.orchestra.run.smithi154.stdout:1378684502214 2024-09-13T15:16:09.470 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:09 smithi154 ceph-mon[102314]: pgmap v1700: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:09.471 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:09 smithi154 ceph-mon[107038]: pgmap v1700: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:09.639 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1378684502210 got 1378684502214 for osd.0 2024-09-13T15:16:09.640 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:09.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:09 smithi172 ceph-mon[100696]: pgmap v1700: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:10.958 INFO:teuthology.orchestra.run.smithi154.stdout:420906795433 2024-09-13T15:16:11.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:11 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/2741727646' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:16:11.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:11 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/2741727646' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:16:11.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:11 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/2741727646' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-13T15:16:11.734 INFO:tasks.cephadm.ceph_manager.ceph:need seq 420906795428 got 420906795433 for osd.1 2024-09-13T15:16:11.735 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:11.970 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:11 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:16:11] "GET /metrics HTTP/1.1" 200 34228 "" "Prometheus/2.51.0" 2024-09-13T15:16:12.224 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757853 got 163208757856 for osd.5 2024-09-13T15:16:12.224 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:12.357 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:12 smithi154 ceph-mon[102314]: pgmap v1701: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:12.357 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:12 smithi154 ceph-mon[107038]: pgmap v1701: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:12.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:12 smithi172 ceph-mon[100696]: pgmap v1701: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:12.854 INFO:teuthology.orchestra.run.smithi154.stdout:463856468381 2024-09-13T15:16:13.086 INFO:teuthology.orchestra.run.smithi154.stdout:128849019497 2024-09-13T15:16:13.441 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:13 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/3479831522' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:16:13.441 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:13 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/3479831522' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:16:13.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:13 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/3479831522' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-13T15:16:13.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 463856468377 got 463856468381 for osd.3 2024-09-13T15:16:13.805 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:14.220 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:14 smithi154 ceph-mon[102314]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:16:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:14 smithi154 ceph-mon[102314]: from='client.? 172.21.15.154:0/962160008' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:16:14.221 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:14 smithi154 ceph-mon[102314]: pgmap v1702: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:14 smithi154 ceph-mon[107038]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:16:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:14 smithi154 ceph-mon[107038]: from='client.? 172.21.15.154:0/962160008' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:16:14.221 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:14 smithi154 ceph-mon[107038]: pgmap v1702: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:14.294 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019493 got 128849019497 for osd.4 2024-09-13T15:16:14.294 DEBUG:teuthology.parallel:result is None 2024-09-13T15:16:14.294 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T15:16:14.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:16:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:14 smithi172 ceph-mon[100696]: from='mgr.25736 172.21.15.154:0/3193919066' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-13T15:16:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:14 smithi172 ceph-mon[100696]: from='client.? 172.21.15.154:0/962160008' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-13T15:16:14.491 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:14 smithi172 ceph-mon[100696]: pgmap v1702: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:14.497 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:16:15.649 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:15 smithi154 ceph-mon[102314]: pgmap v1703: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:15.649 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:15 smithi154 ceph-mon[107038]: pgmap v1703: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:15.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:15 smithi172 ceph-mon[100696]: pgmap v1703: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:16:16.470 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:16:17.393 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1703,"stamp":"2024-09-13T15:16:15.308375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":36635,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5652,"ondisk_log_size":5652,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5901032,"kb_used_data":68552,"kb_used_omap":40,"kb_used_meta":5832343,"kb_avail":744060184,"statfs":{"total":767960285184,"available":761917628416,"internally_reserved":0,"allocated":70197248,"data_stored":47749479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":5972319348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002946"},"pg_stats":[{"pgid":"1.11","version":"89'149","reported_seq":926,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.806726+0000","last_change":"2024-09-13T15:13:01.179220+0000","last_active":"2024-09-13T15:15:30.806726+0000","last_peered":"2024-09-13T15:15:30.806726+0000","last_clean":"2024-09-13T15:15:30.806726+0000","last_became_active":"2024-09-13T15:12:50.169695+0000","last_became_peered":"2024-09-13T15:12:50.169695+0000","last_unstale":"2024-09-13T15:15:30.806726+0000","last_undegraded":"2024-09-13T15:15:30.806726+0000","last_fullsized":"2024-09-13T15:15:30.806726+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.179154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.179154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.084137+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":275,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807241+0000","last_change":"2024-09-13T15:08:31.190995+0000","last_active":"2024-09-13T15:15:30.807241+0000","last_peered":"2024-09-13T15:15:30.807241+0000","last_clean":"2024-09-13T15:15:30.807241+0000","last_became_active":"2024-09-13T15:08:31.190610+0000","last_became_peered":"2024-09-13T15:08:31.190610+0000","last_unstale":"2024-09-13T15:15:30.807241+0000","last_undegraded":"2024-09-13T15:15:30.807241+0000","last_fullsized":"2024-09-13T15:15:30.807241+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:42.124757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807818+0000","last_change":"2024-09-13T15:08:31.191585+0000","last_active":"2024-09-13T15:15:30.807818+0000","last_peered":"2024-09-13T15:15:30.807818+0000","last_clean":"2024-09-13T15:15:30.807818+0000","last_became_active":"2024-09-13T15:08:31.184700+0000","last_became_peered":"2024-09-13T15:08:31.184700+0000","last_unstale":"2024-09-13T15:15:30.807818+0000","last_undegraded":"2024-09-13T15:15:30.807818+0000","last_fullsized":"2024-09-13T15:15:30.807818+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:14.376834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"89'149","reported_seq":865,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549466+0000","last_change":"2024-09-13T15:13:09.203525+0000","last_active":"2024-09-13T15:15:27.549466+0000","last_peered":"2024-09-13T15:15:27.549466+0000","last_clean":"2024-09-13T15:15:27.549466+0000","last_became_active":"2024-09-13T15:12:50.159713+0000","last_became_peered":"2024-09-13T15:12:50.159713+0000","last_unstale":"2024-09-13T15:15:27.549466+0000","last_undegraded":"2024-09-13T15:15:27.549466+0000","last_fullsized":"2024-09-13T15:15:27.549466+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:09.203387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:09.203387+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:07.004095+0000","scrub_duration":0.002438127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":384,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549969+0000","last_change":"2024-09-13T15:08:46.080613+0000","last_active":"2024-09-13T15:15:27.549969+0000","last_peered":"2024-09-13T15:15:27.549969+0000","last_clean":"2024-09-13T15:15:27.549969+0000","last_became_active":"2024-09-13T15:08:46.080320+0000","last_became_peered":"2024-09-13T15:08:46.080320+0000","last_unstale":"2024-09-13T15:15:27.549969+0000","last_undegraded":"2024-09-13T15:15:27.549969+0000","last_fullsized":"2024-09-13T15:15:27.549969+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:09.205609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":889,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807654+0000","last_change":"2024-09-13T15:08:31.196449+0000","last_active":"2024-09-13T15:15:30.807654+0000","last_peered":"2024-09-13T15:15:30.807654+0000","last_clean":"2024-09-13T15:15:30.807654+0000","last_became_active":"2024-09-13T15:08:31.196143+0000","last_became_peered":"2024-09-13T15:08:31.196143+0000","last_unstale":"2024-09-13T15:15:30.807654+0000","last_undegraded":"2024-09-13T15:15:30.807654+0000","last_fullsized":"2024-09-13T15:15:30.807654+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:53.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"89'149","reported_seq":980,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833051+0000","last_change":"2024-09-13T15:13:03.099175+0000","last_active":"2024-09-13T15:15:29.833051+0000","last_peered":"2024-09-13T15:15:29.833051+0000","last_clean":"2024-09-13T15:15:29.833051+0000","last_became_active":"2024-09-13T15:12:50.196974+0000","last_became_peered":"2024-09-13T15:12:50.196974+0000","last_unstale":"2024-09-13T15:15:29.833051+0000","last_undegraded":"2024-09-13T15:15:29.833051+0000","last_fullsized":"2024-09-13T15:15:29.833051+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.098941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.098941+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:06.828156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":892,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807792+0000","last_change":"2024-09-13T15:08:31.195499+0000","last_active":"2024-09-13T15:15:30.807792+0000","last_peered":"2024-09-13T15:15:30.807792+0000","last_clean":"2024-09-13T15:15:30.807792+0000","last_became_active":"2024-09-13T15:08:31.194928+0000","last_became_peered":"2024-09-13T15:08:31.194928+0000","last_unstale":"2024-09-13T15:15:30.807792+0000","last_undegraded":"2024-09-13T15:15:30.807792+0000","last_fullsized":"2024-09-13T15:15:30.807792+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.112396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807614+0000","last_change":"2024-09-13T15:08:31.185658+0000","last_active":"2024-09-13T15:15:30.807614+0000","last_peered":"2024-09-13T15:15:30.807614+0000","last_clean":"2024-09-13T15:15:30.807614+0000","last_became_active":"2024-09-13T15:08:31.185202+0000","last_became_peered":"2024-09-13T15:08:31.185202+0000","last_unstale":"2024-09-13T15:15:30.807614+0000","last_undegraded":"2024-09-13T15:15:30.807614+0000","last_fullsized":"2024-09-13T15:15:30.807614+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:04.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"89'149","reported_seq":865,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549698+0000","last_change":"2024-09-13T15:13:11.154190+0000","last_active":"2024-09-13T15:15:27.549698+0000","last_peered":"2024-09-13T15:15:27.549698+0000","last_clean":"2024-09-13T15:15:27.549698+0000","last_became_active":"2024-09-13T15:12:50.159857+0000","last_became_peered":"2024-09-13T15:12:50.159857+0000","last_unstale":"2024-09-13T15:15:27.549698+0000","last_undegraded":"2024-09-13T15:15:27.549698+0000","last_fullsized":"2024-09-13T15:15:27.549698+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.153852+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.153852+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:54.235242+0000","scrub_duration":0.0043281400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":396,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463814+0000","last_change":"2024-09-13T15:08:31.199682+0000","last_active":"2024-09-13T15:15:30.463814+0000","last_peered":"2024-09-13T15:15:30.463814+0000","last_clean":"2024-09-13T15:15:30.463814+0000","last_became_active":"2024-09-13T15:08:31.199110+0000","last_became_peered":"2024-09-13T15:08:31.199110+0000","last_unstale":"2024-09-13T15:15:30.463814+0000","last_undegraded":"2024-09-13T15:15:30.463814+0000","last_fullsized":"2024-09-13T15:15:30.463814+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:30.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":162,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249434+0000","last_change":"2024-09-13T15:15:10.378633+0000","last_active":"2024-09-13T15:15:27.249434+0000","last_peered":"2024-09-13T15:15:27.249434+0000","last_clean":"2024-09-13T15:15:27.249434+0000","last_became_active":"2024-09-13T15:15:10.378285+0000","last_became_peered":"2024-09-13T15:15:10.378285+0000","last_unstale":"2024-09-13T15:15:27.249434+0000","last_undegraded":"2024-09-13T15:15:27.249434+0000","last_fullsized":"2024-09-13T15:15:27.249434+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:26.676056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"89'149","reported_seq":886,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879021+0000","last_change":"2024-09-13T15:14:30.804233+0000","last_active":"2024-09-13T15:15:27.879021+0000","last_peered":"2024-09-13T15:15:27.879021+0000","last_clean":"2024-09-13T15:15:27.879021+0000","last_became_active":"2024-09-13T15:14:28.397907+0000","last_became_peered":"2024-09-13T15:14:28.397907+0000","last_unstale":"2024-09-13T15:15:27.879021+0000","last_undegraded":"2024-09-13T15:15:27.879021+0000","last_fullsized":"2024-09-13T15:15:27.879021+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.804126+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:14:30.804126+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.804126+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:37.618332+0000","scrub_duration":0.0015459549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":104,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878505+0000","last_change":"2024-09-13T15:08:31.351504+0000","last_active":"2024-09-13T15:15:27.878505+0000","last_peered":"2024-09-13T15:15:27.878505+0000","last_clean":"2024-09-13T15:15:27.878505+0000","last_became_active":"2024-09-13T15:08:31.320950+0000","last_became_peered":"2024-09-13T15:08:31.320950+0000","last_unstale":"2024-09-13T15:15:27.878505+0000","last_undegraded":"2024-09-13T15:15:27.878505+0000","last_fullsized":"2024-09-13T15:15:27.878505+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:38.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":430,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549988+0000","last_change":"2024-09-13T15:08:46.078296+0000","last_active":"2024-09-13T15:15:27.549988+0000","last_peered":"2024-09-13T15:15:27.549988+0000","last_clean":"2024-09-13T15:15:27.549988+0000","last_became_active":"2024-09-13T15:08:46.078037+0000","last_became_peered":"2024-09-13T15:08:46.078037+0000","last_unstale":"2024-09-13T15:15:27.549988+0000","last_undegraded":"2024-09-13T15:15:27.549988+0000","last_fullsized":"2024-09-13T15:15:27.549988+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:49.959361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"89'149","reported_seq":1002,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463916+0000","last_change":"2024-09-13T15:14:30.238997+0000","last_active":"2024-09-13T15:15:30.463916+0000","last_peered":"2024-09-13T15:15:30.463916+0000","last_clean":"2024-09-13T15:15:30.463916+0000","last_became_active":"2024-09-13T15:14:28.395022+0000","last_became_peered":"2024-09-13T15:14:28.395022+0000","last_unstale":"2024-09-13T15:15:30.463916+0000","last_undegraded":"2024-09-13T15:15:30.463916+0000","last_fullsized":"2024-09-13T15:15:30.463916+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.238938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.238938+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:37.186365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":104,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878573+0000","last_change":"2024-09-13T15:08:31.201939+0000","last_active":"2024-09-13T15:15:27.878573+0000","last_peered":"2024-09-13T15:15:27.878573+0000","last_clean":"2024-09-13T15:15:27.878573+0000","last_became_active":"2024-09-13T15:08:31.191089+0000","last_became_peered":"2024-09-13T15:08:31.191089+0000","last_unstale":"2024-09-13T15:15:27.878573+0000","last_undegraded":"2024-09-13T15:15:27.878573+0000","last_fullsized":"2024-09-13T15:15:27.878573+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:06.249945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":594,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.507959+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:15:30.507959+0000","last_peered":"2024-09-13T15:15:30.507959+0000","last_clean":"2024-09-13T15:15:30.507959+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:15:30.507959+0000","last_undegraded":"2024-09-13T15:15:30.507959+0000","last_fullsized":"2024-09-13T15:15:30.507959+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"89'149","reported_seq":780,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.250055+0000","last_change":"2024-09-13T15:15:14.337399+0000","last_active":"2024-09-13T15:15:27.250055+0000","last_peered":"2024-09-13T15:15:27.250055+0000","last_clean":"2024-09-13T15:15:27.250055+0000","last_became_active":"2024-09-13T15:15:10.378910+0000","last_became_peered":"2024-09-13T15:15:10.378910+0000","last_unstale":"2024-09-13T15:15:27.250055+0000","last_undegraded":"2024-09-13T15:15:27.250055+0000","last_fullsized":"2024-09-13T15:15:27.250055+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:15:14.337271+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:15:14.337271+0000","last_clean_scrub_stamp":"2024-09-13T15:15:14.337271+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:45.029466+0000","scrub_duration":0.001704267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":104,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879247+0000","last_change":"2024-09-13T15:08:31.199736+0000","last_active":"2024-09-13T15:15:27.879247+0000","last_peered":"2024-09-13T15:15:27.879247+0000","last_clean":"2024-09-13T15:15:27.879247+0000","last_became_active":"2024-09-13T15:08:31.186018+0000","last_became_peered":"2024-09-13T15:08:31.186018+0000","last_unstale":"2024-09-13T15:15:27.879247+0000","last_undegraded":"2024-09-13T15:15:27.879247+0000","last_fullsized":"2024-09-13T15:15:27.879247+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:26.855187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":460,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249631+0000","last_change":"2024-09-13T15:15:11.072923+0000","last_active":"2024-09-13T15:15:27.249631+0000","last_peered":"2024-09-13T15:15:27.249631+0000","last_clean":"2024-09-13T15:15:27.249631+0000","last_became_active":"2024-09-13T15:15:11.072381+0000","last_became_peered":"2024-09-13T15:15:11.072381+0000","last_unstale":"2024-09-13T15:15:27.249631+0000","last_undegraded":"2024-09-13T15:15:27.249631+0000","last_fullsized":"2024-09-13T15:15:27.249631+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:43.491830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"89'149","reported_seq":955,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.112022+0000","last_change":"2024-09-13T15:14:30.760086+0000","last_active":"2024-09-13T15:15:27.112022+0000","last_peered":"2024-09-13T15:15:27.112022+0000","last_clean":"2024-09-13T15:15:27.112022+0000","last_became_active":"2024-09-13T15:14:28.397451+0000","last_became_peered":"2024-09-13T15:14:28.397451+0000","last_unstale":"2024-09-13T15:15:27.112022+0000","last_undegraded":"2024-09-13T15:15:27.112022+0000","last_fullsized":"2024-09-13T15:15:27.112022+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.759981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.759981+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:42.322864+0000","scrub_duration":0.00187911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":401,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833994+0000","last_change":"2024-09-13T15:08:46.073352+0000","last_active":"2024-09-13T15:15:29.833994+0000","last_peered":"2024-09-13T15:15:29.833994+0000","last_clean":"2024-09-13T15:15:29.833994+0000","last_became_active":"2024-09-13T15:08:46.073048+0000","last_became_peered":"2024-09-13T15:08:46.073048+0000","last_unstale":"2024-09-13T15:15:29.833994+0000","last_undegraded":"2024-09-13T15:15:29.833994+0000","last_fullsized":"2024-09-13T15:15:29.833994+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:57.277685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":801,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807205+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:15:30.807205+0000","last_peered":"2024-09-13T15:15:30.807205+0000","last_clean":"2024-09-13T15:15:30.807205+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:15:30.807205+0000","last_undegraded":"2024-09-13T15:15:30.807205+0000","last_fullsized":"2024-09-13T15:15:30.807205+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878886+0000","last_change":"2024-09-13T15:14:29.775519+0000","last_active":"2024-09-13T15:15:27.878886+0000","last_peered":"2024-09-13T15:15:27.878886+0000","last_clean":"2024-09-13T15:15:27.878886+0000","last_became_active":"2024-09-13T15:14:28.396999+0000","last_became_peered":"2024-09-13T15:14:28.396999+0000","last_unstale":"2024-09-13T15:15:27.878886+0000","last_undegraded":"2024-09-13T15:15:27.878886+0000","last_fullsized":"2024-09-13T15:15:27.878886+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:29.775419+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:14:29.775419+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:01.928165+0000","scrub_duration":0.0020314999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":930,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463536+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:15:30.463536+0000","last_peered":"2024-09-13T15:15:30.463536+0000","last_clean":"2024-09-13T15:15:30.463536+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:15:30.463536+0000","last_undegraded":"2024-09-13T15:15:30.463536+0000","last_fullsized":"2024-09-13T15:15:30.463536+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":536,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508140+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:15:30.508140+0000","last_peered":"2024-09-13T15:15:30.508140+0000","last_clean":"2024-09-13T15:15:30.508140+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:15:30.508140+0000","last_undegraded":"2024-09-13T15:15:30.508140+0000","last_fullsized":"2024-09-13T15:15:30.508140+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"89'149","reported_seq":905,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549722+0000","last_change":"2024-09-13T15:14:34.801562+0000","last_active":"2024-09-13T15:15:27.549722+0000","last_peered":"2024-09-13T15:15:27.549722+0000","last_clean":"2024-09-13T15:15:27.549722+0000","last_became_active":"2024-09-13T15:14:28.398879+0000","last_became_peered":"2024-09-13T15:14:28.398879+0000","last_unstale":"2024-09-13T15:15:27.549722+0000","last_undegraded":"2024-09-13T15:15:27.549722+0000","last_fullsized":"2024-09-13T15:15:27.549722+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:34.801454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:34.801454+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:43.731723+0000","scrub_duration":0.0017318050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":171,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249809+0000","last_change":"2024-09-13T15:15:11.072536+0000","last_active":"2024-09-13T15:15:27.249809+0000","last_peered":"2024-09-13T15:15:27.249809+0000","last_clean":"2024-09-13T15:15:27.249809+0000","last_became_active":"2024-09-13T15:15:11.072203+0000","last_became_peered":"2024-09-13T15:15:11.072203+0000","last_unstale":"2024-09-13T15:15:27.249809+0000","last_undegraded":"2024-09-13T15:15:27.249809+0000","last_fullsized":"2024-09-13T15:15:27.249809+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:42.437845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":530,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508524+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:15:30.508524+0000","last_peered":"2024-09-13T15:15:30.508524+0000","last_clean":"2024-09-13T15:15:30.508524+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:15:30.508524+0000","last_undegraded":"2024-09-13T15:15:30.508524+0000","last_fullsized":"2024-09-13T15:15:30.508524+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"89'149","reported_seq":994,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833197+0000","last_change":"2024-09-13T15:14:29.402435+0000","last_active":"2024-09-13T15:15:29.833197+0000","last_peered":"2024-09-13T15:15:29.833197+0000","last_clean":"2024-09-13T15:15:29.833197+0000","last_became_active":"2024-09-13T15:14:28.396072+0000","last_became_peered":"2024-09-13T15:14:28.396072+0000","last_unstale":"2024-09-13T15:15:29.833197+0000","last_undegraded":"2024-09-13T15:15:29.833197+0000","last_fullsized":"2024-09-13T15:15:29.833197+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:29.402385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:29.402385+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:45.134708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":555,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508004+0000","last_change":"2024-09-13T15:08:31.184973+0000","last_active":"2024-09-13T15:15:30.508004+0000","last_peered":"2024-09-13T15:15:30.508004+0000","last_clean":"2024-09-13T15:15:30.508004+0000","last_became_active":"2024-09-13T15:08:31.184600+0000","last_became_peered":"2024-09-13T15:08:31.184600+0000","last_unstale":"2024-09-13T15:15:30.508004+0000","last_undegraded":"2024-09-13T15:15:30.508004+0000","last_fullsized":"2024-09-13T15:15:30.508004+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:09.452895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":563,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508480+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:15:30.508480+0000","last_peered":"2024-09-13T15:15:30.508480+0000","last_clean":"2024-09-13T15:15:30.508480+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:15:30.508480+0000","last_undegraded":"2024-09-13T15:15:30.508480+0000","last_fullsized":"2024-09-13T15:15:30.508480+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"89'149","reported_seq":905,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549597+0000","last_change":"2024-09-13T15:14:35.764139+0000","last_active":"2024-09-13T15:15:27.549597+0000","last_peered":"2024-09-13T15:15:27.549597+0000","last_clean":"2024-09-13T15:15:27.549597+0000","last_became_active":"2024-09-13T15:14:28.401152+0000","last_became_peered":"2024-09-13T15:14:28.401152+0000","last_unstale":"2024-09-13T15:15:27.549597+0000","last_undegraded":"2024-09-13T15:15:27.549597+0000","last_fullsized":"2024-09-13T15:15:27.549597+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:35.764037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:35.764037+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:59.340623+0000","scrub_duration":0.001781129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":396,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463177+0000","last_change":"2024-09-13T15:08:31.198602+0000","last_active":"2024-09-13T15:15:30.463177+0000","last_peered":"2024-09-13T15:15:30.463177+0000","last_clean":"2024-09-13T15:15:30.463177+0000","last_became_active":"2024-09-13T15:08:31.198289+0000","last_became_peered":"2024-09-13T15:08:31.198289+0000","last_unstale":"2024-09-13T15:15:30.463177+0000","last_undegraded":"2024-09-13T15:15:30.463177+0000","last_fullsized":"2024-09-13T15:15:30.463177+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:36.803833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":675,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.808008+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:15:30.808008+0000","last_peered":"2024-09-13T15:15:30.808008+0000","last_clean":"2024-09-13T15:15:30.808008+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:15:30.808008+0000","last_undegraded":"2024-09-13T15:15:30.808008+0000","last_fullsized":"2024-09-13T15:15:30.808008+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":910,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878161+0000","last_change":"2024-09-13T15:13:11.597792+0000","last_active":"2024-09-13T15:15:27.878161+0000","last_peered":"2024-09-13T15:15:27.878161+0000","last_clean":"2024-09-13T15:15:27.878161+0000","last_became_active":"2024-09-13T15:12:38.938735+0000","last_became_peered":"2024-09-13T15:12:38.938735+0000","last_unstale":"2024-09-13T15:15:27.878161+0000","last_undegraded":"2024-09-13T15:15:27.878161+0000","last_fullsized":"2024-09-13T15:15:27.878161+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.597623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.597623+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:14.909270+0000","scrub_duration":0.012675130999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1197,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834065+0000","last_change":"2024-09-13T15:08:57.728783+0000","last_active":"2024-09-13T15:15:29.834065+0000","last_peered":"2024-09-13T15:15:29.834065+0000","last_clean":"2024-09-13T15:15:29.834065+0000","last_became_active":"2024-09-13T15:08:47.077688+0000","last_became_peered":"2024-09-13T15:08:47.077688+0000","last_unstale":"2024-09-13T15:15:29.834065+0000","last_undegraded":"2024-09-13T15:15:29.834065+0000","last_fullsized":"2024-09-13T15:15:29.834065+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:52.710051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":892,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463291+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:15:30.463291+0000","last_peered":"2024-09-13T15:15:30.463291+0000","last_clean":"2024-09-13T15:15:30.463291+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:15:30.463291+0000","last_undegraded":"2024-09-13T15:15:30.463291+0000","last_fullsized":"2024-09-13T15:15:30.463291+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":299,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249868+0000","last_change":"2024-09-13T15:15:10.379913+0000","last_active":"2024-09-13T15:15:27.249868+0000","last_peered":"2024-09-13T15:15:27.249868+0000","last_clean":"2024-09-13T15:15:27.249868+0000","last_became_active":"2024-09-13T15:15:10.379437+0000","last_became_peered":"2024-09-13T15:15:10.379437+0000","last_unstale":"2024-09-13T15:15:27.249868+0000","last_undegraded":"2024-09-13T15:15:27.249868+0000","last_fullsized":"2024-09-13T15:15:27.249868+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:52.979549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"89'149","reported_seq":977,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833144+0000","last_change":"2024-09-13T15:13:04.107856+0000","last_active":"2024-09-13T15:15:29.833144+0000","last_peered":"2024-09-13T15:15:29.833144+0000","last_clean":"2024-09-13T15:15:29.833144+0000","last_became_active":"2024-09-13T15:12:38.929136+0000","last_became_peered":"2024-09-13T15:12:38.929136+0000","last_unstale":"2024-09-13T15:15:29.833144+0000","last_undegraded":"2024-09-13T15:15:29.833144+0000","last_fullsized":"2024-09-13T15:15:29.833144+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.107315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.107315+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:00.198499+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":118,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878394+0000","last_change":"2024-09-13T15:08:31.218115+0000","last_active":"2024-09-13T15:15:27.878394+0000","last_peered":"2024-09-13T15:15:27.878394+0000","last_clean":"2024-09-13T15:15:27.878394+0000","last_became_active":"2024-09-13T15:08:31.196118+0000","last_became_peered":"2024-09-13T15:08:31.196118+0000","last_unstale":"2024-09-13T15:15:27.878394+0000","last_undegraded":"2024-09-13T15:15:27.878394+0000","last_fullsized":"2024-09-13T15:15:27.878394+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:08.721976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":875,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.806930+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:15:30.806930+0000","last_peered":"2024-09-13T15:15:30.806930+0000","last_clean":"2024-09-13T15:15:30.806930+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:15:30.806930+0000","last_undegraded":"2024-09-13T15:15:30.806930+0000","last_fullsized":"2024-09-13T15:15:30.806930+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":870,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807710+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:15:30.807710+0000","last_peered":"2024-09-13T15:15:30.807710+0000","last_clean":"2024-09-13T15:15:30.807710+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:15:30.807710+0000","last_undegraded":"2024-09-13T15:15:30.807710+0000","last_fullsized":"2024-09-13T15:15:30.807710+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878262+0000","last_change":"2024-09-13T15:13:05.496011+0000","last_active":"2024-09-13T15:15:27.878262+0000","last_peered":"2024-09-13T15:15:27.878262+0000","last_clean":"2024-09-13T15:15:27.878262+0000","last_became_active":"2024-09-13T15:12:38.946687+0000","last_became_peered":"2024-09-13T15:12:38.946687+0000","last_unstale":"2024-09-13T15:15:27.878262+0000","last_undegraded":"2024-09-13T15:15:27.878262+0000","last_fullsized":"2024-09-13T15:15:27.878262+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:05.495826+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:05.495826+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:01.133250+0000","scrub_duration":0.011811428000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1312,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807351+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:15:30.807351+0000","last_peered":"2024-09-13T15:15:30.807351+0000","last_clean":"2024-09-13T15:15:30.807351+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:15:30.807351+0000","last_undegraded":"2024-09-13T15:15:30.807351+0000","last_fullsized":"2024-09-13T15:15:30.807351+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":941,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833030+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:15:29.833030+0000","last_peered":"2024-09-13T15:15:29.833030+0000","last_clean":"2024-09-13T15:15:29.833030+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:15:29.833030+0000","last_undegraded":"2024-09-13T15:15:29.833030+0000","last_fullsized":"2024-09-13T15:15:29.833030+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":679,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834031+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:15:29.834031+0000","last_peered":"2024-09-13T15:15:29.834031+0000","last_clean":"2024-09-13T15:15:29.834031+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:15:29.834031+0000","last_undegraded":"2024-09-13T15:15:29.834031+0000","last_fullsized":"2024-09-13T15:15:29.834031+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"89'149","reported_seq":905,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549438+0000","last_change":"2024-09-13T15:13:03.104893+0000","last_active":"2024-09-13T15:15:27.549438+0000","last_peered":"2024-09-13T15:15:27.549438+0000","last_clean":"2024-09-13T15:15:27.549438+0000","last_became_active":"2024-09-13T15:12:38.933486+0000","last_became_peered":"2024-09-13T15:12:38.933486+0000","last_unstale":"2024-09-13T15:15:27.549438+0000","last_undegraded":"2024-09-13T15:15:27.549438+0000","last_fullsized":"2024-09-13T15:15:27.549438+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.104648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.104648+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:41.623486+0000","scrub_duration":0.01681821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":451,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.131072+0000","last_change":"2024-09-13T15:08:46.074249+0000","last_active":"2024-09-13T15:15:29.131072+0000","last_peered":"2024-09-13T15:15:29.131072+0000","last_clean":"2024-09-13T15:15:29.131072+0000","last_became_active":"2024-09-13T15:08:46.073842+0000","last_became_peered":"2024-09-13T15:08:46.073842+0000","last_unstale":"2024-09-13T15:15:29.131072+0000","last_undegraded":"2024-09-13T15:15:29.131072+0000","last_fullsized":"2024-09-13T15:15:29.131072+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:44.854067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":556,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508483+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:15:30.508483+0000","last_peered":"2024-09-13T15:15:30.508483+0000","last_clean":"2024-09-13T15:15:30.508483+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:15:30.508483+0000","last_undegraded":"2024-09-13T15:15:30.508483+0000","last_fullsized":"2024-09-13T15:15:30.508483+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":368,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549773+0000","last_change":"2024-09-13T15:08:46.075360+0000","last_active":"2024-09-13T15:15:27.549773+0000","last_peered":"2024-09-13T15:15:27.549773+0000","last_clean":"2024-09-13T15:15:27.549773+0000","last_became_active":"2024-09-13T15:08:46.075046+0000","last_became_peered":"2024-09-13T15:08:46.075046+0000","last_unstale":"2024-09-13T15:15:27.549773+0000","last_undegraded":"2024-09-13T15:15:27.549773+0000","last_fullsized":"2024-09-13T15:15:27.549773+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.044150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"89'149","reported_seq":780,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249993+0000","last_change":"2024-09-13T15:15:23.391515+0000","last_active":"2024-09-13T15:15:27.249993+0000","last_peered":"2024-09-13T15:15:27.249993+0000","last_clean":"2024-09-13T15:15:27.249993+0000","last_became_active":"2024-09-13T15:15:10.377371+0000","last_became_peered":"2024-09-13T15:15:10.377371+0000","last_unstale":"2024-09-13T15:15:27.249993+0000","last_undegraded":"2024-09-13T15:15:27.249993+0000","last_fullsized":"2024-09-13T15:15:27.249993+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:15:23.391432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:15:23.391432+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:02.631052+0000","scrub_duration":0.0015151260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":401,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833768+0000","last_change":"2024-09-13T15:08:46.079918+0000","last_active":"2024-09-13T15:15:29.833768+0000","last_peered":"2024-09-13T15:15:29.833768+0000","last_clean":"2024-09-13T15:15:29.833768+0000","last_became_active":"2024-09-13T15:08:46.075051+0000","last_became_peered":"2024-09-13T15:08:46.075051+0000","last_unstale":"2024-09-13T15:15:29.833768+0000","last_undegraded":"2024-09-13T15:15:29.833768+0000","last_fullsized":"2024-09-13T15:15:29.833768+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:54.485416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1593,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833657+0000","last_change":"2024-09-13T15:08:59.341688+0000","last_active":"2024-09-13T15:15:29.833657+0000","last_peered":"2024-09-13T15:15:29.833657+0000","last_clean":"2024-09-13T15:15:29.833657+0000","last_became_active":"2024-09-13T15:08:59.340925+0000","last_became_peered":"2024-09-13T15:08:59.340925+0000","last_unstale":"2024-09-13T15:15:29.833657+0000","last_undegraded":"2024-09-13T15:15:29.833657+0000","last_fullsized":"2024-09-13T15:15:29.833657+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:22.966119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":520,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807507+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:15:30.807507+0000","last_peered":"2024-09-13T15:15:30.807507+0000","last_clean":"2024-09-13T15:15:30.807507+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:15:30.807507+0000","last_undegraded":"2024-09-13T15:15:30.807507+0000","last_fullsized":"2024-09-13T15:15:30.807507+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"89'149","reported_seq":780,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249887+0000","last_change":"2024-09-13T15:15:12.340589+0000","last_active":"2024-09-13T15:15:27.249887+0000","last_peered":"2024-09-13T15:15:27.249887+0000","last_clean":"2024-09-13T15:15:27.249887+0000","last_became_active":"2024-09-13T15:15:10.373811+0000","last_became_peered":"2024-09-13T15:15:10.373811+0000","last_unstale":"2024-09-13T15:15:27.249887+0000","last_undegraded":"2024-09-13T15:15:27.249887+0000","last_fullsized":"2024-09-13T15:15:27.249887+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:15:12.340451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:15:12.340451+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.025924+0000","scrub_duration":0.001977941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":933,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833581+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:15:29.833581+0000","last_peered":"2024-09-13T15:15:29.833581+0000","last_clean":"2024-09-13T15:15:29.833581+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:15:29.833581+0000","last_undegraded":"2024-09-13T15:15:29.833581+0000","last_fullsized":"2024-09-13T15:15:29.833581+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":118,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878555+0000","last_change":"2024-09-13T15:08:31.209459+0000","last_active":"2024-09-13T15:15:27.878555+0000","last_peered":"2024-09-13T15:15:27.878555+0000","last_clean":"2024-09-13T15:15:27.878555+0000","last_became_active":"2024-09-13T15:08:31.188811+0000","last_became_peered":"2024-09-13T15:08:31.188811+0000","last_unstale":"2024-09-13T15:15:27.878555+0000","last_undegraded":"2024-09-13T15:15:27.878555+0000","last_fullsized":"2024-09-13T15:15:27.878555+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.667223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":879,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807297+0000","last_change":"2024-09-13T15:08:31.197894+0000","last_active":"2024-09-13T15:15:30.807297+0000","last_peered":"2024-09-13T15:15:30.807297+0000","last_clean":"2024-09-13T15:15:30.807297+0000","last_became_active":"2024-09-13T15:08:31.197516+0000","last_became_peered":"2024-09-13T15:08:31.197516+0000","last_unstale":"2024-09-13T15:15:30.807297+0000","last_undegraded":"2024-09-13T15:15:30.807297+0000","last_fullsized":"2024-09-13T15:15:30.807297+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:42.650595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.112162+0000","last_change":"2024-09-13T15:14:29.757740+0000","last_active":"2024-09-13T15:15:27.112162+0000","last_peered":"2024-09-13T15:15:27.112162+0000","last_clean":"2024-09-13T15:15:27.112162+0000","last_became_active":"2024-09-13T15:14:28.395093+0000","last_became_peered":"2024-09-13T15:14:28.395093+0000","last_unstale":"2024-09-13T15:15:27.112162+0000","last_undegraded":"2024-09-13T15:15:27.112162+0000","last_fullsized":"2024-09-13T15:15:27.112162+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:29.757627+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:14:29.757627+0000","last_clean_scrub_stamp":"2024-09-13T15:14:29.757627+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:02.686292+0000","scrub_duration":0.001677331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":985,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463908+0000","last_change":"2024-09-13T15:08:31.202795+0000","last_active":"2024-09-13T15:15:30.463908+0000","last_peered":"2024-09-13T15:15:30.463908+0000","last_clean":"2024-09-13T15:15:30.463908+0000","last_became_active":"2024-09-13T15:08:31.199963+0000","last_became_peered":"2024-09-13T15:08:31.199963+0000","last_unstale":"2024-09-13T15:15:30.463908+0000","last_undegraded":"2024-09-13T15:15:30.463908+0000","last_fullsized":"2024-09-13T15:15:30.463908+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:08.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1419,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834073+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:15:29.834073+0000","last_peered":"2024-09-13T15:15:29.834073+0000","last_clean":"2024-09-13T15:15:29.834073+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:15:29.834073+0000","last_undegraded":"2024-09-13T15:15:29.834073+0000","last_fullsized":"2024-09-13T15:15:29.834073+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":690,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463232+0000","last_change":"2024-09-13T15:08:31.198729+0000","last_active":"2024-09-13T15:15:30.463232+0000","last_peered":"2024-09-13T15:15:30.463232+0000","last_clean":"2024-09-13T15:15:30.463232+0000","last_became_active":"2024-09-13T15:08:31.198423+0000","last_became_peered":"2024-09-13T15:08:31.198423+0000","last_unstale":"2024-09-13T15:15:30.463232+0000","last_undegraded":"2024-09-13T15:15:30.463232+0000","last_fullsized":"2024-09-13T15:15:30.463232+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:58.362919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"89'149","reported_seq":975,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463573+0000","last_change":"2024-09-13T15:13:03.079534+0000","last_active":"2024-09-13T15:15:30.463573+0000","last_peered":"2024-09-13T15:15:30.463573+0000","last_clean":"2024-09-13T15:15:30.463573+0000","last_became_active":"2024-09-13T15:12:38.941094+0000","last_became_peered":"2024-09-13T15:12:38.941094+0000","last_unstale":"2024-09-13T15:15:30.463573+0000","last_undegraded":"2024-09-13T15:15:30.463573+0000","last_fullsized":"2024-09-13T15:15:30.463573+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.079258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.079258+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:50.421978+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":806,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807873+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:15:30.807873+0000","last_peered":"2024-09-13T15:15:30.807873+0000","last_clean":"2024-09-13T15:15:30.807873+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:15:30.807873+0000","last_undegraded":"2024-09-13T15:15:30.807873+0000","last_fullsized":"2024-09-13T15:15:30.807873+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":961,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.250122+0000","last_change":"2024-09-13T15:15:10.377063+0000","last_active":"2024-09-13T15:15:27.250122+0000","last_peered":"2024-09-13T15:15:27.250122+0000","last_clean":"2024-09-13T15:15:27.250122+0000","last_became_active":"2024-09-13T15:15:10.376415+0000","last_became_peered":"2024-09-13T15:15:10.376415+0000","last_unstale":"2024-09-13T15:15:27.250122+0000","last_undegraded":"2024-09-13T15:15:27.250122+0000","last_fullsized":"2024-09-13T15:15:27.250122+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:41.800784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":552,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.507891+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:15:30.507891+0000","last_peered":"2024-09-13T15:15:30.507891+0000","last_clean":"2024-09-13T15:15:30.507891+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:15:30.507891+0000","last_undegraded":"2024-09-13T15:15:30.507891+0000","last_fullsized":"2024-09-13T15:15:30.507891+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"89'149","reported_seq":996,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834011+0000","last_change":"2024-09-13T15:14:37.438852+0000","last_active":"2024-09-13T15:15:29.834011+0000","last_peered":"2024-09-13T15:15:29.834011+0000","last_clean":"2024-09-13T15:15:29.834011+0000","last_became_active":"2024-09-13T15:14:37.438385+0000","last_became_peered":"2024-09-13T15:14:37.438385+0000","last_unstale":"2024-09-13T15:15:29.834011+0000","last_undegraded":"2024-09-13T15:15:29.834011+0000","last_fullsized":"2024-09-13T15:15:29.834011+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.094914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.094914+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:12.899244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":535,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111911+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:15:27.111911+0000","last_peered":"2024-09-13T15:15:27.111911+0000","last_clean":"2024-09-13T15:15:27.111911+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:15:27.111911+0000","last_undegraded":"2024-09-13T15:15:27.111911+0000","last_fullsized":"2024-09-13T15:15:27.111911+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":667,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111951+0000","last_change":"2024-09-13T15:08:31.193477+0000","last_active":"2024-09-13T15:15:27.111951+0000","last_peered":"2024-09-13T15:15:27.111951+0000","last_clean":"2024-09-13T15:15:27.111951+0000","last_became_active":"2024-09-13T15:08:31.181861+0000","last_became_peered":"2024-09-13T15:08:31.181861+0000","last_unstale":"2024-09-13T15:15:27.111951+0000","last_undegraded":"2024-09-13T15:15:27.111951+0000","last_fullsized":"2024-09-13T15:15:27.111951+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:02.424164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":33292,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"89'149","reported_seq":975,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.550089+0000","last_change":"2024-09-13T15:14:37.440742+0000","last_active":"2024-09-13T15:15:27.550089+0000","last_peered":"2024-09-13T15:15:27.550089+0000","last_clean":"2024-09-13T15:15:27.550089+0000","last_became_active":"2024-09-13T15:14:37.440304+0000","last_became_peered":"2024-09-13T15:14:37.440304+0000","last_unstale":"2024-09-13T15:15:27.550089+0000","last_undegraded":"2024-09-13T15:15:27.550089+0000","last_fullsized":"2024-09-13T15:15:27.550089+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:31.274137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:31.274137+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:46.299807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":944,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833167+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:15:29.833167+0000","last_peered":"2024-09-13T15:15:29.833167+0000","last_clean":"2024-09-13T15:15:29.833167+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:15:29.833167+0000","last_undegraded":"2024-09-13T15:15:29.833167+0000","last_fullsized":"2024-09-13T15:15:29.833167+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.808090+0000","last_change":"2024-09-13T15:08:31.204593+0000","last_active":"2024-09-13T15:15:30.808090+0000","last_peered":"2024-09-13T15:15:30.808090+0000","last_clean":"2024-09-13T15:15:30.808090+0000","last_became_active":"2024-09-13T15:08:31.198883+0000","last_became_peered":"2024-09-13T15:08:31.198883+0000","last_unstale":"2024-09-13T15:15:30.808090+0000","last_undegraded":"2024-09-13T15:15:30.808090+0000","last_fullsized":"2024-09-13T15:15:30.808090+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:08.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"89'149","reported_seq":978,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807090+0000","last_change":"2024-09-13T15:13:00.998330+0000","last_active":"2024-09-13T15:15:30.807090+0000","last_peered":"2024-09-13T15:15:30.807090+0000","last_clean":"2024-09-13T15:15:30.807090+0000","last_became_active":"2024-09-13T15:12:50.190273+0000","last_became_peered":"2024-09-13T15:12:50.190273+0000","last_unstale":"2024-09-13T15:15:30.807090+0000","last_undegraded":"2024-09-13T15:15:30.807090+0000","last_fullsized":"2024-09-13T15:15:30.807090+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.998266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.998266+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:00.945235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":424,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.550055+0000","last_change":"2024-09-13T15:08:31.188790+0000","last_active":"2024-09-13T15:15:27.550055+0000","last_peered":"2024-09-13T15:15:27.550055+0000","last_clean":"2024-09-13T15:15:27.550055+0000","last_became_active":"2024-09-13T15:08:31.188441+0000","last_became_peered":"2024-09-13T15:08:31.188441+0000","last_unstale":"2024-09-13T15:15:27.550055+0000","last_undegraded":"2024-09-13T15:15:27.550055+0000","last_fullsized":"2024-09-13T15:15:27.550055+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:52.934163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":932,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463327+0000","last_change":"2024-09-13T15:08:31.197112+0000","last_active":"2024-09-13T15:15:30.463327+0000","last_peered":"2024-09-13T15:15:30.463327+0000","last_clean":"2024-09-13T15:15:30.463327+0000","last_became_active":"2024-09-13T15:08:31.196687+0000","last_became_peered":"2024-09-13T15:08:31.196687+0000","last_unstale":"2024-09-13T15:15:30.463327+0000","last_undegraded":"2024-09-13T15:15:30.463327+0000","last_fullsized":"2024-09-13T15:15:30.463327+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:47.532912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"89'149","reported_seq":838,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879042+0000","last_change":"2024-09-13T15:14:42.899519+0000","last_active":"2024-09-13T15:15:27.879042+0000","last_peered":"2024-09-13T15:15:27.879042+0000","last_clean":"2024-09-13T15:15:27.879042+0000","last_became_active":"2024-09-13T15:14:37.443576+0000","last_became_peered":"2024-09-13T15:14:37.443576+0000","last_unstale":"2024-09-13T15:15:27.879042+0000","last_undegraded":"2024-09-13T15:15:27.879042+0000","last_fullsized":"2024-09-13T15:15:27.879042+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:42.899422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:42.899422+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:22.515720+0000","scrub_duration":0.0016158069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":897,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807841+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:15:30.807841+0000","last_peered":"2024-09-13T15:15:30.807841+0000","last_clean":"2024-09-13T15:15:30.807841+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:15:30.807841+0000","last_undegraded":"2024-09-13T15:15:30.807841+0000","last_fullsized":"2024-09-13T15:15:30.807841+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":962,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463277+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:15:30.463277+0000","last_peered":"2024-09-13T15:15:30.463277+0000","last_clean":"2024-09-13T15:15:30.463277+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:15:30.463277+0000","last_undegraded":"2024-09-13T15:15:30.463277+0000","last_fullsized":"2024-09-13T15:15:30.463277+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"89'149","reported_seq":838,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111546+0000","last_change":"2024-09-13T15:14:41.744947+0000","last_active":"2024-09-13T15:15:27.111546+0000","last_peered":"2024-09-13T15:15:27.111546+0000","last_clean":"2024-09-13T15:15:27.111546+0000","last_became_active":"2024-09-13T15:14:37.439746+0000","last_became_peered":"2024-09-13T15:14:37.439746+0000","last_unstale":"2024-09-13T15:15:27.111546+0000","last_undegraded":"2024-09-13T15:15:27.111546+0000","last_fullsized":"2024-09-13T15:15:27.111546+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:41.744839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:41.744839+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:41.276005+0000","scrub_duration":0.001559206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":251,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.250077+0000","last_change":"2024-09-13T15:15:11.072803+0000","last_active":"2024-09-13T15:15:27.250077+0000","last_peered":"2024-09-13T15:15:27.250077+0000","last_clean":"2024-09-13T15:15:27.250077+0000","last_became_active":"2024-09-13T15:15:11.072373+0000","last_became_peered":"2024-09-13T15:15:11.072373+0000","last_unstale":"2024-09-13T15:15:27.250077+0000","last_undegraded":"2024-09-13T15:15:27.250077+0000","last_fullsized":"2024-09-13T15:15:27.250077+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:33.742137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":926,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463989+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:15:30.463989+0000","last_peered":"2024-09-13T15:15:30.463989+0000","last_clean":"2024-09-13T15:15:30.463989+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:15:30.463989+0000","last_undegraded":"2024-09-13T15:15:30.463989+0000","last_fullsized":"2024-09-13T15:15:30.463989+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"89'149","reported_seq":908,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549650+0000","last_change":"2024-09-13T15:14:46.698051+0000","last_active":"2024-09-13T15:15:27.549650+0000","last_peered":"2024-09-13T15:15:27.549650+0000","last_clean":"2024-09-13T15:15:27.549650+0000","last_became_active":"2024-09-13T15:14:37.442325+0000","last_became_peered":"2024-09-13T15:14:37.442325+0000","last_unstale":"2024-09-13T15:15:27.549650+0000","last_undegraded":"2024-09-13T15:15:27.549650+0000","last_fullsized":"2024-09-13T15:15:27.549650+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:46.697941+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:14:46.697941+0000","last_clean_scrub_stamp":"2024-09-13T15:14:46.697941+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:38.304748+0000","scrub_duration":0.0015698439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":895,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807810+0000","last_change":"2024-09-13T15:08:31.195215+0000","last_active":"2024-09-13T15:15:30.807810+0000","last_peered":"2024-09-13T15:15:30.807810+0000","last_clean":"2024-09-13T15:15:30.807810+0000","last_became_active":"2024-09-13T15:08:31.194526+0000","last_became_peered":"2024-09-13T15:08:31.194526+0000","last_unstale":"2024-09-13T15:15:30.807810+0000","last_undegraded":"2024-09-13T15:15:30.807810+0000","last_fullsized":"2024-09-13T15:15:30.807810+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:56.358764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":898,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.832896+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:15:29.832896+0000","last_peered":"2024-09-13T15:15:29.832896+0000","last_clean":"2024-09-13T15:15:29.832896+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:15:29.832896+0000","last_undegraded":"2024-09-13T15:15:29.832896+0000","last_fullsized":"2024-09-13T15:15:29.832896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549856+0000","last_change":"2024-09-13T15:14:42.777420+0000","last_active":"2024-09-13T15:15:27.549856+0000","last_peered":"2024-09-13T15:15:27.549856+0000","last_clean":"2024-09-13T15:15:27.549856+0000","last_became_active":"2024-09-13T15:14:37.440925+0000","last_became_peered":"2024-09-13T15:14:37.440925+0000","last_unstale":"2024-09-13T15:15:27.549856+0000","last_undegraded":"2024-09-13T15:15:27.549856+0000","last_fullsized":"2024-09-13T15:15:27.549856+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:42.777305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:42.777305+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:47.412591+0000","scrub_duration":0.0021482089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":763,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463705+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:15:30.463705+0000","last_peered":"2024-09-13T15:15:30.463705+0000","last_clean":"2024-09-13T15:15:30.463705+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:15:30.463705+0000","last_undegraded":"2024-09-13T15:15:30.463705+0000","last_fullsized":"2024-09-13T15:15:30.463705+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":530,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508152+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:15:30.508152+0000","last_peered":"2024-09-13T15:15:30.508152+0000","last_clean":"2024-09-13T15:15:30.508152+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:15:30.508152+0000","last_undegraded":"2024-09-13T15:15:30.508152+0000","last_fullsized":"2024-09-13T15:15:30.508152+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"89'149","reported_seq":877,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463204+0000","last_change":"2024-09-13T15:14:38.292132+0000","last_active":"2024-09-13T15:15:30.463204+0000","last_peered":"2024-09-13T15:15:30.463204+0000","last_clean":"2024-09-13T15:15:30.463204+0000","last_became_active":"2024-09-13T15:14:37.437551+0000","last_became_peered":"2024-09-13T15:14:37.437551+0000","last_unstale":"2024-09-13T15:15:30.463204+0000","last_undegraded":"2024-09-13T15:15:30.463204+0000","last_fullsized":"2024-09-13T15:15:30.463204+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:38.292076+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:14:38.292076+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:27.703670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":514,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807302+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:15:30.807302+0000","last_peered":"2024-09-13T15:15:30.807302+0000","last_clean":"2024-09-13T15:15:30.807302+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:15:30.807302+0000","last_undegraded":"2024-09-13T15:15:30.807302+0000","last_fullsized":"2024-09-13T15:15:30.807302+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":882,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807378+0000","last_change":"2024-09-13T15:08:31.194912+0000","last_active":"2024-09-13T15:15:30.807378+0000","last_peered":"2024-09-13T15:15:30.807378+0000","last_clean":"2024-09-13T15:15:30.807378+0000","last_became_active":"2024-09-13T15:08:31.194553+0000","last_became_peered":"2024-09-13T15:08:31.194553+0000","last_unstale":"2024-09-13T15:15:30.807378+0000","last_undegraded":"2024-09-13T15:15:30.807378+0000","last_fullsized":"2024-09-13T15:15:30.807378+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:17.457690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"89'149","reported_seq":860,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463759+0000","last_change":"2024-09-13T15:14:39.264123+0000","last_active":"2024-09-13T15:15:30.463759+0000","last_peered":"2024-09-13T15:15:30.463759+0000","last_clean":"2024-09-13T15:15:30.463759+0000","last_became_active":"2024-09-13T15:14:37.441075+0000","last_became_peered":"2024-09-13T15:14:37.441075+0000","last_unstale":"2024-09-13T15:15:30.463759+0000","last_undegraded":"2024-09-13T15:15:30.463759+0000","last_fullsized":"2024-09-13T15:15:30.463759+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:39.264072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:39.264072+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:12.034626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":540,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508273+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:15:30.508273+0000","last_peered":"2024-09-13T15:15:30.508273+0000","last_clean":"2024-09-13T15:15:30.508273+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:15:30.508273+0000","last_undegraded":"2024-09-13T15:15:30.508273+0000","last_fullsized":"2024-09-13T15:15:30.508273+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":367,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549924+0000","last_change":"2024-09-13T15:08:46.073232+0000","last_active":"2024-09-13T15:15:27.549924+0000","last_peered":"2024-09-13T15:15:27.549924+0000","last_clean":"2024-09-13T15:15:27.549924+0000","last_became_active":"2024-09-13T15:08:46.072807+0000","last_became_peered":"2024-09-13T15:08:46.072807+0000","last_unstale":"2024-09-13T15:15:27.549924+0000","last_undegraded":"2024-09-13T15:15:27.549924+0000","last_fullsized":"2024-09-13T15:15:27.549924+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:11.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":528,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508031+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:15:30.508031+0000","last_peered":"2024-09-13T15:15:30.508031+0000","last_clean":"2024-09-13T15:15:30.508031+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:15:30.508031+0000","last_undegraded":"2024-09-13T15:15:30.508031+0000","last_fullsized":"2024-09-13T15:15:30.508031+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":395,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.550029+0000","last_change":"2024-09-13T15:08:46.079600+0000","last_active":"2024-09-13T15:15:27.550029+0000","last_peered":"2024-09-13T15:15:27.550029+0000","last_clean":"2024-09-13T15:15:27.550029+0000","last_became_active":"2024-09-13T15:08:46.075458+0000","last_became_peered":"2024-09-13T15:08:46.075458+0000","last_unstale":"2024-09-13T15:15:27.550029+0000","last_undegraded":"2024-09-13T15:15:27.550029+0000","last_fullsized":"2024-09-13T15:15:27.550029+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:41.707452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"89'149","reported_seq":900,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879002+0000","last_change":"2024-09-13T15:14:41.881812+0000","last_active":"2024-09-13T15:15:27.879002+0000","last_peered":"2024-09-13T15:15:27.879002+0000","last_clean":"2024-09-13T15:15:27.879002+0000","last_became_active":"2024-09-13T15:14:37.442716+0000","last_became_peered":"2024-09-13T15:14:37.442716+0000","last_unstale":"2024-09-13T15:15:27.879002+0000","last_undegraded":"2024-09-13T15:15:27.879002+0000","last_fullsized":"2024-09-13T15:15:27.879002+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:41.881687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:41.881687+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:12.841948+0000","scrub_duration":0.0015457030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":936,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463596+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:15:30.463596+0000","last_peered":"2024-09-13T15:15:30.463596+0000","last_clean":"2024-09-13T15:15:30.463596+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:15:30.463596+0000","last_undegraded":"2024-09-13T15:15:30.463596+0000","last_fullsized":"2024-09-13T15:15:30.463596+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":959,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463870+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:15:30.463870+0000","last_peered":"2024-09-13T15:15:30.463870+0000","last_clean":"2024-09-13T15:15:30.463870+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:15:30.463870+0000","last_undegraded":"2024-09-13T15:15:30.463870+0000","last_fullsized":"2024-09-13T15:15:30.463870+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"89'149","reported_seq":861,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833298+0000","last_change":"2024-09-13T15:14:37.462171+0000","last_active":"2024-09-13T15:15:29.833298+0000","last_peered":"2024-09-13T15:15:29.833298+0000","last_clean":"2024-09-13T15:15:29.833298+0000","last_became_active":"2024-09-13T15:14:37.441914+0000","last_became_peered":"2024-09-13T15:14:37.441914+0000","last_unstale":"2024-09-13T15:15:29.833298+0000","last_undegraded":"2024-09-13T15:15:29.833298+0000","last_fullsized":"2024-09-13T15:15:29.833298+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:37.462127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:37.462127+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:19.978212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":561,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111430+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:15:27.111430+0000","last_peered":"2024-09-13T15:15:27.111430+0000","last_clean":"2024-09-13T15:15:27.111430+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:15:27.111430+0000","last_undegraded":"2024-09-13T15:15:27.111430+0000","last_fullsized":"2024-09-13T15:15:27.111430+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":922,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807728+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:15:30.807728+0000","last_peered":"2024-09-13T15:15:30.807728+0000","last_clean":"2024-09-13T15:15:30.807728+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:15:30.807728+0000","last_undegraded":"2024-09-13T15:15:30.807728+0000","last_fullsized":"2024-09-13T15:15:30.807728+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":36578,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234587,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529596,"kb_used_data":7088,"kb_used_omap":1,"kb_used_meta":522494,"kb_avail":93215556,"statfs":{"total":95995035648,"available":95452729344,"internally_reserved":0,"allocated":7258112,"data_stored":4439817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":535034315},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496223,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":728172,"kb_used_data":7136,"kb_used_omap":1,"kb_used_meta":721022,"kb_avail":93016980,"statfs":{"total":95995035648,"available":95249387520,"internally_reserved":0,"allocated":7307264,"data_stored":4504409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":738326983},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757858,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876588,"kb_used_data":7072,"kb_used_omap":1,"kb_used_meta":869502,"kb_avail":92868564,"statfs":{"total":95995035648,"available":95097409536,"internally_reserved":0,"allocated":7241728,"data_stored":4435574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890370503},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019498,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850100,"kb_used_data":6824,"kb_used_omap":1,"kb_used_meta":843262,"kb_avail":92895052,"statfs":{"total":95995035648,"available":95124533248,"internally_reserved":0,"allocated":6987776,"data_stored":4177368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863500742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468382,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989668,"kb_used_data":11224,"kb_used_omap":4,"kb_used_meta":978427,"kb_avail":92755484,"statfs":{"total":95995035648,"available":94981615616,"internally_reserved":0,"allocated":11493376,"data_stored":8694574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":1001909412},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273282,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433712,"kb_used_data":11172,"kb_used_omap":12,"kb_used_meta":422515,"kb_avail":93311440,"statfs":{"total":95995035648,"available":95550914560,"internally_reserved":0,"allocated":11440128,"data_stored":8630130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":432656256},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795435,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895076,"kb_used_data":6808,"kb_used_omap":7,"kb_used_meta":888248,"kb_avail":92850076,"statfs":{"total":95995035648,"available":95078477824,"internally_reserved":0,"allocated":6971392,"data_stored":4172519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":909566578},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502216,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598120,"kb_used_data":11228,"kb_used_omap":10,"kb_used_meta":586869,"kb_avail":93147032,"statfs":{"total":95995035648,"available":95382560768,"internally_reserved":0,"allocated":11497472,"data_stored":8695088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":600954559},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:16:17.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:16:17.598 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/mon.a/config 2024-09-13T15:16:17.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:17 smithi154 ceph-mon[102314]: from='client.46724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:16:17.719 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:17 smithi154 ceph-mon[102314]: pgmap v1704: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:17.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:17 smithi154 ceph-mon[107038]: from='client.46724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:16:17.720 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:17 smithi154 ceph-mon[107038]: pgmap v1704: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:17 smithi172 ceph-mon[100696]: from='client.46724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:16:17.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:17 smithi172 ceph-mon[100696]: pgmap v1704: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:19.592 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T15:16:19.593 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:19 smithi154 ceph-mon[102314]: pgmap v1705: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:19.593 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:19 smithi154 ceph-mon[107038]: pgmap v1705: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:19.597 INFO:teuthology.orchestra.run.smithi154.stderr:dumped all 2024-09-13T15:16:19.741 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:19 smithi172 ceph-mon[100696]: pgmap v1705: 103 pgs: 103 active+clean; 4.6 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-09-13T15:16:20.437 INFO:teuthology.orchestra.run.smithi154.stdout:{"pg_ready":true,"pg_map":{"version":1705,"stamp":"2024-09-13T15:16:19.309449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790245,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":908,"num_write_kb":12625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":36635,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5652,"ondisk_log_size":5652,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5901032,"kb_used_data":68552,"kb_used_omap":40,"kb_used_meta":5832343,"kb_avail":744060184,"statfs":{"total":767960285184,"available":761917628416,"internally_reserved":0,"allocated":70197248,"data_stored":47749479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41868,"internal_metadata":5972319348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002987"},"pg_stats":[{"pgid":"1.11","version":"89'149","reported_seq":926,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.806726+0000","last_change":"2024-09-13T15:13:01.179220+0000","last_active":"2024-09-13T15:15:30.806726+0000","last_peered":"2024-09-13T15:15:30.806726+0000","last_clean":"2024-09-13T15:15:30.806726+0000","last_became_active":"2024-09-13T15:12:50.169695+0000","last_became_peered":"2024-09-13T15:12:50.169695+0000","last_unstale":"2024-09-13T15:15:30.806726+0000","last_undegraded":"2024-09-13T15:15:30.806726+0000","last_fullsized":"2024-09-13T15:15:30.806726+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:01.179154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:01.179154+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:08.084137+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":275,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807241+0000","last_change":"2024-09-13T15:08:31.190995+0000","last_active":"2024-09-13T15:15:30.807241+0000","last_peered":"2024-09-13T15:15:30.807241+0000","last_clean":"2024-09-13T15:15:30.807241+0000","last_became_active":"2024-09-13T15:08:31.190610+0000","last_became_peered":"2024-09-13T15:08:31.190610+0000","last_unstale":"2024-09-13T15:15:30.807241+0000","last_undegraded":"2024-09-13T15:15:30.807241+0000","last_fullsized":"2024-09-13T15:15:30.807241+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:29:42.124757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807818+0000","last_change":"2024-09-13T15:08:31.191585+0000","last_active":"2024-09-13T15:15:30.807818+0000","last_peered":"2024-09-13T15:15:30.807818+0000","last_clean":"2024-09-13T15:15:30.807818+0000","last_became_active":"2024-09-13T15:08:31.184700+0000","last_became_peered":"2024-09-13T15:08:31.184700+0000","last_unstale":"2024-09-13T15:15:30.807818+0000","last_undegraded":"2024-09-13T15:15:30.807818+0000","last_fullsized":"2024-09-13T15:15:30.807818+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:14.376834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"89'149","reported_seq":865,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549466+0000","last_change":"2024-09-13T15:13:09.203525+0000","last_active":"2024-09-13T15:15:27.549466+0000","last_peered":"2024-09-13T15:15:27.549466+0000","last_clean":"2024-09-13T15:15:27.549466+0000","last_became_active":"2024-09-13T15:12:50.159713+0000","last_became_peered":"2024-09-13T15:12:50.159713+0000","last_unstale":"2024-09-13T15:15:27.549466+0000","last_undegraded":"2024-09-13T15:15:27.549466+0000","last_fullsized":"2024-09-13T15:15:27.549466+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:09.203387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:09.203387+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:07.004095+0000","scrub_duration":0.002438127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":384,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549969+0000","last_change":"2024-09-13T15:08:46.080613+0000","last_active":"2024-09-13T15:15:27.549969+0000","last_peered":"2024-09-13T15:15:27.549969+0000","last_clean":"2024-09-13T15:15:27.549969+0000","last_became_active":"2024-09-13T15:08:46.080320+0000","last_became_peered":"2024-09-13T15:08:46.080320+0000","last_unstale":"2024-09-13T15:15:27.549969+0000","last_undegraded":"2024-09-13T15:15:27.549969+0000","last_fullsized":"2024-09-13T15:15:27.549969+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:53:09.205609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":889,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807654+0000","last_change":"2024-09-13T15:08:31.196449+0000","last_active":"2024-09-13T15:15:30.807654+0000","last_peered":"2024-09-13T15:15:30.807654+0000","last_clean":"2024-09-13T15:15:30.807654+0000","last_became_active":"2024-09-13T15:08:31.196143+0000","last_became_peered":"2024-09-13T15:08:31.196143+0000","last_unstale":"2024-09-13T15:15:30.807654+0000","last_undegraded":"2024-09-13T15:15:30.807654+0000","last_fullsized":"2024-09-13T15:15:30.807654+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:53.421723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"89'149","reported_seq":980,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833051+0000","last_change":"2024-09-13T15:13:03.099175+0000","last_active":"2024-09-13T15:15:29.833051+0000","last_peered":"2024-09-13T15:15:29.833051+0000","last_clean":"2024-09-13T15:15:29.833051+0000","last_became_active":"2024-09-13T15:12:50.196974+0000","last_became_peered":"2024-09-13T15:12:50.196974+0000","last_unstale":"2024-09-13T15:15:29.833051+0000","last_undegraded":"2024-09-13T15:15:29.833051+0000","last_fullsized":"2024-09-13T15:15:29.833051+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.098941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.098941+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:06.828156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":892,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807792+0000","last_change":"2024-09-13T15:08:31.195499+0000","last_active":"2024-09-13T15:15:30.807792+0000","last_peered":"2024-09-13T15:15:30.807792+0000","last_clean":"2024-09-13T15:15:30.807792+0000","last_became_active":"2024-09-13T15:08:31.194928+0000","last_became_peered":"2024-09-13T15:08:31.194928+0000","last_unstale":"2024-09-13T15:15:30.807792+0000","last_undegraded":"2024-09-13T15:15:30.807792+0000","last_fullsized":"2024-09-13T15:15:30.807792+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.112396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":100,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807614+0000","last_change":"2024-09-13T15:08:31.185658+0000","last_active":"2024-09-13T15:15:30.807614+0000","last_peered":"2024-09-13T15:15:30.807614+0000","last_clean":"2024-09-13T15:15:30.807614+0000","last_became_active":"2024-09-13T15:08:31.185202+0000","last_became_peered":"2024-09-13T15:08:31.185202+0000","last_unstale":"2024-09-13T15:15:30.807614+0000","last_undegraded":"2024-09-13T15:15:30.807614+0000","last_fullsized":"2024-09-13T15:15:30.807614+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:04.668371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"89'149","reported_seq":865,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549698+0000","last_change":"2024-09-13T15:13:11.154190+0000","last_active":"2024-09-13T15:15:27.549698+0000","last_peered":"2024-09-13T15:15:27.549698+0000","last_clean":"2024-09-13T15:15:27.549698+0000","last_became_active":"2024-09-13T15:12:50.159857+0000","last_became_peered":"2024-09-13T15:12:50.159857+0000","last_unstale":"2024-09-13T15:15:27.549698+0000","last_undegraded":"2024-09-13T15:15:27.549698+0000","last_fullsized":"2024-09-13T15:15:27.549698+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.153852+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.153852+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:54.235242+0000","scrub_duration":0.0043281400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":396,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463814+0000","last_change":"2024-09-13T15:08:31.199682+0000","last_active":"2024-09-13T15:15:30.463814+0000","last_peered":"2024-09-13T15:15:30.463814+0000","last_clean":"2024-09-13T15:15:30.463814+0000","last_became_active":"2024-09-13T15:08:31.199110+0000","last_became_peered":"2024-09-13T15:08:31.199110+0000","last_unstale":"2024-09-13T15:15:30.463814+0000","last_undegraded":"2024-09-13T15:15:30.463814+0000","last_fullsized":"2024-09-13T15:15:30.463814+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:30.096808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":162,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249434+0000","last_change":"2024-09-13T15:15:10.378633+0000","last_active":"2024-09-13T15:15:27.249434+0000","last_peered":"2024-09-13T15:15:27.249434+0000","last_clean":"2024-09-13T15:15:27.249434+0000","last_became_active":"2024-09-13T15:15:10.378285+0000","last_became_peered":"2024-09-13T15:15:10.378285+0000","last_unstale":"2024-09-13T15:15:27.249434+0000","last_undegraded":"2024-09-13T15:15:27.249434+0000","last_fullsized":"2024-09-13T15:15:27.249434+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:26.676056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"89'149","reported_seq":886,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879021+0000","last_change":"2024-09-13T15:14:30.804233+0000","last_active":"2024-09-13T15:15:27.879021+0000","last_peered":"2024-09-13T15:15:27.879021+0000","last_clean":"2024-09-13T15:15:27.879021+0000","last_became_active":"2024-09-13T15:14:28.397907+0000","last_became_peered":"2024-09-13T15:14:28.397907+0000","last_unstale":"2024-09-13T15:15:27.879021+0000","last_undegraded":"2024-09-13T15:15:27.879021+0000","last_fullsized":"2024-09-13T15:15:27.879021+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.804126+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:14:30.804126+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.804126+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:37.618332+0000","scrub_duration":0.0015459549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":104,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878505+0000","last_change":"2024-09-13T15:08:31.351504+0000","last_active":"2024-09-13T15:15:27.878505+0000","last_peered":"2024-09-13T15:15:27.878505+0000","last_clean":"2024-09-13T15:15:27.878505+0000","last_became_active":"2024-09-13T15:08:31.320950+0000","last_became_peered":"2024-09-13T15:08:31.320950+0000","last_unstale":"2024-09-13T15:15:27.878505+0000","last_undegraded":"2024-09-13T15:15:27.878505+0000","last_fullsized":"2024-09-13T15:15:27.878505+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:38.291931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":430,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549988+0000","last_change":"2024-09-13T15:08:46.078296+0000","last_active":"2024-09-13T15:15:27.549988+0000","last_peered":"2024-09-13T15:15:27.549988+0000","last_clean":"2024-09-13T15:15:27.549988+0000","last_became_active":"2024-09-13T15:08:46.078037+0000","last_became_peered":"2024-09-13T15:08:46.078037+0000","last_unstale":"2024-09-13T15:15:27.549988+0000","last_undegraded":"2024-09-13T15:15:27.549988+0000","last_fullsized":"2024-09-13T15:15:27.549988+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:49.959361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"89'149","reported_seq":1002,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463916+0000","last_change":"2024-09-13T15:14:30.238997+0000","last_active":"2024-09-13T15:15:30.463916+0000","last_peered":"2024-09-13T15:15:30.463916+0000","last_clean":"2024-09-13T15:15:30.463916+0000","last_became_active":"2024-09-13T15:14:28.395022+0000","last_became_peered":"2024-09-13T15:14:28.395022+0000","last_unstale":"2024-09-13T15:15:30.463916+0000","last_undegraded":"2024-09-13T15:15:30.463916+0000","last_fullsized":"2024-09-13T15:15:30.463916+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.238938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.238938+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:37.186365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":104,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878573+0000","last_change":"2024-09-13T15:08:31.201939+0000","last_active":"2024-09-13T15:15:27.878573+0000","last_peered":"2024-09-13T15:15:27.878573+0000","last_clean":"2024-09-13T15:15:27.878573+0000","last_became_active":"2024-09-13T15:08:31.191089+0000","last_became_peered":"2024-09-13T15:08:31.191089+0000","last_unstale":"2024-09-13T15:15:27.878573+0000","last_undegraded":"2024-09-13T15:15:27.878573+0000","last_fullsized":"2024-09-13T15:15:27.878573+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:06.249945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":594,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.507959+0000","last_change":"2024-09-13T15:01:17.847839+0000","last_active":"2024-09-13T15:15:30.507959+0000","last_peered":"2024-09-13T15:15:30.507959+0000","last_clean":"2024-09-13T15:15:30.507959+0000","last_became_active":"2024-09-13T15:01:17.847578+0000","last_became_peered":"2024-09-13T15:01:17.847578+0000","last_unstale":"2024-09-13T15:15:30.507959+0000","last_undegraded":"2024-09-13T15:15:30.507959+0000","last_fullsized":"2024-09-13T15:15:30.507959+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:28.826247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"89'149","reported_seq":780,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.250055+0000","last_change":"2024-09-13T15:15:14.337399+0000","last_active":"2024-09-13T15:15:27.250055+0000","last_peered":"2024-09-13T15:15:27.250055+0000","last_clean":"2024-09-13T15:15:27.250055+0000","last_became_active":"2024-09-13T15:15:10.378910+0000","last_became_peered":"2024-09-13T15:15:10.378910+0000","last_unstale":"2024-09-13T15:15:27.250055+0000","last_undegraded":"2024-09-13T15:15:27.250055+0000","last_fullsized":"2024-09-13T15:15:27.250055+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:15:14.337271+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:15:14.337271+0000","last_clean_scrub_stamp":"2024-09-13T15:15:14.337271+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:45.029466+0000","scrub_duration":0.001704267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":104,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879247+0000","last_change":"2024-09-13T15:08:31.199736+0000","last_active":"2024-09-13T15:15:27.879247+0000","last_peered":"2024-09-13T15:15:27.879247+0000","last_clean":"2024-09-13T15:15:27.879247+0000","last_became_active":"2024-09-13T15:08:31.186018+0000","last_became_peered":"2024-09-13T15:08:31.186018+0000","last_unstale":"2024-09-13T15:15:27.879247+0000","last_undegraded":"2024-09-13T15:15:27.879247+0000","last_fullsized":"2024-09-13T15:15:27.879247+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:26.855187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":460,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249631+0000","last_change":"2024-09-13T15:15:11.072923+0000","last_active":"2024-09-13T15:15:27.249631+0000","last_peered":"2024-09-13T15:15:27.249631+0000","last_clean":"2024-09-13T15:15:27.249631+0000","last_became_active":"2024-09-13T15:15:11.072381+0000","last_became_peered":"2024-09-13T15:15:11.072381+0000","last_unstale":"2024-09-13T15:15:27.249631+0000","last_undegraded":"2024-09-13T15:15:27.249631+0000","last_fullsized":"2024-09-13T15:15:27.249631+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:43.491830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"89'149","reported_seq":955,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.112022+0000","last_change":"2024-09-13T15:14:30.760086+0000","last_active":"2024-09-13T15:15:27.112022+0000","last_peered":"2024-09-13T15:15:27.112022+0000","last_clean":"2024-09-13T15:15:27.112022+0000","last_became_active":"2024-09-13T15:14:28.397451+0000","last_became_peered":"2024-09-13T15:14:28.397451+0000","last_unstale":"2024-09-13T15:15:27.112022+0000","last_undegraded":"2024-09-13T15:15:27.112022+0000","last_fullsized":"2024-09-13T15:15:27.112022+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.759981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.759981+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:42.322864+0000","scrub_duration":0.00187911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":401,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833994+0000","last_change":"2024-09-13T15:08:46.073352+0000","last_active":"2024-09-13T15:15:29.833994+0000","last_peered":"2024-09-13T15:15:29.833994+0000","last_clean":"2024-09-13T15:15:29.833994+0000","last_became_active":"2024-09-13T15:08:46.073048+0000","last_became_peered":"2024-09-13T15:08:46.073048+0000","last_unstale":"2024-09-13T15:15:29.833994+0000","last_undegraded":"2024-09-13T15:15:29.833994+0000","last_fullsized":"2024-09-13T15:15:29.833994+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:57.277685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":801,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807205+0000","last_change":"2024-09-13T14:59:42.581880+0000","last_active":"2024-09-13T15:15:30.807205+0000","last_peered":"2024-09-13T15:15:30.807205+0000","last_clean":"2024-09-13T15:15:30.807205+0000","last_became_active":"2024-09-13T14:59:42.581566+0000","last_became_peered":"2024-09-13T14:59:42.581566+0000","last_unstale":"2024-09-13T15:15:30.807205+0000","last_undegraded":"2024-09-13T15:15:30.807205+0000","last_fullsized":"2024-09-13T15:15:30.807205+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:16.816334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878886+0000","last_change":"2024-09-13T15:14:29.775519+0000","last_active":"2024-09-13T15:15:27.878886+0000","last_peered":"2024-09-13T15:15:27.878886+0000","last_clean":"2024-09-13T15:15:27.878886+0000","last_became_active":"2024-09-13T15:14:28.396999+0000","last_became_peered":"2024-09-13T15:14:28.396999+0000","last_unstale":"2024-09-13T15:15:27.878886+0000","last_undegraded":"2024-09-13T15:15:27.878886+0000","last_fullsized":"2024-09-13T15:15:27.878886+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:29.775419+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:14:29.775419+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:01.928165+0000","scrub_duration":0.0020314999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":930,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463536+0000","last_change":"2024-09-13T14:59:42.578134+0000","last_active":"2024-09-13T15:15:30.463536+0000","last_peered":"2024-09-13T15:15:30.463536+0000","last_clean":"2024-09-13T15:15:30.463536+0000","last_became_active":"2024-09-13T14:59:42.577790+0000","last_became_peered":"2024-09-13T14:59:42.577790+0000","last_unstale":"2024-09-13T15:15:30.463536+0000","last_undegraded":"2024-09-13T15:15:30.463536+0000","last_fullsized":"2024-09-13T15:15:30.463536+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:05.432075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":536,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508140+0000","last_change":"2024-09-13T15:04:05.015474+0000","last_active":"2024-09-13T15:15:30.508140+0000","last_peered":"2024-09-13T15:15:30.508140+0000","last_clean":"2024-09-13T15:15:30.508140+0000","last_became_active":"2024-09-13T15:04:05.014940+0000","last_became_peered":"2024-09-13T15:04:05.014940+0000","last_unstale":"2024-09-13T15:15:30.508140+0000","last_undegraded":"2024-09-13T15:15:30.508140+0000","last_fullsized":"2024-09-13T15:15:30.508140+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:44:51.511492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"89'149","reported_seq":905,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549722+0000","last_change":"2024-09-13T15:14:34.801562+0000","last_active":"2024-09-13T15:15:27.549722+0000","last_peered":"2024-09-13T15:15:27.549722+0000","last_clean":"2024-09-13T15:15:27.549722+0000","last_became_active":"2024-09-13T15:14:28.398879+0000","last_became_peered":"2024-09-13T15:14:28.398879+0000","last_unstale":"2024-09-13T15:15:27.549722+0000","last_undegraded":"2024-09-13T15:15:27.549722+0000","last_fullsized":"2024-09-13T15:15:27.549722+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:34.801454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:34.801454+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:43.731723+0000","scrub_duration":0.0017318050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":171,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249809+0000","last_change":"2024-09-13T15:15:11.072536+0000","last_active":"2024-09-13T15:15:27.249809+0000","last_peered":"2024-09-13T15:15:27.249809+0000","last_clean":"2024-09-13T15:15:27.249809+0000","last_became_active":"2024-09-13T15:15:11.072203+0000","last_became_peered":"2024-09-13T15:15:11.072203+0000","last_unstale":"2024-09-13T15:15:27.249809+0000","last_undegraded":"2024-09-13T15:15:27.249809+0000","last_fullsized":"2024-09-13T15:15:27.249809+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:42.437845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":530,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508524+0000","last_change":"2024-09-13T15:00:14.951998+0000","last_active":"2024-09-13T15:15:30.508524+0000","last_peered":"2024-09-13T15:15:30.508524+0000","last_clean":"2024-09-13T15:15:30.508524+0000","last_became_active":"2024-09-13T15:00:14.951780+0000","last_became_peered":"2024-09-13T15:00:14.951780+0000","last_unstale":"2024-09-13T15:15:30.508524+0000","last_undegraded":"2024-09-13T15:15:30.508524+0000","last_fullsized":"2024-09-13T15:15:30.508524+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:58.197064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"89'149","reported_seq":994,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833197+0000","last_change":"2024-09-13T15:14:29.402435+0000","last_active":"2024-09-13T15:15:29.833197+0000","last_peered":"2024-09-13T15:15:29.833197+0000","last_clean":"2024-09-13T15:15:29.833197+0000","last_became_active":"2024-09-13T15:14:28.396072+0000","last_became_peered":"2024-09-13T15:14:28.396072+0000","last_unstale":"2024-09-13T15:15:29.833197+0000","last_undegraded":"2024-09-13T15:15:29.833197+0000","last_fullsized":"2024-09-13T15:15:29.833197+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:29.402385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:29.402385+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:45.134708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":555,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508004+0000","last_change":"2024-09-13T15:08:31.184973+0000","last_active":"2024-09-13T15:15:30.508004+0000","last_peered":"2024-09-13T15:15:30.508004+0000","last_clean":"2024-09-13T15:15:30.508004+0000","last_became_active":"2024-09-13T15:08:31.184600+0000","last_became_peered":"2024-09-13T15:08:31.184600+0000","last_unstale":"2024-09-13T15:15:30.508004+0000","last_undegraded":"2024-09-13T15:15:30.508004+0000","last_fullsized":"2024-09-13T15:15:30.508004+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:09.452895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":563,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508480+0000","last_change":"2024-09-13T15:04:05.008259+0000","last_active":"2024-09-13T15:15:30.508480+0000","last_peered":"2024-09-13T15:15:30.508480+0000","last_clean":"2024-09-13T15:15:30.508480+0000","last_became_active":"2024-09-13T15:04:05.007869+0000","last_became_peered":"2024-09-13T15:04:05.007869+0000","last_unstale":"2024-09-13T15:15:30.508480+0000","last_undegraded":"2024-09-13T15:15:30.508480+0000","last_fullsized":"2024-09-13T15:15:30.508480+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:34:50.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"89'149","reported_seq":905,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549597+0000","last_change":"2024-09-13T15:14:35.764139+0000","last_active":"2024-09-13T15:15:27.549597+0000","last_peered":"2024-09-13T15:15:27.549597+0000","last_clean":"2024-09-13T15:15:27.549597+0000","last_became_active":"2024-09-13T15:14:28.401152+0000","last_became_peered":"2024-09-13T15:14:28.401152+0000","last_unstale":"2024-09-13T15:15:27.549597+0000","last_undegraded":"2024-09-13T15:15:27.549597+0000","last_fullsized":"2024-09-13T15:15:27.549597+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:35.764037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:35.764037+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:59.340623+0000","scrub_duration":0.001781129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":396,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463177+0000","last_change":"2024-09-13T15:08:31.198602+0000","last_active":"2024-09-13T15:15:30.463177+0000","last_peered":"2024-09-13T15:15:30.463177+0000","last_clean":"2024-09-13T15:15:30.463177+0000","last_became_active":"2024-09-13T15:08:31.198289+0000","last_became_peered":"2024-09-13T15:08:31.198289+0000","last_unstale":"2024-09-13T15:15:30.463177+0000","last_undegraded":"2024-09-13T15:15:30.463177+0000","last_fullsized":"2024-09-13T15:15:30.463177+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:36.803833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":675,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.808008+0000","last_change":"2024-09-13T15:00:14.950922+0000","last_active":"2024-09-13T15:15:30.808008+0000","last_peered":"2024-09-13T15:15:30.808008+0000","last_clean":"2024-09-13T15:15:30.808008+0000","last_became_active":"2024-09-13T15:00:14.950586+0000","last_became_peered":"2024-09-13T15:00:14.950586+0000","last_unstale":"2024-09-13T15:15:30.808008+0000","last_undegraded":"2024-09-13T15:15:30.808008+0000","last_fullsized":"2024-09-13T15:15:30.808008+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:21.851455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"89'149","reported_seq":910,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878161+0000","last_change":"2024-09-13T15:13:11.597792+0000","last_active":"2024-09-13T15:15:27.878161+0000","last_peered":"2024-09-13T15:15:27.878161+0000","last_clean":"2024-09-13T15:15:27.878161+0000","last_became_active":"2024-09-13T15:12:38.938735+0000","last_became_peered":"2024-09-13T15:12:38.938735+0000","last_unstale":"2024-09-13T15:15:27.878161+0000","last_undegraded":"2024-09-13T15:15:27.878161+0000","last_fullsized":"2024-09-13T15:15:27.878161+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:11.597623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:11.597623+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:14.909270+0000","scrub_duration":0.012675130999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"85'157","reported_seq":1197,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834065+0000","last_change":"2024-09-13T15:08:57.728783+0000","last_active":"2024-09-13T15:15:29.834065+0000","last_peered":"2024-09-13T15:15:29.834065+0000","last_clean":"2024-09-13T15:15:29.834065+0000","last_became_active":"2024-09-13T15:08:47.077688+0000","last_became_peered":"2024-09-13T15:08:47.077688+0000","last_unstale":"2024-09-13T15:15:29.834065+0000","last_undegraded":"2024-09-13T15:15:29.834065+0000","last_fullsized":"2024-09-13T15:15:29.834065+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:28:52.710051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048547199999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":809,"num_write":135,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":892,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463291+0000","last_change":"2024-09-13T15:04:05.017053+0000","last_active":"2024-09-13T15:15:30.463291+0000","last_peered":"2024-09-13T15:15:30.463291+0000","last_clean":"2024-09-13T15:15:30.463291+0000","last_became_active":"2024-09-13T15:04:05.016765+0000","last_became_peered":"2024-09-13T15:04:05.016765+0000","last_unstale":"2024-09-13T15:15:30.463291+0000","last_undegraded":"2024-09-13T15:15:30.463291+0000","last_fullsized":"2024-09-13T15:15:30.463291+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:48.252457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":299,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249868+0000","last_change":"2024-09-13T15:15:10.379913+0000","last_active":"2024-09-13T15:15:27.249868+0000","last_peered":"2024-09-13T15:15:27.249868+0000","last_clean":"2024-09-13T15:15:27.249868+0000","last_became_active":"2024-09-13T15:15:10.379437+0000","last_became_peered":"2024-09-13T15:15:10.379437+0000","last_unstale":"2024-09-13T15:15:27.249868+0000","last_undegraded":"2024-09-13T15:15:27.249868+0000","last_fullsized":"2024-09-13T15:15:27.249868+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:52.979549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"89'149","reported_seq":977,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833144+0000","last_change":"2024-09-13T15:13:04.107856+0000","last_active":"2024-09-13T15:15:29.833144+0000","last_peered":"2024-09-13T15:15:29.833144+0000","last_clean":"2024-09-13T15:15:29.833144+0000","last_became_active":"2024-09-13T15:12:38.929136+0000","last_became_peered":"2024-09-13T15:12:38.929136+0000","last_unstale":"2024-09-13T15:15:29.833144+0000","last_undegraded":"2024-09-13T15:15:29.833144+0000","last_fullsized":"2024-09-13T15:15:29.833144+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:04.107315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:04.107315+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:00.198499+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"85'65","reported_seq":118,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878394+0000","last_change":"2024-09-13T15:08:31.218115+0000","last_active":"2024-09-13T15:15:27.878394+0000","last_peered":"2024-09-13T15:15:27.878394+0000","last_clean":"2024-09-13T15:15:27.878394+0000","last_became_active":"2024-09-13T15:08:31.196118+0000","last_became_peered":"2024-09-13T15:08:31.196118+0000","last_unstale":"2024-09-13T15:15:27.878394+0000","last_undegraded":"2024-09-13T15:15:27.878394+0000","last_fullsized":"2024-09-13T15:15:27.878394+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:41:08.721976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032771299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":273,"num_read_kb":660,"num_write":62,"num_write_kb":260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":875,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.806930+0000","last_change":"2024-09-13T15:01:17.856370+0000","last_active":"2024-09-13T15:15:30.806930+0000","last_peered":"2024-09-13T15:15:30.806930+0000","last_clean":"2024-09-13T15:15:30.806930+0000","last_became_active":"2024-09-13T15:01:17.856075+0000","last_became_peered":"2024-09-13T15:01:17.856075+0000","last_unstale":"2024-09-13T15:15:30.806930+0000","last_undegraded":"2024-09-13T15:15:30.806930+0000","last_fullsized":"2024-09-13T15:15:30.806930+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:09.497080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":870,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807710+0000","last_change":"2024-09-13T15:00:14.950070+0000","last_active":"2024-09-13T15:15:30.807710+0000","last_peered":"2024-09-13T15:15:30.807710+0000","last_clean":"2024-09-13T15:15:30.807710+0000","last_became_active":"2024-09-13T15:00:14.949642+0000","last_became_peered":"2024-09-13T15:00:14.949642+0000","last_unstale":"2024-09-13T15:15:30.807710+0000","last_undegraded":"2024-09-13T15:15:30.807710+0000","last_fullsized":"2024-09-13T15:15:30.807710+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:59.900792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878262+0000","last_change":"2024-09-13T15:13:05.496011+0000","last_active":"2024-09-13T15:15:27.878262+0000","last_peered":"2024-09-13T15:15:27.878262+0000","last_clean":"2024-09-13T15:15:27.878262+0000","last_became_active":"2024-09-13T15:12:38.946687+0000","last_became_peered":"2024-09-13T15:12:38.946687+0000","last_unstale":"2024-09-13T15:15:27.878262+0000","last_undegraded":"2024-09-13T15:15:27.878262+0000","last_fullsized":"2024-09-13T15:15:27.878262+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:05.495826+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:25.509161+0000","last_clean_scrub_stamp":"2024-09-13T15:13:05.495826+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:01.133250+0000","scrub_duration":0.011811428000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"85'130","reported_seq":1312,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807351+0000","last_change":"2024-09-13T15:00:14.949209+0000","last_active":"2024-09-13T15:15:30.807351+0000","last_peered":"2024-09-13T15:15:30.807351+0000","last_clean":"2024-09-13T15:15:30.807351+0000","last_became_active":"2024-09-13T15:00:14.948633+0000","last_became_peered":"2024-09-13T15:00:14.948633+0000","last_unstale":"2024-09-13T15:15:30.807351+0000","last_undegraded":"2024-09-13T15:15:30.807351+0000","last_fullsized":"2024-09-13T15:15:30.807351+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:49.691160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011252230000000001,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":950,"num_read_kb":3108,"num_write":119,"num_write_kb":6368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":941,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833030+0000","last_change":"2024-09-13T15:04:05.011355+0000","last_active":"2024-09-13T15:15:29.833030+0000","last_peered":"2024-09-13T15:15:29.833030+0000","last_clean":"2024-09-13T15:15:29.833030+0000","last_became_active":"2024-09-13T15:04:05.011041+0000","last_became_peered":"2024-09-13T15:04:05.011041+0000","last_unstale":"2024-09-13T15:15:29.833030+0000","last_undegraded":"2024-09-13T15:15:29.833030+0000","last_fullsized":"2024-09-13T15:15:29.833030+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:59.674038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":679,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834031+0000","last_change":"2024-09-13T14:59:42.568330+0000","last_active":"2024-09-13T15:15:29.834031+0000","last_peered":"2024-09-13T15:15:29.834031+0000","last_clean":"2024-09-13T15:15:29.834031+0000","last_became_active":"2024-09-13T14:59:42.568039+0000","last_became_peered":"2024-09-13T14:59:42.568039+0000","last_unstale":"2024-09-13T15:15:29.834031+0000","last_undegraded":"2024-09-13T15:15:29.834031+0000","last_fullsized":"2024-09-13T15:15:29.834031+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:53.734496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"89'149","reported_seq":905,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549438+0000","last_change":"2024-09-13T15:13:03.104893+0000","last_active":"2024-09-13T15:15:27.549438+0000","last_peered":"2024-09-13T15:15:27.549438+0000","last_clean":"2024-09-13T15:15:27.549438+0000","last_became_active":"2024-09-13T15:12:38.933486+0000","last_became_peered":"2024-09-13T15:12:38.933486+0000","last_unstale":"2024-09-13T15:15:27.549438+0000","last_undegraded":"2024-09-13T15:15:27.549438+0000","last_fullsized":"2024-09-13T15:15:27.549438+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":4,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.104648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.104648+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:22:41.623486+0000","scrub_duration":0.01681821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'90","reported_seq":451,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.131072+0000","last_change":"2024-09-13T15:08:46.074249+0000","last_active":"2024-09-13T15:15:29.131072+0000","last_peered":"2024-09-13T15:15:29.131072+0000","last_clean":"2024-09-13T15:15:29.131072+0000","last_became_active":"2024-09-13T15:08:46.073842+0000","last_became_peered":"2024-09-13T15:08:46.073842+0000","last_unstale":"2024-09-13T15:15:29.131072+0000","last_undegraded":"2024-09-13T15:15:29.131072+0000","last_fullsized":"2024-09-13T15:15:29.131072+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:44.854067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051509699999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":929,"num_read_kb":2882,"num_write":67,"num_write_kb":2070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":556,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508483+0000","last_change":"2024-09-13T15:04:05.013455+0000","last_active":"2024-09-13T15:15:30.508483+0000","last_peered":"2024-09-13T15:15:30.508483+0000","last_clean":"2024-09-13T15:15:30.508483+0000","last_became_active":"2024-09-13T15:04:05.012957+0000","last_became_peered":"2024-09-13T15:04:05.012957+0000","last_unstale":"2024-09-13T15:15:30.508483+0000","last_undegraded":"2024-09-13T15:15:30.508483+0000","last_fullsized":"2024-09-13T15:15:30.508483+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:05.072958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":368,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549773+0000","last_change":"2024-09-13T15:08:46.075360+0000","last_active":"2024-09-13T15:15:27.549773+0000","last_peered":"2024-09-13T15:15:27.549773+0000","last_clean":"2024-09-13T15:15:27.549773+0000","last_became_active":"2024-09-13T15:08:46.075046+0000","last_became_peered":"2024-09-13T15:08:46.075046+0000","last_unstale":"2024-09-13T15:15:27.549773+0000","last_undegraded":"2024-09-13T15:15:27.549773+0000","last_fullsized":"2024-09-13T15:15:27.549773+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.044150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"89'149","reported_seq":780,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249993+0000","last_change":"2024-09-13T15:15:23.391515+0000","last_active":"2024-09-13T15:15:27.249993+0000","last_peered":"2024-09-13T15:15:27.249993+0000","last_clean":"2024-09-13T15:15:27.249993+0000","last_became_active":"2024-09-13T15:15:10.377371+0000","last_became_peered":"2024-09-13T15:15:10.377371+0000","last_unstale":"2024-09-13T15:15:27.249993+0000","last_undegraded":"2024-09-13T15:15:27.249993+0000","last_fullsized":"2024-09-13T15:15:27.249993+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:15:23.391432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:15:23.391432+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:02.631052+0000","scrub_duration":0.0015151260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":401,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833768+0000","last_change":"2024-09-13T15:08:46.079918+0000","last_active":"2024-09-13T15:15:29.833768+0000","last_peered":"2024-09-13T15:15:29.833768+0000","last_clean":"2024-09-13T15:15:29.833768+0000","last_became_active":"2024-09-13T15:08:46.075051+0000","last_became_peered":"2024-09-13T15:08:46.075051+0000","last_unstale":"2024-09-13T15:15:29.833768+0000","last_undegraded":"2024-09-13T15:15:29.833768+0000","last_fullsized":"2024-09-13T15:15:29.833768+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:54.485416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"85'148","reported_seq":1593,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833657+0000","last_change":"2024-09-13T15:08:59.341688+0000","last_active":"2024-09-13T15:15:29.833657+0000","last_peered":"2024-09-13T15:15:29.833657+0000","last_clean":"2024-09-13T15:15:29.833657+0000","last_became_active":"2024-09-13T15:08:59.340925+0000","last_became_peered":"2024-09-13T15:08:59.340925+0000","last_unstale":"2024-09-13T15:15:29.833657+0000","last_undegraded":"2024-09-13T15:15:29.833657+0000","last_fullsized":"2024-09-13T15:15:29.833657+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:22.966119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050660899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":757,"num_read_kb":918,"num_write":138,"num_write_kb":203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":520,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807507+0000","last_change":"2024-09-13T14:59:42.567653+0000","last_active":"2024-09-13T15:15:30.807507+0000","last_peered":"2024-09-13T15:15:30.807507+0000","last_clean":"2024-09-13T15:15:30.807507+0000","last_became_active":"2024-09-13T14:59:42.559939+0000","last_became_peered":"2024-09-13T14:59:42.559939+0000","last_unstale":"2024-09-13T15:15:30.807507+0000","last_undegraded":"2024-09-13T15:15:30.807507+0000","last_fullsized":"2024-09-13T15:15:30.807507+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:53:47.825895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"89'149","reported_seq":780,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.249887+0000","last_change":"2024-09-13T15:15:12.340589+0000","last_active":"2024-09-13T15:15:27.249887+0000","last_peered":"2024-09-13T15:15:27.249887+0000","last_clean":"2024-09-13T15:15:27.249887+0000","last_became_active":"2024-09-13T15:15:10.373811+0000","last_became_peered":"2024-09-13T15:15:10.373811+0000","last_unstale":"2024-09-13T15:15:27.249887+0000","last_undegraded":"2024-09-13T15:15:27.249887+0000","last_fullsized":"2024-09-13T15:15:27.249887+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:15:12.340451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:15:12.340451+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:40.025924+0000","scrub_duration":0.001977941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":933,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833581+0000","last_change":"2024-09-13T15:00:14.948361+0000","last_active":"2024-09-13T15:15:29.833581+0000","last_peered":"2024-09-13T15:15:29.833581+0000","last_clean":"2024-09-13T15:15:29.833581+0000","last_became_active":"2024-09-13T15:00:14.948043+0000","last_became_peered":"2024-09-13T15:00:14.948043+0000","last_unstale":"2024-09-13T15:15:29.833581+0000","last_undegraded":"2024-09-13T15:15:29.833581+0000","last_fullsized":"2024-09-13T15:15:29.833581+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:08:45.091703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'90","reported_seq":118,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.878555+0000","last_change":"2024-09-13T15:08:31.209459+0000","last_active":"2024-09-13T15:15:27.878555+0000","last_peered":"2024-09-13T15:15:27.878555+0000","last_clean":"2024-09-13T15:15:27.878555+0000","last_became_active":"2024-09-13T15:08:31.188811+0000","last_became_peered":"2024-09-13T15:08:31.188811+0000","last_unstale":"2024-09-13T15:15:27.878555+0000","last_undegraded":"2024-09-13T15:15:27.878555+0000","last_fullsized":"2024-09-13T15:15:27.878555+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.667223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021454800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":652,"num_write":82,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":879,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807297+0000","last_change":"2024-09-13T15:08:31.197894+0000","last_active":"2024-09-13T15:15:30.807297+0000","last_peered":"2024-09-13T15:15:30.807297+0000","last_clean":"2024-09-13T15:15:30.807297+0000","last_became_active":"2024-09-13T15:08:31.197516+0000","last_became_peered":"2024-09-13T15:08:31.197516+0000","last_unstale":"2024-09-13T15:15:30.807297+0000","last_undegraded":"2024-09-13T15:15:30.807297+0000","last_fullsized":"2024-09-13T15:15:30.807297+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:58:42.650595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.112162+0000","last_change":"2024-09-13T15:14:29.757740+0000","last_active":"2024-09-13T15:15:27.112162+0000","last_peered":"2024-09-13T15:15:27.112162+0000","last_clean":"2024-09-13T15:15:27.112162+0000","last_became_active":"2024-09-13T15:14:28.395093+0000","last_became_peered":"2024-09-13T15:14:28.395093+0000","last_unstale":"2024-09-13T15:15:27.112162+0000","last_undegraded":"2024-09-13T15:15:27.112162+0000","last_fullsized":"2024-09-13T15:15:27.112162+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:29.757627+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:14:29.757627+0000","last_clean_scrub_stamp":"2024-09-13T15:14:29.757627+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:02.686292+0000","scrub_duration":0.001677331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":985,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463908+0000","last_change":"2024-09-13T15:08:31.202795+0000","last_active":"2024-09-13T15:15:30.463908+0000","last_peered":"2024-09-13T15:15:30.463908+0000","last_clean":"2024-09-13T15:15:30.463908+0000","last_became_active":"2024-09-13T15:08:31.199963+0000","last_became_peered":"2024-09-13T15:08:31.199963+0000","last_unstale":"2024-09-13T15:15:30.463908+0000","last_undegraded":"2024-09-13T15:15:30.463908+0000","last_fullsized":"2024-09-13T15:15:30.463908+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:41:08.346717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'116","reported_seq":1419,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834073+0000","last_change":"2024-09-13T15:04:08.053571+0000","last_active":"2024-09-13T15:15:29.834073+0000","last_peered":"2024-09-13T15:15:29.834073+0000","last_clean":"2024-09-13T15:15:29.834073+0000","last_became_active":"2024-09-13T15:04:08.053063+0000","last_became_peered":"2024-09-13T15:04:08.053063+0000","last_unstale":"2024-09-13T15:15:29.834073+0000","last_undegraded":"2024-09-13T15:15:29.834073+0000","last_fullsized":"2024-09-13T15:15:29.834073+0000","mapping_epoch":361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:37.348046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00020071299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":620,"num_read_kb":502,"num_write":79,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":690,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463232+0000","last_change":"2024-09-13T15:08:31.198729+0000","last_active":"2024-09-13T15:15:30.463232+0000","last_peered":"2024-09-13T15:15:30.463232+0000","last_clean":"2024-09-13T15:15:30.463232+0000","last_became_active":"2024-09-13T15:08:31.198423+0000","last_became_peered":"2024-09-13T15:08:31.198423+0000","last_unstale":"2024-09-13T15:15:30.463232+0000","last_undegraded":"2024-09-13T15:15:30.463232+0000","last_fullsized":"2024-09-13T15:15:30.463232+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:58.362919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"89'149","reported_seq":975,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463573+0000","last_change":"2024-09-13T15:13:03.079534+0000","last_active":"2024-09-13T15:15:30.463573+0000","last_peered":"2024-09-13T15:15:30.463573+0000","last_clean":"2024-09-13T15:15:30.463573+0000","last_became_active":"2024-09-13T15:12:38.941094+0000","last_became_peered":"2024-09-13T15:12:38.941094+0000","last_unstale":"2024-09-13T15:15:30.463573+0000","last_undegraded":"2024-09-13T15:15:30.463573+0000","last_fullsized":"2024-09-13T15:15:30.463573+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":456,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":3,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:03.079258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:03.079258+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:50.421978+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":806,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807873+0000","last_change":"2024-09-13T14:59:42.567925+0000","last_active":"2024-09-13T15:15:30.807873+0000","last_peered":"2024-09-13T15:15:30.807873+0000","last_clean":"2024-09-13T15:15:30.807873+0000","last_became_active":"2024-09-13T14:59:42.565380+0000","last_became_peered":"2024-09-13T14:59:42.565380+0000","last_unstale":"2024-09-13T15:15:30.807873+0000","last_undegraded":"2024-09-13T15:15:30.807873+0000","last_fullsized":"2024-09-13T15:15:30.807873+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:45.481989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"85'181","reported_seq":961,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.250122+0000","last_change":"2024-09-13T15:15:10.377063+0000","last_active":"2024-09-13T15:15:27.250122+0000","last_peered":"2024-09-13T15:15:27.250122+0000","last_clean":"2024-09-13T15:15:27.250122+0000","last_became_active":"2024-09-13T15:15:10.376415+0000","last_became_peered":"2024-09-13T15:15:10.376415+0000","last_unstale":"2024-09-13T15:15:27.250122+0000","last_undegraded":"2024-09-13T15:15:27.250122+0000","last_fullsized":"2024-09-13T15:15:27.250122+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:27:00.133957+0000","last_clean_scrub_stamp":"2024-09-13T14:27:00.133957+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:41.800784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084968999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1086,"num_read_kb":3423,"num_write":164,"num_write_kb":3376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":552,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.507891+0000","last_change":"2024-09-13T15:04:05.012487+0000","last_active":"2024-09-13T15:15:30.507891+0000","last_peered":"2024-09-13T15:15:30.507891+0000","last_clean":"2024-09-13T15:15:30.507891+0000","last_became_active":"2024-09-13T15:04:05.012159+0000","last_became_peered":"2024-09-13T15:04:05.012159+0000","last_unstale":"2024-09-13T15:15:30.507891+0000","last_undegraded":"2024-09-13T15:15:30.507891+0000","last_fullsized":"2024-09-13T15:15:30.507891+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:48.052172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"89'149","reported_seq":996,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.834011+0000","last_change":"2024-09-13T15:14:37.438852+0000","last_active":"2024-09-13T15:15:29.834011+0000","last_peered":"2024-09-13T15:15:29.834011+0000","last_clean":"2024-09-13T15:15:29.834011+0000","last_became_active":"2024-09-13T15:14:37.438385+0000","last_became_peered":"2024-09-13T15:14:37.438385+0000","last_unstale":"2024-09-13T15:15:29.834011+0000","last_undegraded":"2024-09-13T15:15:29.834011+0000","last_fullsized":"2024-09-13T15:15:29.834011+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:30.094914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:30.094914+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:12.899244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":535,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111911+0000","last_change":"2024-09-13T15:04:05.013275+0000","last_active":"2024-09-13T15:15:27.111911+0000","last_peered":"2024-09-13T15:15:27.111911+0000","last_clean":"2024-09-13T15:15:27.111911+0000","last_became_active":"2024-09-13T15:04:05.012504+0000","last_became_peered":"2024-09-13T15:04:05.012504+0000","last_unstale":"2024-09-13T15:15:27.111911+0000","last_undegraded":"2024-09-13T15:15:27.111911+0000","last_fullsized":"2024-09-13T15:15:27.111911+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:04.045530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"89'23","reported_seq":667,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111951+0000","last_change":"2024-09-13T15:08:31.193477+0000","last_active":"2024-09-13T15:15:27.111951+0000","last_peered":"2024-09-13T15:15:27.111951+0000","last_clean":"2024-09-13T15:15:27.111951+0000","last_became_active":"2024-09-13T15:08:31.181861+0000","last_became_peered":"2024-09-13T15:08:31.181861+0000","last_unstale":"2024-09-13T15:15:27.111951+0000","last_undegraded":"2024-09-13T15:15:27.111951+0000","last_fullsized":"2024-09-13T15:15:27.111951+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:19:02.424164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":33292,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"89'149","reported_seq":975,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.550089+0000","last_change":"2024-09-13T15:14:37.440742+0000","last_active":"2024-09-13T15:15:27.550089+0000","last_peered":"2024-09-13T15:15:27.550089+0000","last_clean":"2024-09-13T15:15:27.550089+0000","last_became_active":"2024-09-13T15:14:37.440304+0000","last_became_peered":"2024-09-13T15:14:37.440304+0000","last_unstale":"2024-09-13T15:15:27.550089+0000","last_undegraded":"2024-09-13T15:15:27.550089+0000","last_fullsized":"2024-09-13T15:15:27.550089+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:31.274137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:31.274137+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:36:46.299807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":944,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833167+0000","last_change":"2024-09-13T15:04:05.013108+0000","last_active":"2024-09-13T15:15:29.833167+0000","last_peered":"2024-09-13T15:15:29.833167+0000","last_clean":"2024-09-13T15:15:29.833167+0000","last_became_active":"2024-09-13T15:04:05.012847+0000","last_became_peered":"2024-09-13T15:04:05.012847+0000","last_unstale":"2024-09-13T15:15:29.833167+0000","last_undegraded":"2024-09-13T15:15:29.833167+0000","last_fullsized":"2024-09-13T15:15:29.833167+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:27.006037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":106,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.808090+0000","last_change":"2024-09-13T15:08:31.204593+0000","last_active":"2024-09-13T15:15:30.808090+0000","last_peered":"2024-09-13T15:15:30.808090+0000","last_clean":"2024-09-13T15:15:30.808090+0000","last_became_active":"2024-09-13T15:08:31.198883+0000","last_became_peered":"2024-09-13T15:08:31.198883+0000","last_unstale":"2024-09-13T15:15:30.808090+0000","last_undegraded":"2024-09-13T15:15:30.808090+0000","last_fullsized":"2024-09-13T15:15:30.808090+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:08.656620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"89'149","reported_seq":978,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807090+0000","last_change":"2024-09-13T15:13:00.998330+0000","last_active":"2024-09-13T15:15:30.807090+0000","last_peered":"2024-09-13T15:15:30.807090+0000","last_clean":"2024-09-13T15:15:30.807090+0000","last_became_active":"2024-09-13T15:12:50.190273+0000","last_became_peered":"2024-09-13T15:12:50.190273+0000","last_unstale":"2024-09-13T15:15:30.807090+0000","last_undegraded":"2024-09-13T15:15:30.807090+0000","last_fullsized":"2024-09-13T15:15:30.807090+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":475,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:13:00.998266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:13:00.998266+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:48:00.945235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":424,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.550055+0000","last_change":"2024-09-13T15:08:31.188790+0000","last_active":"2024-09-13T15:15:27.550055+0000","last_peered":"2024-09-13T15:15:27.550055+0000","last_clean":"2024-09-13T15:15:27.550055+0000","last_became_active":"2024-09-13T15:08:31.188441+0000","last_became_peered":"2024-09-13T15:08:31.188441+0000","last_unstale":"2024-09-13T15:15:27.550055+0000","last_undegraded":"2024-09-13T15:15:27.550055+0000","last_fullsized":"2024-09-13T15:15:27.550055+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:21:52.934163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":932,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463327+0000","last_change":"2024-09-13T15:08:31.197112+0000","last_active":"2024-09-13T15:15:30.463327+0000","last_peered":"2024-09-13T15:15:30.463327+0000","last_clean":"2024-09-13T15:15:30.463327+0000","last_became_active":"2024-09-13T15:08:31.196687+0000","last_became_peered":"2024-09-13T15:08:31.196687+0000","last_unstale":"2024-09-13T15:15:30.463327+0000","last_undegraded":"2024-09-13T15:15:30.463327+0000","last_fullsized":"2024-09-13T15:15:30.463327+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:47.532912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"89'149","reported_seq":838,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879042+0000","last_change":"2024-09-13T15:14:42.899519+0000","last_active":"2024-09-13T15:15:27.879042+0000","last_peered":"2024-09-13T15:15:27.879042+0000","last_clean":"2024-09-13T15:15:27.879042+0000","last_became_active":"2024-09-13T15:14:37.443576+0000","last_became_peered":"2024-09-13T15:14:37.443576+0000","last_unstale":"2024-09-13T15:15:27.879042+0000","last_undegraded":"2024-09-13T15:15:27.879042+0000","last_fullsized":"2024-09-13T15:15:27.879042+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:42.899422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:42.899422+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:27:22.515720+0000","scrub_duration":0.0016158069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":897,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807841+0000","last_change":"2024-09-13T14:59:42.570899+0000","last_active":"2024-09-13T15:15:30.807841+0000","last_peered":"2024-09-13T15:15:30.807841+0000","last_clean":"2024-09-13T15:15:30.807841+0000","last_became_active":"2024-09-13T14:59:42.564258+0000","last_became_peered":"2024-09-13T14:59:42.564258+0000","last_unstale":"2024-09-13T15:15:30.807841+0000","last_undegraded":"2024-09-13T15:15:30.807841+0000","last_fullsized":"2024-09-13T15:15:30.807841+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:11.061467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":962,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463277+0000","last_change":"2024-09-13T15:04:05.010410+0000","last_active":"2024-09-13T15:15:30.463277+0000","last_peered":"2024-09-13T15:15:30.463277+0000","last_clean":"2024-09-13T15:15:30.463277+0000","last_became_active":"2024-09-13T15:04:05.010119+0000","last_became_peered":"2024-09-13T15:04:05.010119+0000","last_unstale":"2024-09-13T15:15:30.463277+0000","last_undegraded":"2024-09-13T15:15:30.463277+0000","last_fullsized":"2024-09-13T15:15:30.463277+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:34:37.281867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"89'149","reported_seq":838,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111546+0000","last_change":"2024-09-13T15:14:41.744947+0000","last_active":"2024-09-13T15:15:27.111546+0000","last_peered":"2024-09-13T15:15:27.111546+0000","last_clean":"2024-09-13T15:15:27.111546+0000","last_became_active":"2024-09-13T15:14:37.439746+0000","last_became_peered":"2024-09-13T15:14:37.439746+0000","last_unstale":"2024-09-13T15:15:27.111546+0000","last_undegraded":"2024-09-13T15:15:27.111546+0000","last_fullsized":"2024-09-13T15:15:27.111546+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:41.744839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:41.744839+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:11:41.276005+0000","scrub_duration":0.001559206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":251,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.250077+0000","last_change":"2024-09-13T15:15:11.072803+0000","last_active":"2024-09-13T15:15:27.250077+0000","last_peered":"2024-09-13T15:15:27.250077+0000","last_clean":"2024-09-13T15:15:27.250077+0000","last_became_active":"2024-09-13T15:15:11.072373+0000","last_became_peered":"2024-09-13T15:15:11.072373+0000","last_unstale":"2024-09-13T15:15:27.250077+0000","last_undegraded":"2024-09-13T15:15:27.250077+0000","last_fullsized":"2024-09-13T15:15:27.250077+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:33.742137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":926,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463989+0000","last_change":"2024-09-13T14:59:42.556857+0000","last_active":"2024-09-13T15:15:30.463989+0000","last_peered":"2024-09-13T15:15:30.463989+0000","last_clean":"2024-09-13T15:15:30.463989+0000","last_became_active":"2024-09-13T14:59:42.556231+0000","last_became_peered":"2024-09-13T14:59:42.556231+0000","last_unstale":"2024-09-13T15:15:30.463989+0000","last_undegraded":"2024-09-13T15:15:30.463989+0000","last_fullsized":"2024-09-13T15:15:30.463989+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:30.064825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"89'149","reported_seq":908,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549650+0000","last_change":"2024-09-13T15:14:46.698051+0000","last_active":"2024-09-13T15:15:27.549650+0000","last_peered":"2024-09-13T15:15:27.549650+0000","last_clean":"2024-09-13T15:15:27.549650+0000","last_became_active":"2024-09-13T15:14:37.442325+0000","last_became_peered":"2024-09-13T15:14:37.442325+0000","last_unstale":"2024-09-13T15:15:27.549650+0000","last_undegraded":"2024-09-13T15:15:27.549650+0000","last_fullsized":"2024-09-13T15:15:27.549650+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:46.697941+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:14:46.697941+0000","last_clean_scrub_stamp":"2024-09-13T15:14:46.697941+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:38.304748+0000","scrub_duration":0.0015698439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":895,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807810+0000","last_change":"2024-09-13T15:08:31.195215+0000","last_active":"2024-09-13T15:15:30.807810+0000","last_peered":"2024-09-13T15:15:30.807810+0000","last_clean":"2024-09-13T15:15:30.807810+0000","last_became_active":"2024-09-13T15:08:31.194526+0000","last_became_peered":"2024-09-13T15:08:31.194526+0000","last_unstale":"2024-09-13T15:15:30.807810+0000","last_undegraded":"2024-09-13T15:15:30.807810+0000","last_fullsized":"2024-09-13T15:15:30.807810+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:38:56.358764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":898,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.832896+0000","last_change":"2024-09-13T14:41:41.486014+0000","last_active":"2024-09-13T15:15:29.832896+0000","last_peered":"2024-09-13T15:15:29.832896+0000","last_clean":"2024-09-13T15:15:29.832896+0000","last_became_active":"2024-09-13T14:41:41.485681+0000","last_became_peered":"2024-09-13T14:41:41.485681+0000","last_unstale":"2024-09-13T15:15:29.832896+0000","last_undegraded":"2024-09-13T15:15:29.832896+0000","last_fullsized":"2024-09-13T15:15:29.832896+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:42.104937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"89'149","reported_seq":926,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.549856+0000","last_change":"2024-09-13T15:14:42.777420+0000","last_active":"2024-09-13T15:15:27.549856+0000","last_peered":"2024-09-13T15:15:27.549856+0000","last_clean":"2024-09-13T15:15:27.549856+0000","last_became_active":"2024-09-13T15:14:37.440925+0000","last_became_peered":"2024-09-13T15:14:37.440925+0000","last_unstale":"2024-09-13T15:15:27.549856+0000","last_undegraded":"2024-09-13T15:15:27.549856+0000","last_fullsized":"2024-09-13T15:15:27.549856+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:42.777305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:42.777305+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:52:47.412591+0000","scrub_duration":0.0021482089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":763,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463705+0000","last_change":"2024-09-13T14:51:28.825419+0000","last_active":"2024-09-13T15:15:30.463705+0000","last_peered":"2024-09-13T15:15:30.463705+0000","last_clean":"2024-09-13T15:15:30.463705+0000","last_became_active":"2024-09-13T14:51:28.825054+0000","last_became_peered":"2024-09-13T14:51:28.825054+0000","last_unstale":"2024-09-13T15:15:30.463705+0000","last_undegraded":"2024-09-13T15:15:30.463705+0000","last_fullsized":"2024-09-13T15:15:30.463705+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:40:19.753274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":530,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508152+0000","last_change":"2024-09-13T15:00:14.949325+0000","last_active":"2024-09-13T15:15:30.508152+0000","last_peered":"2024-09-13T15:15:30.508152+0000","last_clean":"2024-09-13T15:15:30.508152+0000","last_became_active":"2024-09-13T15:00:14.949121+0000","last_became_peered":"2024-09-13T15:00:14.949121+0000","last_unstale":"2024-09-13T15:15:30.508152+0000","last_undegraded":"2024-09-13T15:15:30.508152+0000","last_fullsized":"2024-09-13T15:15:30.508152+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:32.647365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"89'149","reported_seq":877,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463204+0000","last_change":"2024-09-13T15:14:38.292132+0000","last_active":"2024-09-13T15:15:30.463204+0000","last_peered":"2024-09-13T15:15:30.463204+0000","last_clean":"2024-09-13T15:15:30.463204+0000","last_became_active":"2024-09-13T15:14:37.437551+0000","last_became_peered":"2024-09-13T15:14:37.437551+0000","last_unstale":"2024-09-13T15:15:30.463204+0000","last_undegraded":"2024-09-13T15:15:30.463204+0000","last_fullsized":"2024-09-13T15:15:30.463204+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:38.292076+0000","last_deep_scrub":"89'149","last_deep_scrub_stamp":"2024-09-13T15:10:26.504369+0000","last_clean_scrub_stamp":"2024-09-13T15:14:38.292076+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:27.703670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":514,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807302+0000","last_change":"2024-09-13T15:00:14.948643+0000","last_active":"2024-09-13T15:15:30.807302+0000","last_peered":"2024-09-13T15:15:30.807302+0000","last_clean":"2024-09-13T15:15:30.807302+0000","last_became_active":"2024-09-13T15:00:14.948255+0000","last_became_peered":"2024-09-13T15:00:14.948255+0000","last_unstale":"2024-09-13T15:15:30.807302+0000","last_undegraded":"2024-09-13T15:15:30.807302+0000","last_fullsized":"2024-09-13T15:15:30.807302+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:35.355246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":882,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807378+0000","last_change":"2024-09-13T15:08:31.194912+0000","last_active":"2024-09-13T15:15:30.807378+0000","last_peered":"2024-09-13T15:15:30.807378+0000","last_clean":"2024-09-13T15:15:30.807378+0000","last_became_active":"2024-09-13T15:08:31.194553+0000","last_became_peered":"2024-09-13T15:08:31.194553+0000","last_unstale":"2024-09-13T15:15:30.807378+0000","last_undegraded":"2024-09-13T15:15:30.807378+0000","last_fullsized":"2024-09-13T15:15:30.807378+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:17.457690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"89'149","reported_seq":860,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463759+0000","last_change":"2024-09-13T15:14:39.264123+0000","last_active":"2024-09-13T15:15:30.463759+0000","last_peered":"2024-09-13T15:15:30.463759+0000","last_clean":"2024-09-13T15:15:30.463759+0000","last_became_active":"2024-09-13T15:14:37.441075+0000","last_became_peered":"2024-09-13T15:14:37.441075+0000","last_unstale":"2024-09-13T15:15:30.463759+0000","last_undegraded":"2024-09-13T15:15:30.463759+0000","last_fullsized":"2024-09-13T15:15:30.463759+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:39.264072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:39.264072+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:12.034626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":540,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508273+0000","last_change":"2024-09-13T14:59:42.580759+0000","last_active":"2024-09-13T15:15:30.508273+0000","last_peered":"2024-09-13T15:15:30.508273+0000","last_clean":"2024-09-13T15:15:30.508273+0000","last_became_active":"2024-09-13T14:59:42.580505+0000","last_became_peered":"2024-09-13T14:59:42.580505+0000","last_unstale":"2024-09-13T15:15:30.508273+0000","last_undegraded":"2024-09-13T15:15:30.508273+0000","last_fullsized":"2024-09-13T15:15:30.508273+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:04.356619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":368,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833920+0000","last_change":"2024-09-13T15:08:46.073232+0000","last_active":"2024-09-13T15:15:29.833920+0000","last_peered":"2024-09-13T15:15:29.833920+0000","last_clean":"2024-09-13T15:15:29.833920+0000","last_became_active":"2024-09-13T15:08:46.072807+0000","last_became_peered":"2024-09-13T15:08:46.072807+0000","last_unstale":"2024-09-13T15:15:29.833920+0000","last_undegraded":"2024-09-13T15:15:29.833920+0000","last_fullsized":"2024-09-13T15:15:29.833920+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:07:11.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":528,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.508031+0000","last_change":"2024-09-13T14:59:42.554956+0000","last_active":"2024-09-13T15:15:30.508031+0000","last_peered":"2024-09-13T15:15:30.508031+0000","last_clean":"2024-09-13T15:15:30.508031+0000","last_became_active":"2024-09-13T14:59:42.554685+0000","last_became_peered":"2024-09-13T14:59:42.554685+0000","last_unstale":"2024-09-13T15:15:30.508031+0000","last_undegraded":"2024-09-13T15:15:30.508031+0000","last_fullsized":"2024-09-13T15:15:30.508031+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:12.042414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":396,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833894+0000","last_change":"2024-09-13T15:08:46.079600+0000","last_active":"2024-09-13T15:15:29.833894+0000","last_peered":"2024-09-13T15:15:29.833894+0000","last_clean":"2024-09-13T15:15:29.833894+0000","last_became_active":"2024-09-13T15:08:46.075458+0000","last_became_peered":"2024-09-13T15:08:46.075458+0000","last_unstale":"2024-09-13T15:15:29.833894+0000","last_undegraded":"2024-09-13T15:15:29.833894+0000","last_fullsized":"2024-09-13T15:15:29.833894+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:41.707452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"89'149","reported_seq":900,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.879002+0000","last_change":"2024-09-13T15:14:41.881812+0000","last_active":"2024-09-13T15:15:27.879002+0000","last_peered":"2024-09-13T15:15:27.879002+0000","last_clean":"2024-09-13T15:15:27.879002+0000","last_became_active":"2024-09-13T15:14:37.442716+0000","last_became_peered":"2024-09-13T15:14:37.442716+0000","last_unstale":"2024-09-13T15:15:27.879002+0000","last_undegraded":"2024-09-13T15:15:27.879002+0000","last_fullsized":"2024-09-13T15:15:27.879002+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:41.881687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:41.881687+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:12.841948+0000","scrub_duration":0.0015457030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":936,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463596+0000","last_change":"2024-09-13T15:00:14.950716+0000","last_active":"2024-09-13T15:15:30.463596+0000","last_peered":"2024-09-13T15:15:30.463596+0000","last_clean":"2024-09-13T15:15:30.463596+0000","last_became_active":"2024-09-13T15:00:14.950418+0000","last_became_peered":"2024-09-13T15:00:14.950418+0000","last_unstale":"2024-09-13T15:15:30.463596+0000","last_undegraded":"2024-09-13T15:15:30.463596+0000","last_fullsized":"2024-09-13T15:15:30.463596+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.736122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":959,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.463870+0000","last_change":"2024-09-13T15:04:05.011454+0000","last_active":"2024-09-13T15:15:30.463870+0000","last_peered":"2024-09-13T15:15:30.463870+0000","last_clean":"2024-09-13T15:15:30.463870+0000","last_became_active":"2024-09-13T15:04:05.011099+0000","last_became_peered":"2024-09-13T15:04:05.011099+0000","last_unstale":"2024-09-13T15:15:30.463870+0000","last_undegraded":"2024-09-13T15:15:30.463870+0000","last_fullsized":"2024-09-13T15:15:30.463870+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:22:37.280583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"89'149","reported_seq":861,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:29.833298+0000","last_change":"2024-09-13T15:14:37.462171+0000","last_active":"2024-09-13T15:15:29.833298+0000","last_peered":"2024-09-13T15:15:29.833298+0000","last_clean":"2024-09-13T15:15:29.833298+0000","last_became_active":"2024-09-13T15:14:37.441914+0000","last_became_peered":"2024-09-13T15:14:37.441914+0000","last_unstale":"2024-09-13T15:15:29.833298+0000","last_undegraded":"2024-09-13T15:15:29.833298+0000","last_fullsized":"2024-09-13T15:15:29.833298+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":502,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":5,"last_scrub":"89'149","last_scrub_stamp":"2024-09-13T15:14:37.462127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:24:31.653483+0000","last_clean_scrub_stamp":"2024-09-13T15:14:37.462127+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:19.978212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":561,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-09-13T15:15:27.111430+0000","last_change":"2024-09-13T15:04:05.012779+0000","last_active":"2024-09-13T15:15:27.111430+0000","last_peered":"2024-09-13T15:15:27.111430+0000","last_clean":"2024-09-13T15:15:27.111430+0000","last_became_active":"2024-09-13T15:04:05.012471+0000","last_became_peered":"2024-09-13T15:04:05.012471+0000","last_unstale":"2024-09-13T15:15:27.111430+0000","last_undegraded":"2024-09-13T15:15:27.111430+0000","last_fullsized":"2024-09-13T15:15:27.111430+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:14.273202+0000","last_clean_scrub_stamp":"2024-09-13T14:28:14.273202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:38:27.433582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":922,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-09-13T15:15:30.807728+0000","last_change":"2024-09-13T15:00:14.952752+0000","last_active":"2024-09-13T15:15:30.807728+0000","last_peered":"2024-09-13T15:15:30.807728+0000","last_clean":"2024-09-13T15:15:30.807728+0000","last_became_active":"2024-09-13T15:00:14.952277+0000","last_became_peered":"2024-09-13T15:00:14.952277+0000","last_unstale":"2024-09-13T15:15:30.807728+0000","last_undegraded":"2024-09-13T15:15:30.807728+0000","last_fullsized":"2024-09-13T15:15:30.807728+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T14:28:13.267497+0000","last_clean_scrub_stamp":"2024-09-13T14:28:13.267497+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:45.140843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2910,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5554,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":36578,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24998,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5941,"num_read_kb":12954,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234588,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529596,"kb_used_data":7088,"kb_used_omap":1,"kb_used_meta":522494,"kb_avail":93215556,"statfs":{"total":95995035648,"available":95452729344,"internally_reserved":0,"allocated":7258112,"data_stored":4439817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":535034315},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496224,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":728172,"kb_used_data":7136,"kb_used_omap":1,"kb_used_meta":721022,"kb_avail":93016980,"statfs":{"total":95995035648,"available":95249387520,"internally_reserved":0,"allocated":7307264,"data_stored":4504409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":738326983},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757859,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876588,"kb_used_data":7072,"kb_used_omap":1,"kb_used_meta":869502,"kb_avail":92868564,"statfs":{"total":95995035648,"available":95097409536,"internally_reserved":0,"allocated":7241728,"data_stored":4435574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890370503},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019499,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850100,"kb_used_data":6824,"kb_used_omap":1,"kb_used_meta":843262,"kb_avail":92895052,"statfs":{"total":95995035648,"available":95124533248,"internally_reserved":0,"allocated":6987776,"data_stored":4177368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863500742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":108,"seq":463856468383,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989668,"kb_used_data":11224,"kb_used_omap":4,"kb_used_meta":978427,"kb_avail":92755484,"statfs":{"total":95995035648,"available":94981615616,"internally_reserved":0,"allocated":11493376,"data_stored":8694574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4956,"internal_metadata":1001909412},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":328,"seq":1408749273283,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433712,"kb_used_data":11172,"kb_used_omap":12,"kb_used_meta":422515,"kb_avail":93311440,"statfs":{"total":95995035648,"available":95550914560,"internally_reserved":0,"allocated":11440128,"data_stored":8630130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12416,"internal_metadata":432656256},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":98,"seq":420906795435,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895076,"kb_used_data":6808,"kb_used_omap":7,"kb_used_meta":888248,"kb_avail":92850076,"statfs":{"total":95995035648,"available":95078477824,"internally_reserved":0,"allocated":6971392,"data_stored":4172519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7566,"internal_metadata":909566578},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":321,"seq":1378684502217,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":598120,"kb_used_data":11228,"kb_used_omap":10,"kb_used_meta":586869,"kb_avail":93147032,"statfs":{"total":95995035648,"available":95382560768,"internally_reserved":0,"allocated":11497472,"data_stored":8695088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10561,"internal_metadata":600954559},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4127,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8967,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5972,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6695,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3364,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:16:20.442 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-13T15:16:20.442 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:16:20.628 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9f6577a310db4e1dad891c305d37716b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-13T15:16:20.631 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-13T15:16:20.640 INFO:tasks.cephadm:Teardown begin 2024-09-13T15:16:20.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:16:20.682 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:20 smithi154 ceph-mon[102314]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:16:20.682 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:20 smithi154 ceph-mon[107038]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:16:20.701 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:16:20.731 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:20 smithi172 ceph-mon[100696]: from='client.36804 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:16:20.753 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-13T15:16:20.753 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:16:20.771 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:16:20.817 INFO:tasks.cephadm:Stopping all daemons... 2024-09-13T15:16:20.817 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-13T15:16:20.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a 2024-09-13T15:16:20.970 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:20 smithi154 systemd[1]: Stopping Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:21.394 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T15:16:21.139+0000 7ff2a60b3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:16:21.394 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a[102310]: 2024-09-13T15:16:21.139+0000 7ff2a60b3640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-13T15:16:21.395 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:21 smithi154 podman[356830]: 2024-09-13 15:16:21.241432659 +0000 UTC m=+0.233176795 container died c1bb51eb0c909c7900de8536e457b99383828a0a8794dc4d8d534fc5746b3065 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-13T15:16:21.395 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:15:16:21] ENGINE Bus STOPPING 2024-09-13T15:16:21.395 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:15:16:21] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-13T15:16:21.395 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:15:16:21] ENGINE Bus STOPPED 2024-09-13T15:16:21.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:15:16:21] ENGINE Bus STARTING 2024-09-13T15:16:21.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:15:16:21] ENGINE Serving on http://:::9283 2024-09-13T15:16:21.721 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: [13/Sep/2024:15:16:21] ENGINE Bus STARTED 2024-09-13T15:16:22.001 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:21 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y[84630]: ::ffff:172.21.15.172 - - [13/Sep/2024:15:16:21] "GET /metrics HTTP/1.1" 200 34224 "" "Prometheus/2.51.0" 2024-09-13T15:16:22.001 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:21 smithi154 podman[356830]: 2024-09-13 15:16:21.748583697 +0000 UTC m=+0.740327833 container cleanup c1bb51eb0c909c7900de8536e457b99383828a0a8794dc4d8d534fc5746b3065 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.license=GPLv2) 2024-09-13T15:16:22.002 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:21 smithi154 bash[356830]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a 2024-09-13T15:16:22.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service' 2024-09-13T15:16:22.262 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:22 smithi154 podman[356842]: 2024-09-13 15:16:22.000307355 +0000 UTC m=+0.754412066 container remove c1bb51eb0c909c7900de8536e457b99383828a0a8794dc4d8d534fc5746b3065 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:16:22.262 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:22 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-13T15:16:22.262 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:22 smithi154 systemd[1]: Stopped Ceph mon.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T15:16:22.262 INFO:journalctl@ceph.mon.a.smithi154.stdout:Sep 13 15:16:22 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.a.service: Consumed 36.802s CPU time. 2024-09-13T15:16:22.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:22.278 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-13T15:16:22.278 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-13T15:16:22.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.c 2024-09-13T15:16:22.605 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:22 smithi154 systemd[1]: Stopping Ceph mon.c for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:22.970 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c[107034]: 2024-09-13T15:16:22.602+0000 7f498e1f9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:16:22.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:22 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c[107034]: 2024-09-13T15:16:22.602+0000 7f498e1f9640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:16:22.971 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:22 smithi154 podman[356964]: 2024-09-13 15:16:22.746665569 +0000 UTC m=+0.244117384 container died fcefc4676e768076d37e06c4f614daf2581dd877ad0a3f6d7bdbb1fdf55e84c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2) 2024-09-13T15:16:23.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:23 smithi154 podman[356964]: 2024-09-13 15:16:23.245875368 +0000 UTC m=+0.743327251 container cleanup fcefc4676e768076d37e06c4f614daf2581dd877ad0a3f6d7bdbb1fdf55e84c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T15:16:23.470 INFO:journalctl@ceph.mon.c.smithi154.stdout:Sep 13 15:16:23 smithi154 bash[356964]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-c 2024-09-13T15:16:23.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.c.service' 2024-09-13T15:16:23.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:23.708 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-13T15:16:23.709 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-13T15:16:23.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.b 2024-09-13T15:16:23.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:23 smithi172 systemd[1]: Stopping Ceph mon.b for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:24.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b[100692]: 2024-09-13T15:16:24.258+0000 7f968d36f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:16:24.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b[100692]: 2024-09-13T15:16:24.258+0000 7f968d36f640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-13T15:16:24.736 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:24 smithi172 podman[265110]: 2024-09-13 15:16:24.472726743 +0000 UTC m=+0.313173395 container died 50c3380c21bb6a99cfa6fdbea685b4c17371383311dbd2663c639bf0fb072421 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-13T15:16:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:24 smithi172 podman[265110]: 2024-09-13 15:16:24.900096051 +0000 UTC m=+0.740542701 container cleanup 50c3380c21bb6a99cfa6fdbea685b4c17371383311dbd2663c639bf0fb072421 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T15:16:24.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:24 smithi172 bash[265110]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b 2024-09-13T15:16:25.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.b.service' 2024-09-13T15:16:25.361 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:25 smithi172 podman[265122]: 2024-09-13 15:16:25.085131038 +0000 UTC m=+0.612623305 container remove 50c3380c21bb6a99cfa6fdbea685b4c17371383311dbd2663c639bf0fb072421 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mon-b, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-13T15:16:25.361 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:25 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-13T15:16:25.361 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:25 smithi172 systemd[1]: Stopped Ceph mon.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T15:16:25.361 INFO:journalctl@ceph.mon.b.smithi172.stdout:Sep 13 15:16:25 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mon.b.service: Consumed 22.037s CPU time. 2024-09-13T15:16:25.392 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:25.393 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-13T15:16:25.393 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-13T15:16:25.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y 2024-09-13T15:16:25.720 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:25 smithi154 systemd[1]: Stopping Ceph mgr.y for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:26.012 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:25 smithi154 podman[357082]: 2024-09-13 15:16:25.733495642 +0000 UTC m=+0.148485442 container died fc4464b991bdf848adc3d9cdc7a0286f24c11ce3d960fbec45543707885b0966 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-13T15:16:26.404 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:26 smithi154 podman[357082]: 2024-09-13 15:16:26.167855463 +0000 UTC m=+0.582845263 container cleanup fc4464b991bdf848adc3d9cdc7a0286f24c11ce3d960fbec45543707885b0966 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:16:26.404 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Sep 13 15:16:26 smithi154 bash[357082]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-y 2024-09-13T15:16:26.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-13T15:16:26.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:26.656 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-13T15:16:26.656 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-13T15:16:26.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x 2024-09-13T15:16:26.978 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 15:16:26 smithi172 systemd[1]: Stopping Ceph mgr.x for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:27.241 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 15:16:26 smithi172 podman[265231]: 2024-09-13 15:16:26.977279532 +0000 UTC m=+0.131397958 container died b3bb4b6384823601d2cae6e5ce7da8821f119fdfa6b8794026847c236f488dbc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-13T15:16:27.665 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 15:16:27 smithi172 podman[265231]: 2024-09-13 15:16:27.337592186 +0000 UTC m=+0.491710618 container cleanup b3bb4b6384823601d2cae6e5ce7da8821f119fdfa6b8794026847c236f488dbc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:16:27.665 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Sep 13 15:16:27 smithi172 bash[265231]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-mgr-x 2024-09-13T15:16:27.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-13T15:16:27.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:27.847 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-13T15:16:27.847 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T15:16:27.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0 2024-09-13T15:16:28.149 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:27 smithi154 systemd[1]: Stopping Ceph osd.0 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:28.470 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[253281]: 2024-09-13T15:16:28.147+0000 7f746573a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:16:28.470 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[253281]: 2024-09-13T15:16:28.147+0000 7f746573a640 -1 osd.0 525 *** Got signal Terminated *** 2024-09-13T15:16:28.471 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:28 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0[253281]: 2024-09-13T15:16:28.147+0000 7f746573a640 -1 osd.0 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:16:33.970 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:33 smithi154 podman[357204]: 2024-09-13 15:16:33.709474545 +0000 UTC m=+5.672589753 container died ab20bf2d0511297fe9a2d35b428088bf7b0849df4cf6d152a0f7dfc94c7f38f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-09-13T15:16:34.381 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:34 smithi154 podman[357204]: 2024-09-13 15:16:34.173083565 +0000 UTC m=+6.136198776 container cleanup ab20bf2d0511297fe9a2d35b428088bf7b0849df4cf6d152a0f7dfc94c7f38f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-13T15:16:34.381 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:34 smithi154 bash[357204]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0 2024-09-13T15:16:34.721 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:34 smithi154 podman[357219]: 2024-09-13 15:16:34.516693734 +0000 UTC m=+0.806169146 container remove ab20bf2d0511297fe9a2d35b428088bf7b0849df4cf6d152a0f7dfc94c7f38f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-13T15:16:35.189 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:34 smithi154 podman[357317]: 2024-09-13 15:16:34.804368944 +0000 UTC m=+0.018425182 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:16:35.189 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:34 smithi154 podman[357317]: 2024-09-13 15:16:34.909309674 +0000 UTC m=+0.123365926 container create c95751a30dabb084699a45b916647c6b75919c11fe073d95490693f6a8f211dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2) 2024-09-13T15:16:35.189 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:35 smithi154 podman[357317]: 2024-09-13 15:16:35.136830735 +0000 UTC m=+0.350886974 container init c95751a30dabb084699a45b916647c6b75919c11fe073d95490693f6a8f211dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-13T15:16:35.189 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:35 smithi154 podman[357317]: 2024-09-13 15:16:35.14115715 +0000 UTC m=+0.355213388 container start c95751a30dabb084699a45b916647c6b75919c11fe073d95490693f6a8f211dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T15:16:35.471 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:35 smithi154 podman[357317]: 2024-09-13 15:16:35.23040594 +0000 UTC m=+0.444462184 container attach c95751a30dabb084699a45b916647c6b75919c11fe073d95490693f6a8f211dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T15:16:35.471 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:35 smithi154 podman[357317]: 2024-09-13 15:16:35.357105012 +0000 UTC m=+0.571161251 container died c95751a30dabb084699a45b916647c6b75919c11fe073d95490693f6a8f211dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T15:16:36.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:16:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:16:35.858Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T15:16:36.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:16:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:16:35.858Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T15:16:36.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:16:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:16:35.860Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T15:16:36.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:16:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:16:35.860Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T15:16:36.242 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:16:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:16:35.860Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T15:16:36.243 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:16:35 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:16:35.863Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.154:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.154:8765: connect: connection refused" 2024-09-13T15:16:36.263 INFO:journalctl@ceph.osd.0.smithi154.stdout:Sep 13 15:16:36 smithi154 podman[357341]: 2024-09-13 15:16:36.074666153 +0000 UTC m=+0.709377309 container remove c95751a30dabb084699a45b916647c6b75919c11fe073d95490693f6a8f211dc (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-13T15:16:36.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T15:16:36.309 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:36.309 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T15:16:36.310 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-13T15:16:36.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.1 2024-09-13T15:16:36.671 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:36 smithi154 systemd[1]: Stopping Ceph osd.1 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:36.970 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:36 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T15:16:36.669+0000 7f48553ac640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:16:36.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:36 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T15:16:36.669+0000 7f48553ac640 -1 osd.1 525 *** Got signal Terminated *** 2024-09-13T15:16:36.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:36 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1[123114]: 2024-09-13T15:16:36.669+0000 7f48553ac640 -1 osd.1 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:16:42.534 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:42 smithi154 podman[357423]: 2024-09-13 15:16:42.245472581 +0000 UTC m=+5.717535833 container died 9199417c7bce908d26996813f14ad9f327e5cb6ecbf495f04ade49c5f6e39a88 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-13T15:16:42.970 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:42 smithi154 podman[357423]: 2024-09-13 15:16:42.727716872 +0000 UTC m=+6.199780124 container cleanup 9199417c7bce908d26996813f14ad9f327e5cb6ecbf495f04ade49c5f6e39a88 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-13T15:16:42.971 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:42 smithi154 bash[357423]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1 2024-09-13T15:16:43.339 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:43 smithi154 podman[357441]: 2024-09-13 15:16:43.021415678 +0000 UTC m=+0.765046491 container remove 9199417c7bce908d26996813f14ad9f327e5cb6ecbf495f04ade49c5f6e39a88 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-13T15:16:43.659 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:43 smithi154 podman[357529]: 2024-09-13 15:16:43.339016165 +0000 UTC m=+0.110469283 container create ce7ef363c88fa933b73f2cde2a73095a583f55139f49a2f86d6949d147f09cb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T15:16:43.659 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:43 smithi154 podman[357529]: 2024-09-13 15:16:43.244363249 +0000 UTC m=+0.015816250 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:16:43.660 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:43 smithi154 podman[357529]: 2024-09-13 15:16:43.558452714 +0000 UTC m=+0.329905725 container init ce7ef363c88fa933b73f2cde2a73095a583f55139f49a2f86d6949d147f09cb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True) 2024-09-13T15:16:43.660 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:43 smithi154 podman[357529]: 2024-09-13 15:16:43.562112676 +0000 UTC m=+0.333565688 container start ce7ef363c88fa933b73f2cde2a73095a583f55139f49a2f86d6949d147f09cb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-13T15:16:43.660 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:43 smithi154 podman[357529]: 2024-09-13 15:16:43.658013785 +0000 UTC m=+0.429466788 container attach ce7ef363c88fa933b73f2cde2a73095a583f55139f49a2f86d6949d147f09cb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-aclamk-cbt-combined-d58f730) 2024-09-13T15:16:43.970 INFO:journalctl@ceph.osd.1.smithi154.stdout:Sep 13 15:16:43 smithi154 podman[357529]: 2024-09-13 15:16:43.755422829 +0000 UTC m=+0.526875835 container died ce7ef363c88fa933b73f2cde2a73095a583f55139f49a2f86d6949d147f09cb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-13T15:16:44.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.1.service' 2024-09-13T15:16:44.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:44.472 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-13T15:16:44.473 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T15:16:44.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2 2024-09-13T15:16:44.720 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:44 smithi154 systemd[1]: Stopping Ceph osd.2 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:45.220 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:44 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[257425]: 2024-09-13T15:16:44.798+0000 7f6fefec1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:16:45.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:44 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[257425]: 2024-09-13T15:16:44.798+0000 7f6fefec1640 -1 osd.2 525 *** Got signal Terminated *** 2024-09-13T15:16:45.221 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:44 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2[257425]: 2024-09-13T15:16:44.798+0000 7f6fefec1640 -1 osd.2 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:16:50.608 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:50 smithi154 podman[357632]: 2024-09-13 15:16:50.313907599 +0000 UTC m=+5.651657691 container died 7bcbe6e813375b16d2cb5144dfbf05e1f770e37ee9bcab210bc8ea34b8e4672e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-13T15:16:50.970 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:50 smithi154 podman[357632]: 2024-09-13 15:16:50.781878972 +0000 UTC m=+6.119629061 container cleanup 7bcbe6e813375b16d2cb5144dfbf05e1f770e37ee9bcab210bc8ea34b8e4672e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T15:16:50.971 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:50 smithi154 bash[357632]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2 2024-09-13T15:16:51.385 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:51 smithi154 podman[357647]: 2024-09-13 15:16:51.074900786 +0000 UTC m=+0.764493349 container remove 7bcbe6e813375b16d2cb5144dfbf05e1f770e37ee9bcab210bc8ea34b8e4672e (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-13T15:16:51.696 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:51 smithi154 podman[357737]: 2024-09-13 15:16:51.287427157 +0000 UTC m=+0.015975625 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:16:51.696 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:51 smithi154 podman[357737]: 2024-09-13 15:16:51.400988783 +0000 UTC m=+0.129537244 container create f02c2d241dde267825b203cb02db5e53eece74e751a44cf9cb3fcfcdcaa185f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:16:51.697 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:51 smithi154 podman[357737]: 2024-09-13 15:16:51.61176166 +0000 UTC m=+0.340310125 container init f02c2d241dde267825b203cb02db5e53eece74e751a44cf9cb3fcfcdcaa185f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:16:51.697 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:51 smithi154 podman[357737]: 2024-09-13 15:16:51.615093867 +0000 UTC m=+0.343642337 container start f02c2d241dde267825b203cb02db5e53eece74e751a44cf9cb3fcfcdcaa185f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-aclamk-cbt-combined-d58f730, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-13T15:16:51.697 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:51 smithi154 podman[357737]: 2024-09-13 15:16:51.695058265 +0000 UTC m=+0.423606741 container attach f02c2d241dde267825b203cb02db5e53eece74e751a44cf9cb3fcfcdcaa185f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-aclamk-cbt-combined-d58f730, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-13T15:16:51.970 INFO:journalctl@ceph.osd.2.smithi154.stdout:Sep 13 15:16:51 smithi154 podman[357737]: 2024-09-13 15:16:51.80612659 +0000 UTC m=+0.534675060 container died f02c2d241dde267825b203cb02db5e53eece74e751a44cf9cb3fcfcdcaa185f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-aclamk-cbt-combined-d58f730, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-13T15:16:52.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T15:16:52.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:16:52.528 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T15:16:52.528 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-13T15:16:52.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.3 2024-09-13T15:16:52.720 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:52 smithi154 systemd[1]: Stopping Ceph osd.3 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:16:53.221 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T15:16:52.843+0000 7fb5f83f3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:16:53.221 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T15:16:52.843+0000 7fb5f83f3640 -1 osd.3 525 *** Got signal Terminated *** 2024-09-13T15:16:53.221 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:52 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T15:16:52.843+0000 7fb5f83f3640 -1 osd.3 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:16:54.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:16:54 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:16:54.024+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:16:54.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:16:54 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:16:54.546+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:16:55.480 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:16:54 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:16:54.993+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:16:55.741 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:16:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:16:55.591+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:16:55.742 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:16:55 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:16:55.541+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:16:56.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:16:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:16:56.017+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:16:56.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:56 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T15:16:56.636+0000 7fb5f0a00640 -1 osd.3 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.622397+0000 front 2024-09-13T15:16:30.622442+0000 (oldest deadline 2024-09-13T15:16:55.922569+0000) 2024-09-13T15:16:56.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:16:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:16:56.527+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:16:56.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:16:56 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:16:56.591+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:16:57.480 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:16:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:16:57.013+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:16:57.741 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:16:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:16:57.478+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:16:57.742 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:16:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:16:57.596+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:16:57.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:57 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3[132673]: 2024-09-13T15:16:57.682+0000 7fb5f0a00640 -1 osd.3 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.622397+0000 front 2024-09-13T15:16:30.622442+0000 (oldest deadline 2024-09-13T15:16:55.922569+0000) 2024-09-13T15:16:58.241 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:16:57 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:16:57.973+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:16:58.242 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:16:58 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:16:58.141+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:16:58.626 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:58 smithi154 podman[357840]: 2024-09-13 15:16:58.369280322 +0000 UTC m=+5.623362909 container died df625f79bb4780e5a25547e8f451b439c6252d53276264f09cd1b28f154201d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-13T15:16:58.741 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:16:58 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:16:58.464+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:16:58.741 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:16:58 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:16:58.601+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:16:58.970 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:58 smithi154 podman[357840]: 2024-09-13 15:16:58.826475343 +0000 UTC m=+6.080557917 container cleanup df625f79bb4780e5a25547e8f451b439c6252d53276264f09cd1b28f154201d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-13T15:16:58.971 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:58 smithi154 bash[357840]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3 2024-09-13T15:16:59.241 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:16:58 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:16:58.946+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:16:59.242 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:16:59 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:16:59.145+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:16:59.444 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:59 smithi154 podman[357856]: 2024-09-13 15:16:59.119642019 +0000 UTC m=+0.748724644 container remove df625f79bb4780e5a25547e8f451b439c6252d53276264f09cd1b28f154201d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T15:16:59.721 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:59 smithi154 podman[357949]: 2024-09-13 15:16:59.34609321 +0000 UTC m=+0.016102438 image pull fec07016a2b0bf363d0b69240081049b65cf13f1c44f6a7196cf8309aeb428cd quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f 2024-09-13T15:16:59.721 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:59 smithi154 podman[357949]: 2024-09-13 15:16:59.453740063 +0000 UTC m=+0.123749320 container create d8f11506c936951dbd9c910b33720a95ce94d9779505d8ea21fc2316e5417449 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:16:59.721 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:59 smithi154 podman[357949]: 2024-09-13 15:16:59.681118554 +0000 UTC m=+0.351127769 container init d8f11506c936951dbd9c910b33720a95ce94d9779505d8ea21fc2316e5417449 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T15:16:59.721 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:59 smithi154 podman[357949]: 2024-09-13 15:16:59.68531053 +0000 UTC m=+0.355319752 container start d8f11506c936951dbd9c910b33720a95ce94d9779505d8ea21fc2316e5417449 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-aclamk-cbt-combined-d58f730, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:16:59.741 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:16:59 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:16:59.476+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:16:59.741 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:16:59 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:16:59.557+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:17:00.169 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:59 smithi154 podman[357949]: 2024-09-13 15:16:59.772875495 +0000 UTC m=+0.442884724 container attach d8f11506c936951dbd9c910b33720a95ce94d9779505d8ea21fc2316e5417449 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-aclamk-cbt-combined-d58f730, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-09-13T15:17:00.169 INFO:journalctl@ceph.osd.3.smithi154.stdout:Sep 13 15:16:59 smithi154 podman[357949]: 2024-09-13 15:16:59.876562465 +0000 UTC m=+0.546571692 container died d8f11506c936951dbd9c910b33720a95ce94d9779505d8ea21fc2316e5417449 (image=quay.ceph.io/ceph-ci/ceph@sha256:914e971332a58dba507db28f0971f6f283c67edecc2d9a6537eea399760b8c1f, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=wip-aclamk-cbt-combined-d58f730, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-13T15:17:00.241 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:16:59 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:16:59.984+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:00.241 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:00 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:00.154+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:00.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.3.service' 2024-09-13T15:17:00.554 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:17:00.554 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-13T15:17:00.554 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T15:17:00.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.4 2024-09-13T15:17:00.858 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:00 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:00.525+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:00.858 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:00 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:00.553+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:17:00.858 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:00 smithi172 systemd[1]: Stopping Ceph osd.4 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:00.858 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:00 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:00.856+0000 7f441bc09640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:17:00.859 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:00 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:00.856+0000 7f441bc09640 -1 osd.4 525 *** Got signal Terminated *** 2024-09-13T15:17:00.859 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:00 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:00.856+0000 7f441bc09640 -1 osd.4 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:17:01.134 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:00 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:00.983+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:01.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:01 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:01.133+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:01.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:01 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:01.597+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:17:01.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:01 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:01.490+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:02.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:02 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:02.016+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:02.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:02 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:02.136+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:02.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:02 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:02.534+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:02.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:02 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:02.566+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:17:02.992 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:02 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:02.566+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.818897+0000 front 2024-09-13T15:16:37.819009+0000 (oldest deadline 2024-09-13T15:17:02.518805+0000) 2024-09-13T15:17:03.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:03 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:03.031+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:03.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:03 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:03.153+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:03.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:03 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:03.538+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:17:03.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:03 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:03.538+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.818897+0000 front 2024-09-13T15:16:37.819009+0000 (oldest deadline 2024-09-13T15:17:02.518805+0000) 2024-09-13T15:17:03.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:03 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:03.546+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:04.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:04.157+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:04.492 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:04.068+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:04.492 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:04.068+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:04.988 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:04.552+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:04.988 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:04.545+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:17:04.989 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:04 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:04.545+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.818897+0000 front 2024-09-13T15:16:37.819009+0000 (oldest deadline 2024-09-13T15:17:02.518805+0000) 2024-09-13T15:17:05.241 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:05.040+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:05.242 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:05.040+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:05.242 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:05.174+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:05.242 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:05.174+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:05.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:05.520+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:05.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:05.559+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:29.018042+0000 front 2024-09-13T15:16:29.018028+0000 (oldest deadline 2024-09-13T15:16:53.717880+0000) 2024-09-13T15:17:05.992 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:05 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4[37568]: 2024-09-13T15:17:05.559+0000 7f4414215640 -1 osd.4 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.818897+0000 front 2024-09-13T15:16:37.819009+0000 (oldest deadline 2024-09-13T15:17:02.518805+0000) 2024-09-13T15:17:06.334 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:06 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:06.082+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:06.334 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:06 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:06.082+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:06.334 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:06 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:06.157+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:06.334 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:06 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:06.157+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:06.640 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:06 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:06.532+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:06.640 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:06 smithi172 podman[265369]: 2024-09-13 15:17:06.333048705 +0000 UTC m=+5.578950666 container died 7490623a5a63c30c1625f3257394e0f94c14a14d2a00c9b7767330a77ecef042 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-13T15:17:06.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:06 smithi172 podman[265369]: 2024-09-13 15:17:06.714274547 +0000 UTC m=+5.960176495 container cleanup 7490623a5a63c30c1625f3257394e0f94c14a14d2a00c9b7767330a77ecef042 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-13T15:17:06.991 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:06 smithi172 bash[265369]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4 2024-09-13T15:17:07.304 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:06 smithi172 podman[265385]: 2024-09-13 15:17:06.990768511 +0000 UTC m=+0.644400995 container remove 7490623a5a63c30c1625f3257394e0f94c14a14d2a00c9b7767330a77ecef042 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T15:17:07.305 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:07.033+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:07.305 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:07.033+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:07.305 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:07.130+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:07.305 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:07.130+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:07.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:07 smithi172 podman[265476]: 2024-09-13 15:17:07.206787656 +0000 UTC m=+0.016497310 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:17:07.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:07 smithi172 podman[265476]: 2024-09-13 15:17:07.333539222 +0000 UTC m=+0.143248875 container create ee9e081509b1b7ccea3d3b3a9368366cbad2956fc5ff2f4b57a645afe373a52d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4-deactivate, RELEASE=reef-2c01458, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-09-13T15:17:07.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:07 smithi172 podman[265476]: 2024-09-13 15:17:07.552620559 +0000 UTC m=+0.362330212 container init ee9e081509b1b7ccea3d3b3a9368366cbad2956fc5ff2f4b57a645afe373a52d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=reef-2c01458) 2024-09-13T15:17:07.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:07 smithi172 podman[265476]: 2024-09-13 15:17:07.555994462 +0000 UTC m=+0.365704119 container start ee9e081509b1b7ccea3d3b3a9368366cbad2956fc5ff2f4b57a645afe373a52d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-13T15:17:07.870 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:07.556+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:07.870 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:07 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:07.556+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:07.870 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:07 smithi172 podman[265476]: 2024-09-13 15:17:07.687198455 +0000 UTC m=+0.496908109 container attach ee9e081509b1b7ccea3d3b3a9368366cbad2956fc5ff2f4b57a645afe373a52d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-13T15:17:08.166 INFO:journalctl@ceph.osd.4.smithi172.stdout:Sep 13 15:17:07 smithi172 podman[265476]: 2024-09-13 15:17:07.869687163 +0000 UTC m=+0.679396817 container died ee9e081509b1b7ccea3d3b3a9368366cbad2956fc5ff2f4b57a645afe373a52d (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-13T15:17:08.167 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:08.065+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:08.167 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:08.065+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:08.450 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:08.173+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:08.450 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:08.173+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:08.472 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T15:17:08.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:17:08.513 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T15:17:08.513 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-13T15:17:08.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.5 2024-09-13T15:17:08.735 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:08 smithi172 systemd[1]: Stopping Ceph osd.5 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:08.736 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:08.598+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:08.736 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:08.598+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:08.991 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:08.835+0000 7f151ac5b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:17:08.991 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:08.835+0000 7f151ac5b640 -1 osd.5 525 *** Got signal Terminated *** 2024-09-13T15:17:08.991 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:08 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:08.835+0000 7f151ac5b640 -1 osd.5 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:17:09.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:09 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:09.083+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:09.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:09 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:09.083+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:09.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:09 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:09.136+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:09.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:09 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:09.136+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:09.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:09 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:09.622+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:09.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:09 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:09.622+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:10.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:10 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:10.090+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:10.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:10 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:10.090+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:10.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:10 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:10.156+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:10.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:10 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:10.156+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:10.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:10 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:10.579+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:10.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:10 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:10.579+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:11.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:11.110+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:11.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:11.110+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:11.492 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:11.110+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:45.361628+0000 front 2024-09-13T15:16:45.361665+0000 (oldest deadline 2024-09-13T15:17:10.661563+0000) 2024-09-13T15:17:11.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:11.173+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:11.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:11.173+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:11.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:11.592+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:11.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:11 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:11.592+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:12.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:12.089+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:12.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:12.089+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:12.492 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:12.089+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:45.361628+0000 front 2024-09-13T15:16:45.361665+0000 (oldest deadline 2024-09-13T15:17:10.661563+0000) 2024-09-13T15:17:12.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:12.124+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:12.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:12.124+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:12.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:12.583+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:12.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:12 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:12.583+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:13.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:13.109+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:13.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:13.109+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:13.492 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:13.104+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.259014+0000 front 2024-09-13T15:16:30.259193+0000 (oldest deadline 2024-09-13T15:16:53.759150+0000) 2024-09-13T15:17:13.492 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:13.104+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:36.660308+0000 front 2024-09-13T15:16:36.660496+0000 (oldest deadline 2024-09-13T15:17:03.660517+0000) 2024-09-13T15:17:13.492 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5[43362]: 2024-09-13T15:17:13.104+0000 7f1513267640 -1 osd.5 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:45.361628+0000 front 2024-09-13T15:16:45.361665+0000 (oldest deadline 2024-09-13T15:17:10.661563+0000) 2024-09-13T15:17:13.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:13.573+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:13.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:13 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:13.573+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:14.491 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:14 smithi172 podman[265577]: 2024-09-13 15:17:14.322049875 +0000 UTC m=+5.585844895 container died bd1089408fdb85b977d7d29d36af75e4e1b333807aad44c3ad82a4b8b1678c2e (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=reef-2c01458, CEPH_POINT_RELEASE=) 2024-09-13T15:17:14.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:14 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:14.092+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:14.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:14 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:14.092+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:14.965 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:14 smithi172 podman[265577]: 2024-09-13 15:17:14.693463626 +0000 UTC m=+5.957258644 container cleanup bd1089408fdb85b977d7d29d36af75e4e1b333807aad44c3ad82a4b8b1678c2e (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:17:14.965 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:14 smithi172 bash[265577]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5 2024-09-13T15:17:14.965 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:14 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:14.527+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:14.965 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:14 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:14.527+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:15.241 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:14 smithi172 podman[265604]: 2024-09-13 15:17:14.96389892 +0000 UTC m=+0.637505142 container remove bd1089408fdb85b977d7d29d36af75e4e1b333807aad44c3ad82a4b8b1678c2e (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-13T15:17:15.242 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:15 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:15.125+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:15.242 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:15 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:15.125+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:15.242 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:15 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:15.125+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:15.532 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:15 smithi172 podman[265691]: 2024-09-13 15:17:15.17982016 +0000 UTC m=+0.016212421 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:17:15.532 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:15 smithi172 podman[265691]: 2024-09-13 15:17:15.28847122 +0000 UTC m=+0.124863469 container create bf904b8ecaeafb6b0e6eebbee939ee7462f979d402fc9b06f5f714ffe2960bdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=reef-2c01458, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-13T15:17:15.533 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:15 smithi172 podman[265691]: 2024-09-13 15:17:15.531565034 +0000 UTC m=+0.367957297 container init bf904b8ecaeafb6b0e6eebbee939ee7462f979d402fc9b06f5f714ffe2960bdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS) 2024-09-13T15:17:15.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:15 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:15.573+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:15.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:15 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:15.573+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:15.992 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:15 smithi172 podman[265691]: 2024-09-13 15:17:15.53533666 +0000 UTC m=+0.371728916 container start bf904b8ecaeafb6b0e6eebbee939ee7462f979d402fc9b06f5f714ffe2960bdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-2c01458, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-13T15:17:15.992 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:15 smithi172 podman[265691]: 2024-09-13 15:17:15.59826397 +0000 UTC m=+0.434656226 container attach bf904b8ecaeafb6b0e6eebbee939ee7462f979d402fc9b06f5f714ffe2960bdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, ceph=True) 2024-09-13T15:17:15.992 INFO:journalctl@ceph.osd.5.smithi172.stdout:Sep 13 15:17:15 smithi172 podman[265691]: 2024-09-13 15:17:15.725555083 +0000 UTC m=+0.561947341 container died bf904b8ecaeafb6b0e6eebbee939ee7462f979d402fc9b06f5f714ffe2960bdd (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=reef-2c01458, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-13T15:17:16.327 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:16.148+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:16.328 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:16.148+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:16.328 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:16.148+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:16.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.5.service' 2024-09-13T15:17:16.401 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:17:16.402 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-13T15:17:16.402 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T15:17:16.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.6 2024-09-13T15:17:16.714 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:16 smithi172 systemd[1]: Stopping Ceph osd.6 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:16.715 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:16.560+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:16.715 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:16.560+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:16.715 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:16.560+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:48.259203+0000 front 2024-09-13T15:16:48.259228+0000 (oldest deadline 2024-09-13T15:17:15.859526+0000) 2024-09-13T15:17:16.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:16.713+0000 7f7fc91d0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:17:16.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:16.713+0000 7f7fc91d0640 -1 osd.6 525 *** Got signal Terminated *** 2024-09-13T15:17:16.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:16 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:16.713+0000 7f7fc91d0640 -1 osd.6 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:17:17.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:17 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:17.180+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:17.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:17 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:17.180+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:17.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:17 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:17.180+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:17.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:17 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:17.550+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:17.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:17 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:17.550+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:17.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:17 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:17.550+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:48.259203+0000 front 2024-09-13T15:16:48.259228+0000 (oldest deadline 2024-09-13T15:17:15.859526+0000) 2024-09-13T15:17:18.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:18.176+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:18.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:18.176+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:18.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:18.176+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:18.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:18.569+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:18.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:18.569+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:18.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:18 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:18.569+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:48.259203+0000 front 2024-09-13T15:16:48.259228+0000 (oldest deadline 2024-09-13T15:17:15.859526+0000) 2024-09-13T15:17:19.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:19.149+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:19.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:19.149+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:19.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:19.149+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:19.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:19.149+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:19.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:19.558+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:19.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:19.558+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:19.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:19 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:19.558+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:48.259203+0000 front 2024-09-13T15:16:48.259228+0000 (oldest deadline 2024-09-13T15:17:15.859526+0000) 2024-09-13T15:17:20.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:20.160+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:20.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:20.160+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:20.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:20.160+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:20.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:20.160+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:20.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:20.604+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:20.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:20.604+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:20.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:20.604+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:48.259203+0000 front 2024-09-13T15:16:48.259228+0000 (oldest deadline 2024-09-13T15:17:15.859526+0000) 2024-09-13T15:17:20.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:20 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:20.604+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:55.860002+0000 front 2024-09-13T15:16:55.859985+0000 (oldest deadline 2024-09-13T15:17:19.959845+0000) 2024-09-13T15:17:21.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:21.124+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:21.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:21.124+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:21.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:21.124+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:21.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:21.124+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:21.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:21.649+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:30.256178+0000 front 2024-09-13T15:16:30.256285+0000 (oldest deadline 2024-09-13T15:16:54.956304+0000) 2024-09-13T15:17:21.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:21.649+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:37.257249+0000 front 2024-09-13T15:16:37.257420+0000 (oldest deadline 2024-09-13T15:17:07.257667+0000) 2024-09-13T15:17:21.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:21.649+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:48.259203+0000 front 2024-09-13T15:16:48.259228+0000 (oldest deadline 2024-09-13T15:17:15.859526+0000) 2024-09-13T15:17:21.992 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:21 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6[48984]: 2024-09-13T15:17:21.649+0000 7f7fc17dc640 -1 osd.6 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:55.860002+0000 front 2024-09-13T15:16:55.859985+0000 (oldest deadline 2024-09-13T15:17:19.959845+0000) 2024-09-13T15:17:22.423 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:22 smithi172 podman[265794]: 2024-09-13 15:17:22.176575873 +0000 UTC m=+5.556982356 container died ac940ce031a6976f553553b83128561eff1c6ab492058e0b8ac3b3dcb01d2059 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=reef-2c01458) 2024-09-13T15:17:22.424 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:22.122+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:22.424 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:22.122+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:22.424 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:22.122+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:22.424 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:22 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:22.122+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:22.690 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:22 smithi172 podman[265794]: 2024-09-13 15:17:22.512572265 +0000 UTC m=+5.892978739 container cleanup ac940ce031a6976f553553b83128561eff1c6ab492058e0b8ac3b3dcb01d2059 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=reef-2c01458, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:17:22.690 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:22 smithi172 bash[265794]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6 2024-09-13T15:17:22.991 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:22 smithi172 podman[265810]: 2024-09-13 15:17:22.689081668 +0000 UTC m=+0.512671844 container remove ac940ce031a6976f553553b83128561eff1c6ab492058e0b8ac3b3dcb01d2059 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, RELEASE=reef-2c01458, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:17:23.360 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:23 smithi172 podman[265902]: 2024-09-13 15:17:23.098314961 +0000 UTC m=+0.096215131 container create 48ba2693facbd21836efd267418437047d0926114f595413e267da918e53f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-13T15:17:23.361 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:23 smithi172 podman[265902]: 2024-09-13 15:17:23.017991679 +0000 UTC m=+0.015891847 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:17:23.361 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:23.115+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:23.361 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:23.115+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:23.361 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:23.115+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:23.361 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:23 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:23.115+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:23.741 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:23 smithi172 podman[265902]: 2024-09-13 15:17:23.359334761 +0000 UTC m=+0.357234940 container init 48ba2693facbd21836efd267418437047d0926114f595413e267da918e53f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-13T15:17:23.742 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:23 smithi172 podman[265902]: 2024-09-13 15:17:23.36303629 +0000 UTC m=+0.360936466 container start 48ba2693facbd21836efd267418437047d0926114f595413e267da918e53f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, CEPH_POINT_RELEASE=) 2024-09-13T15:17:23.742 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:23 smithi172 podman[265902]: 2024-09-13 15:17:23.458947641 +0000 UTC m=+0.456847810 container attach 48ba2693facbd21836efd267418437047d0926114f595413e267da918e53f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T15:17:23.742 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:23 smithi172 podman[265902]: 2024-09-13 15:17:23.543807471 +0000 UTC m=+0.541707642 container died 48ba2693facbd21836efd267418437047d0926114f595413e267da918e53f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:17:24.140 INFO:journalctl@ceph.osd.6.smithi172.stdout:Sep 13 15:17:24 smithi172 podman[265926]: 2024-09-13 15:17:24.137017495 +0000 UTC m=+0.587228405 container remove 48ba2693facbd21836efd267418437047d0926114f595413e267da918e53f482 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-2c01458, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True) 2024-09-13T15:17:24.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T15:17:24.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:17:24.204 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T15:17:24.204 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-13T15:17:24.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.7 2024-09-13T15:17:24.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:24.155+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:24.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:24.155+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:24.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:24.155+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:24.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:24.155+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:24.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 systemd[1]: Stopping Ceph osd.7 for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:24.991 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:24.557+0000 7fd60709d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:17:24.992 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:24.557+0000 7fd60709d640 -1 osd.7 525 *** Got signal Terminated *** 2024-09-13T15:17:24.992 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:24 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:24.557+0000 7fd60709d640 -1 osd.7 525 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:17:25.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:25.180+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:25.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:25.180+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:25.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:25.180+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:25.492 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:25 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:25.180+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:26.481 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:26 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:26.216+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:26.481 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:26 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:26.216+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:26.481 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:26 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:26.216+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:26.482 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:26 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:26.216+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:27.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:27 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:27.265+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:27.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:27 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:27.265+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:27.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:27 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:27.265+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:27.742 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:27 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:27.265+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:27.742 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:27 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:27.265+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-09-13T15:17:04.316594+0000 front 2024-09-13T15:17:04.316645+0000 (oldest deadline 2024-09-13T15:17:27.216574+0000) 2024-09-13T15:17:28.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:28 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:28.253+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:28.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:28 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:28.253+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:28.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:28 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:28.253+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:28.742 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:28 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:28.253+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:28.742 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:28 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:28.253+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-09-13T15:17:04.316594+0000 front 2024-09-13T15:17:04.316645+0000 (oldest deadline 2024-09-13T15:17:27.216574+0000) 2024-09-13T15:17:29.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:29 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:29.267+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6806 osd.0 since back 2024-09-13T15:16:32.212367+0000 front 2024-09-13T15:16:32.212237+0000 (oldest deadline 2024-09-13T15:16:58.012853+0000) 2024-09-13T15:17:29.741 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:29 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:29.267+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6814 osd.1 since back 2024-09-13T15:16:38.513850+0000 front 2024-09-13T15:16:38.513997+0000 (oldest deadline 2024-09-13T15:17:04.413768+0000) 2024-09-13T15:17:29.742 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:29 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:29.267+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6822 osd.2 since back 2024-09-13T15:16:49.615335+0000 front 2024-09-13T15:16:49.615251+0000 (oldest deadline 2024-09-13T15:17:14.315216+0000) 2024-09-13T15:17:29.742 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:29 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:29.267+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.154:6830 osd.3 since back 2024-09-13T15:16:54.315665+0000 front 2024-09-13T15:16:54.315720+0000 (oldest deadline 2024-09-13T15:17:19.015654+0000) 2024-09-13T15:17:29.742 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:29 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7[54962]: 2024-09-13T15:17:29.267+0000 7fd6036b1640 -1 osd.7 525 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-09-13T15:17:04.316594+0000 front 2024-09-13T15:17:04.316645+0000 (oldest deadline 2024-09-13T15:17:27.216574+0000) 2024-09-13T15:17:30.276 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:30 smithi172 podman[266006]: 2024-09-13 15:17:30.00127091 +0000 UTC m=+5.574108621 container died aa72c07d8bc1c1981de28fa9e288c15fea1d60e8585ed11266d2a11f6702d437 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-13T15:17:30.634 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:30 smithi172 podman[266006]: 2024-09-13 15:17:30.398312142 +0000 UTC m=+5.971149841 container cleanup aa72c07d8bc1c1981de28fa9e288c15fea1d60e8585ed11266d2a11f6702d437 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:17:30.634 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:30 smithi172 bash[266006]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7 2024-09-13T15:17:30.926 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:30 smithi172 podman[266034]: 2024-09-13 15:17:30.632956344 +0000 UTC m=+0.625479943 container remove aa72c07d8bc1c1981de28fa9e288c15fea1d60e8585ed11266d2a11f6702d437 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-13T15:17:31.228 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:30 smithi172 podman[266123]: 2024-09-13 15:17:30.925435514 +0000 UTC m=+0.095449623 container create 91d78fded5a430a4182a5908562ce0ce5dd4b5c6e50cd450439d66e83f528272 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:17:31.228 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:30 smithi172 podman[266123]: 2024-09-13 15:17:30.845784399 +0000 UTC m=+0.015798511 image pull c129f5a2fd4eccce3e6eec43830eff1929af98bc48f3b850cf144ce5b326e8b5 quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11 2024-09-13T15:17:31.229 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:31 smithi172 podman[266123]: 2024-09-13 15:17:31.152543272 +0000 UTC m=+0.322557381 container init 91d78fded5a430a4182a5908562ce0ce5dd4b5c6e50cd450439d66e83f528272 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, RELEASE=reef-2c01458, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T15:17:31.229 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:31 smithi172 podman[266123]: 2024-09-13 15:17:31.155817857 +0000 UTC m=+0.325831976 container start 91d78fded5a430a4182a5908562ce0ce5dd4b5c6e50cd450439d66e83f528272 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T15:17:31.229 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:31 smithi172 podman[266123]: 2024-09-13 15:17:31.227521144 +0000 UTC m=+0.397535258 container attach 91d78fded5a430a4182a5908562ce0ce5dd4b5c6e50cd450439d66e83f528272 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-2c01458, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-13T15:17:31.491 INFO:journalctl@ceph.osd.7.smithi172.stdout:Sep 13 15:17:31 smithi172 podman[266123]: 2024-09-13 15:17:31.34291172 +0000 UTC m=+0.512925833 container died 91d78fded5a430a4182a5908562ce0ce5dd4b5c6e50cd450439d66e83f528272 (image=quay.ceph.io/ceph-ci/ceph@sha256:c37c250ffe653639d7a4124ac8acdc3c8f6a53cb4ef1f26b1d2df36648631a11, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=reef-2c01458, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-13T15:17:31.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@osd.7.service' 2024-09-13T15:17:32.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:17:32.035 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-13T15:17:32.035 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-13T15:17:32.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a 2024-09-13T15:17:32.241 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 systemd[1]: Stopping Ceph prometheus.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:32.672 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:17:32.672 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-13T15:17:32.673 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-13T15:17:32.673 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-13T15:17:32.673 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-13T15:17:32.673 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-13T15:17:32.673 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-13T15:17:32.673 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-13T15:17:32.673 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.367Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-13T15:17:32.674 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.370Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-13T15:17:32.674 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.370Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-13T15:17:32.674 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a[86426]: ts=2024-09-13T15:17:32.370Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-13T15:17:32.674 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 podman[266225]: 2024-09-13 15:17:32.377976056 +0000 UTC m=+0.118974021 container died 12b1d8bc5be4b55197b1a17ea4a00c6f5cb064126e61751ff30a0c01a9aaf36d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:32.991 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 podman[266225]: 2024-09-13 15:17:32.758606276 +0000 UTC m=+0.499604240 container cleanup 12b1d8bc5be4b55197b1a17ea4a00c6f5cb064126e61751ff30a0c01a9aaf36d (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:32.991 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Sep 13 15:17:32 smithi172 bash[266225]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-prometheus-a 2024-09-13T15:17:33.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-447b0cd2-71db-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-13T15:17:33.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:17:33.177 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-13T15:17:33.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:17:33.333 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T15:17:34.971 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:34 smithi154 systemd[1]: Stopping Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:35.404 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:35 smithi154 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a[97582]: ts=2024-09-13T15:17:35.051Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-13T15:17:35.404 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:35 smithi154 podman[358352]: 2024-09-13 15:17:35.102255087 +0000 UTC m=+0.178143218 container died 4971176d5c4c08bc19e4a2410b51a6bca7643fb47425a3e9849cd0fd14c06fd2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:35.720 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:35 smithi154 podman[358352]: 2024-09-13 15:17:35.593200325 +0000 UTC m=+0.669088504 container cleanup 4971176d5c4c08bc19e4a2410b51a6bca7643fb47425a3e9849cd0fd14c06fd2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:35.721 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:35 smithi154 bash[358352]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a 2024-09-13T15:17:36.057 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:35 smithi154 podman[358364]: 2024-09-13 15:17:35.794425874 +0000 UTC m=+0.692829187 container remove 4971176d5c4c08bc19e4a2410b51a6bca7643fb47425a3e9849cd0fd14c06fd2 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:36.057 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:35 smithi154 podman[358364]: 2024-09-13 15:17:35.862172007 +0000 UTC m=+0.760575346 volume remove 4e4e4eaf54cf399fb58c1b7cdfb8b9c98f9ea87184b3f22dd11c1c62ecfe86b7 2024-09-13T15:17:36.471 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:36 smithi154 systemd[1]: Stopping Ceph node-exporter.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:36.471 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:36 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-13T15:17:36.471 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:36 smithi154 systemd[1]: Stopped Ceph alertmanager.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T15:17:36.471 INFO:journalctl@ceph.alertmanager.a.smithi154.stdout:Sep 13 15:17:36 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 3.168s CPU time. 2024-09-13T15:17:36.941 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:36 smithi154 podman[358473]: 2024-09-13 15:17:36.675971499 +0000 UTC m=+0.277162966 container died c2e00912b58cf27fe98516d0507ac06175dfd66f99c5604248d44c97134ab2e4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:37.220 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:37 smithi154 podman[358473]: 2024-09-13 15:17:37.115052626 +0000 UTC m=+0.716244138 container cleanup c2e00912b58cf27fe98516d0507ac06175dfd66f99c5604248d44c97134ab2e4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:37.220 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:37 smithi154 bash[358473]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a 2024-09-13T15:17:37.720 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:37 smithi154 podman[358485]: 2024-09-13 15:17:37.391273529 +0000 UTC m=+0.718766909 container remove c2e00912b58cf27fe98516d0507ac06175dfd66f99c5604248d44c97134ab2e4 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-13T15:17:37.721 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:37 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:17:37.721 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:37 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-13T15:17:37.721 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:37 smithi154 systemd[1]: Stopped Ceph node-exporter.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T15:17:37.721 INFO:journalctl@ceph.node-exporter.a.smithi154.stdout:Sep 13 15:17:37 smithi154 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 7.478s CPU time. 2024-09-13T15:17:46.225 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T15:17:46.373 INFO:teuthology.orchestra.run.smithi172.stdout:Deleting cluster with fsid: 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T15:17:47.985 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:47 smithi172 systemd[1]: Stopping Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:47.986 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=server t=2024-09-13T15:17:47.953706204Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-13T15:17:47.986 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=tracing t=2024-09-13T15:17:47.954010026Z level=info msg="Closing tracing" 2024-09-13T15:17:47.986 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=ticker t=2024-09-13T15:17:47.954053506Z level=info msg=stopped last_tick=2024-09-13T15:17:40Z 2024-09-13T15:17:47.986 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:47 smithi172 ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a[91424]: logger=grafana-apiserver t=2024-09-13T15:17:47.954104795Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-13T15:17:48.241 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:47 smithi172 podman[266632]: 2024-09-13 15:17:47.984480044 +0000 UTC m=+0.129067145 container died 4bed2859671631cd5728c3df744e1b3622445dc1abcdb69a58ee7e4867892335 (image=quay.io/ceph/grafana:10.4.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T15:17:48.741 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:48 smithi172 podman[266632]: 2024-09-13 15:17:48.428409066 +0000 UTC m=+0.572996166 container cleanup 4bed2859671631cd5728c3df744e1b3622445dc1abcdb69a58ee7e4867892335 (image=quay.io/ceph/grafana:10.4.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T15:17:48.741 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:48 smithi172 bash[266632]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a 2024-09-13T15:17:49.039 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:48 smithi172 podman[266648]: 2024-09-13 15:17:48.74640061 +0000 UTC m=+0.764662975 container remove 4bed2859671631cd5728c3df744e1b3622445dc1abcdb69a58ee7e4867892335 (image=quay.io/ceph/grafana:10.4.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-09-13T15:17:49.040 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:48 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-13T15:17:49.040 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:48 smithi172 systemd[1]: Stopped Ceph grafana.a for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T15:17:49.040 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Sep 13 15:17:48 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@grafana.a.service: Consumed 29.004s CPU time. 2024-09-13T15:17:49.294 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:49 smithi172 systemd[1]: Stopping Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968... 2024-09-13T15:17:49.550 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:49 smithi172 podman[266754]: 2024-09-13 15:17:49.293319732 +0000 UTC m=+0.117905006 container died 60a8496c91efed099f0187e733f24cfdc9c3b83e14e40634edc67f83b280ffb6 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:17:49.909 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:49 smithi172 podman[266754]: 2024-09-13 15:17:49.640833652 +0000 UTC m=+0.465418931 container cleanup 60a8496c91efed099f0187e733f24cfdc9c3b83e14e40634edc67f83b280ffb6 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:17:49.909 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:49 smithi172 bash[266754]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b 2024-09-13T15:17:50.238 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:49 smithi172 podman[266767]: 2024-09-13 15:17:49.908506161 +0000 UTC m=+0.617144844 container remove 60a8496c91efed099f0187e733f24cfdc9c3b83e14e40634edc67f83b280ffb6 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-447b0cd2-71db-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-13T15:17:50.238 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:49 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-13T15:17:50.239 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:50 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-13T15:17:50.239 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:50 smithi172 systemd[1]: Stopped Ceph node-exporter.b for 447b0cd2-71db-11ef-bceb-c7b262605968. 2024-09-13T15:17:50.239 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Sep 13 15:17:50 smithi172 systemd[1]: ceph-447b0cd2-71db-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 6.732s CPU time. 2024-09-13T15:17:54.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:17:54.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:17:55.010 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-13T15:17:55.012 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564/remote/smithi154/crash 2024-09-13T15:17:55.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:17:55.065 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:17:55.065 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:17:55.067 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564/remote/smithi172/crash 2024-09-13T15:17:55.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/crash -- . 2024-09-13T15:17:55.109 INFO:teuthology.orchestra.run.smithi172.stderr:tar: /var/lib/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T15:17:55.109 INFO:teuthology.orchestra.run.smithi172.stderr:tar: Error is not recoverable: exiting now 2024-09-13T15:17:55.111 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-13T15:17:55.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_AVAILABILITY | head -n 1 2024-09-13T15:17:55.182 INFO:teuthology.orchestra.run.smithi154.stdout:2024-09-13T14:27:02.028372+0000 mon.a (mon.0) 665 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:17:55.187 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-13T15:17:55.187 INFO:tasks.cephadm:Compressing logs... 2024-09-13T15:17:55.188 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:17:55.190 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T15:17:55.230 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T15:17:55.230 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-13T15:17:55.230 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-13T15:17:55.232 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:17:55.232 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:17:55.232 INFO:teuthology.orchestra.run.smithi154.stderr:gzip/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-volume.log: -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-13T15:17:55.236 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:17:55.236 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-13T15:17:55.236 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-13T15:17:55.236 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-13T15:17:55.236 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:17:55.236 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mds.foo.smithi154.daekzd.log 2024-09-13T15:17:55.237 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-13T15:17:55.237 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:17:55.238 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.c.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.cephadm.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.a.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mgr.y.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.2.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mds.foo.smithi154.daekzd.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.3.log: 86.1% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mds.foo.smithi154.daekzd.log.gz 2024-09-13T15:17:55.245 INFO:teuthology.orchestra.run.smithi154.stderr: 92.9% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:17:55.248 INFO:teuthology.orchestra.run.smithi172.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T15:17:55.248 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-13T15:17:55.248 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-13T15:17:55.250 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-13T15:17:55.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-13T15:17:55.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-volume.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-13T15:17:55.252 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mds.foo.smithi172.pmmovd.log 2024-09-13T15:17:55.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-13T15:17:55.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mds.foo.smithi172.pmmovd.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-13T15:17:55.253 INFO:teuthology.orchestra.run.smithi172.stderr: 84.1% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mds.foo.smithi172.pmmovd.log.gz 2024-09-13T15:17:55.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.log 2024-09-13T15:17:55.253 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-13T15:17:55.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.5.log: /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-13T15:17:55.254 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-13T15:17:55.406 INFO:teuthology.orchestra.run.smithi154.stderr: 92.0% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:17:55.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 85.8% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-13T15:17:55.585 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-13T15:17:55.586 INFO:teuthology.orchestra.run.smithi154.stderr: 95.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:17:55.590 INFO:teuthology.orchestra.run.smithi154.stderr: 91.2% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:17:55.592 INFO:teuthology.orchestra.run.smithi172.stderr: 90.5% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:17:55.711 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-13T15:17:55.932 INFO:teuthology.orchestra.run.smithi172.stderr: 90.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-13T15:17:56.318 INFO:teuthology.orchestra.run.smithi172.stderr: 91.3% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-13T15:17:57.140 INFO:teuthology.orchestra.run.smithi154.stderr: 94.3% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-13T15:17:57.672 INFO:teuthology.orchestra.run.smithi154.stderr: 89.8% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-13T15:17:59.617 INFO:teuthology.orchestra.run.smithi172.stderr: 92.3% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-13T15:18:19.046 INFO:teuthology.orchestra.run.smithi154.stderr: 91.7% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-13T15:18:41.739 INFO:teuthology.orchestra.run.smithi172.stderr: 92.9% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-13T15:18:55.947 INFO:teuthology.orchestra.run.smithi154.stderr: 90.4% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-13T15:19:11.712 INFO:teuthology.orchestra.run.smithi172.stderr: 94.0% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-13T15:21:40.804 INFO:teuthology.orchestra.run.smithi172.stderr: 93.7% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-13T15:22:33.208 INFO:teuthology.orchestra.run.smithi172.stderr: 92.3% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-13T15:22:33.209 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-13T15:22:33.210 INFO:teuthology.orchestra.run.smithi172.stderr:real 4m38.004s 2024-09-13T15:22:33.210 INFO:teuthology.orchestra.run.smithi172.stderr:user 4m26.543s 2024-09-13T15:22:33.210 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m11.555s 2024-09-13T15:36:28.343 INFO:teuthology.orchestra.run.smithi154.stderr: 93.3% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-13T15:40:34.626 INFO:teuthology.orchestra.run.smithi154.stderr: 93.0% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-13T15:43:14.245 INFO:teuthology.orchestra.run.smithi154.stderr: 92.8% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-13T15:43:32.692 INFO:teuthology.orchestra.run.smithi154.stderr: 92.8% -- replaced with /var/log/ceph/447b0cd2-71db-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-13T15:43:32.734 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-13T15:43:32.734 INFO:teuthology.orchestra.run.smithi154.stderr:real 25m37.529s 2024-09-13T15:43:32.734 INFO:teuthology.orchestra.run.smithi154.stderr:user 8m0.809s 2024-09-13T15:43:32.734 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m25.739s 2024-09-13T15:43:32.735 INFO:tasks.cephadm:Archiving logs... 2024-09-13T15:43:32.736 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564/remote/smithi154/log 2024-09-13T15:43:32.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:44:46.765 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564/remote/smithi172/log 2024-09-13T15:44:46.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T15:45:32.292 INFO:tasks.cephadm:Removing cluster... 2024-09-13T15:45:32.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 --force 2024-09-13T15:45:32.907 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T15:45:33.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 447b0cd2-71db-11ef-bceb-c7b262605968 --force 2024-09-13T15:45:33.890 INFO:teuthology.orchestra.run.smithi172.stdout:Deleting cluster with fsid: 447b0cd2-71db-11ef-bceb-c7b262605968 2024-09-13T15:45:34.515 INFO:tasks.cephadm:Removing cephadm ... 2024-09-13T15:45:34.516 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:45:34.532 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T15:45:34.548 INFO:tasks.cephadm:Teardown complete 2024-09-13T15:45:34.548 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T15:45:34.559 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T15:45:34.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:45:34.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T15:45:34.686 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T15:45:34.698 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T15:45:34.698 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:45:34.701 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:45:34.713 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:45:34.716 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:45:34.735 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:45:34.735 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-09-13T15:45:34.735 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv01.front.sepia.ceph.com 3 6 377 22 +277us[ +280us] +/- 46ms 2024-09-13T15:45:34.735 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -749us[ -746us] +/- 62ms 2024-09-13T15:45:34.735 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +396us[ +396us] +/- 74ms 2024-09-13T15:45:34.735 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:45:34.811 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:45:34.811 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-13T15:45:34.812 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 +308us[ +302us] +/- 46ms 2024-09-13T15:45:34.812 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 28 -694us[ -700us] +/- 62ms 2024-09-13T15:45:34.812 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 25 +400us[ +395us] +/- 74ms 2024-09-13T15:45:34.812 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:45:34.814 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T15:45:34.823 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T15:45:34.823 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T15:45:34.834 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:45:35.048 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T15:45:35.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T15:45:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:45:35.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T15:45:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:45:35.225 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T15:45:35.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T15:45:35.247 INFO:teuthology.task.internal:Duration was 5499.214479 seconds 2024-09-13T15:45:35.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T15:45:35.265 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T15:45:35.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:45:35.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T15:45:35.330 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:45:35.367 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:45:35.553 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T15:45:35.553 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T15:45:35.553 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:45:35.578 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-09-13T15:45:35.578 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T15:45:35.603 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T15:45:35.604 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:45:35.620 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T15:45:35.670 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T15:45:35.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:45:35.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T15:45:37.717 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T15:45:38.373 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T15:45:38.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:45:38.404 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T15:45:38.441 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T15:45:39.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:45:39.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:45:39.170 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-09-13T15:45:39.202 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-09-13T15:45:39.221 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:45:39.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:45:39.238 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:45:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:45:39.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T15:45:39.543 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T15:45:39.624 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564/remote/smithi154 2024-09-13T15:45:39.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:45:39.858 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_10:00:18-upgrade-wip-aclamk-cbt-combined-distro-default-smithi/7903564/remote/smithi172 2024-09-13T15:45:39.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T15:45:39.910 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T15:45:39.910 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:45:39.913 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T15:45:39.959 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T15:45:39.970 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T15:45:39.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T15:45:39.980 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T15:45:39.981 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:45:39.983 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T15:45:40.016 INFO:teuthology.orchestra.run.smithi154.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:45 /home/ubuntu/cephtest 2024-09-13T15:45:40.016 INFO:teuthology.orchestra.run.smithi154.stdout: 531937 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:13 /home/ubuntu/cephtest/mnt.0 2024-09-13T15:45:40.017 INFO:teuthology.orchestra.run.smithi154.stdout: 531938 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 13 15:13 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T15:45:40.017 INFO:teuthology.orchestra.run.smithi154.stdout: 4470108 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 15:13 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T15:45:40.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:45:40.019 INFO:teuthology.orchestra.run.smithi172.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 15:45 /home/ubuntu/cephtest 2024-09-13T15:45:40.019 INFO:teuthology.orchestra.run.smithi154.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-13T15:45:40.019 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:45:40.028 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T15:45:40.039 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T15:45:40.053 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi154 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-13T15:45:40.058 INFO:teuthology.run:Summary data: description: upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} duration: 5499.214478969574 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi154 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_akupczyk@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9f6577a310db4e1dad891c305d37716b status: fail success: false 2024-09-13T15:45:40.058 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T15:45:40.183 INFO:teuthology.run:FAIL